SSL, Slow ?

We have just setup a shared SSL certificate, and all appears to be well.
The only problem we have noticed is the speed: It feels terribly slow (slower that the CPanel / WHM over SSL definatly). Most specifically with images. Other SSL sites that I've seen don't exhibt this problem.

I will admit this is the first ssl system I've setup.

I have tested under XP SP2 with IE6 and Win98 with IE 5.5 -- the Win98 definatly is a LOT faster at loading. Is this a IE6 problem? Any way round it? (I have checked that the do not save encrypted pages is unticked in options).

I have also checked the state of the server (TOP & I/O activity: Both are ok - top is 0.01 ).

Any ideas?