Post a traceroute to the server. Run it from where you are having problems and also from a "looking glass" website or from www.dnsstuff.com and post the results. Could be anything, we need to see a traceroute to say anything approaching intelligence.
As brian said run some traceroutes to look into the problem. Also run them from your server to your home ip. It could be a local routing problem or it could be a problem with the server/data center.
