Website Running Extremely Slow At Times

[ghanaexpo]

Folks,
My website, ghanaexpo.com, has been running very slow at times. Sometimes, It does not open at all. I just subscribed to a 100Mbits dedicated server from unmeteredservers.com for $299.00/month, plus $35.00/month for the CPanel. I uploaded my website just last Tuesday, (4 days ago).

This is a movie/music website, with an average of about 50 people on the site at the same time for now. It will definitely increase to between 400 and 500, as from past experience. At the moment, I have about 60GB of MP3 music and about 70 movies (movies are on the average of less than 200mb/movie, with 300 bitrate VP6-E FLV, 12,000 framerate in 320x240 size, of between 60 and 90 minutes each). I will be uploading about 500 more of such movies

Now, the site keeps going down from time to time - sometimes about 7 times in a day. Or it gets really slow, taking forever to open. When I contacted the Technical Support, they told me that I am already close to the limit of my 100Mbps as at that time (when I had only about 53 people on the site at the same time). They are already suggesting that I upgrade my bandwidth port to a highrer level (the next level that they have is 250Mbps for $749.00/month.

HERE IS MY QUESTION: IS THE 100MBPS explanation from technical support really true?

What kind of server and bandwidth would I need to successfully host such a site, if Technical Support is right on their bandwidth explanation?

The following are the server specifications:
Dell 1425
Intel Dual Xeon 3.0Ghz
4GB RAM
2 x 500GB SATA2 SW Raid0/1
100Mbit Unmetered Guaranteed Usage Port
/28 Vlan - 11 Usable IPS
Top Ranked Tier1 Network
No Cogent!
No Contract
Administration/Support
Self-Managed Server
99.99% Network Uptime Guarantee
Remote Reboot Port
24x7 Network Monitoring
24x7 Ticket/Phone/Email
Full Root Access
On-Demand Managed Services $55/hr

Thanks.