Harddisk suddenly fills up with hundreds of GB of data until it's full

Ebridge

Member
May 3, 2012
16
1
53
cPanel Access Level
Root Administrator
Last weeks I had the following happening on our server, a couple of times:
  • suddenly something starts occupying disk space on the disk
  • this continues until the harddisk is full (within minutes, > 1GB/sec!)
  • at that point, the system becomes fully unusable (system load > 120)
  • after a while (about 30 minutes), the harddisk starts freeing up, at the same rate (>1GB/sec)

Because of the high system loads it's almost impossible to run any commands, but at the worst moment, df -h shows:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
446G 446G 0G 100% /
/dev/sda1 99M 33M 62M 35% /boot
tmpfs 1.6G 0 1.6G 0% /dev/shm
/usr/tmpDSK 2.0G 642M 1.3G 34% /tmp
So it seems to me that it's not the tmp disk causing the problems.

The high system load at these moments seems to be caused by httpd, but I'm not sure it's also responsible for the occupation of disk space.
The increase/decrease of file space occupation is bigger than the fysical speed of the harddisk, so I'm assuming something is allocating the disk space.

Any ideas on how to troubleshoot this problem?

Kind regards
 

nospa

Well-Known Member
Apr 23, 2012
110
0
66
cPanel Access Level
Reseller Owner
Try to locate core.XXXXXX files, for example:

#updatedb
#locate core.

This could be because of core dumps.
 

ruzbehraja

Well-Known Member
May 19, 2011
392
11
68
cPanel Access Level
Root Administrator
It may be a failling HDD. I would contact the Data Center and ask them to investigate too.

The indication in WHM maybe wrong.

What is it getting filled up with? Are you able to see / delete the files?

Does this happen at regular intervals or is it random?
 

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
40
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
Can you check if you have /root/.cpanel/comet on the server when it starts filling up and what size that folder happens to be?