Could not create new files inside "/", COMET is to blame ?

[nyjimbo]

We just has a situation on one of our servers were everything seemed to be fine but we could no longer create NEW files on the "/" mount point. Disk space used was only about 40% but nothing NEW could be written. Lots of stuff in WHM were failing but the server seemed fine, SMART was ok, nothing weird in logs.

After banging my head against the wall I found that "/root/.cpanel/comet/channels" had almost a gigabyte of files in there that appear to be email related.

After deleting some of these files suddenly we could write new files again..

What is "comet" and why is it doing this ? Was this about using up all file handles and therefore not being able to write new files ? Can we just delete all of these files or will it break something ?

What can we do to avoid this in the future ?

Thank you.

 

[quietFinn]

I found a thread regarding that:
http://forums.cpanel.net/f5/strange-directory-inside-root-270191.html

You can check the inode usage like this:

df -i /

 

[nyjimbo]

Thank you. It is inodes, after deleting a lot of stuff it shows:

[email protected][~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda6             262144  226897   35247   87% /
tmpfs                 487769       1  487768    1% /dev/shm
/dev/sdb1            18317312 1725052 16592260   10% /home2
/dev/sda1              52416      46   52370    1% /boot
/dev/sdc1            30531584     363 30531221    1% /backup
/dev/sda8            5218304   47717 5170587    1% /home
/dev/sda7             131072     194  130878    1% /tmp
/dev/sda2            1638400  219199 1419201   14% /usr
/dev/sda3            1638400   34844 1603556    3% /var
[email protected] [~]#

I am going to delete more so we don't see this again for a while.

 

[cPanelKenneth]

Comet (Comet (programming) - Wikipedia, the free encyclopedia) is a way of streaming information from the web server (in this case the cPanel & WHM daemon cpsrvd) and the browser. In cPanel & WHM its primary use is with the Email Queue Manager, and Email Delivery Reports.

We have an internal case (59137) regarding the problem you describe. I'll speak to some of our developers about prioritizing a fix for this. In the meantime I recommend having something monitor /root/.cpanel/comet, clearing out files once it reaches a certain threshold.

 

[R2B2]

Thanks.

Follow-up question -- I'm on a server that's used over 1.7M Inodes inside of /root/.comet.

Running /usr/local/cpanel/bin/purge_dead_comet_files only deleted about 140K files.

Is it ok to delete everything inside of /root/.cpanel/comet/channels, and subscriptions?

 

[cPanelMichael]

Thanks.

Follow-up question -- I'm on a server that's used over 1.7M Inodes inside of /root/.comet.

Running /usr/local/cpanel/bin/purge_dead_comet_files only deleted about 140K files.

Is it ok to delete everything inside of /root/.cpanel/comet/channels, and subscriptions?

There is an additional case open (#88721) to further improve the handling of comet datastore files. You can monitor the cPanel 11.42 change log for this case number:

cPanel 11.42 - Change Log

Manually deleting the files is not recommended unless you are at or close to 100% inode usage and need the additional space.

Thank you.