The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Could not create new files inside "/", COMET is to blame ?

Discussion in 'General Discussion' started by nyjimbo, Sep 8, 2013.

  1. nyjimbo

    nyjimbo Well-Known Member

    Joined:
    Jan 25, 2003
    Messages:
    1,125
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    New York
    We just has a situation on one of our servers were everything seemed to be fine but we could no longer create NEW files on the "/" mount point. Disk space used was only about 40% but nothing NEW could be written. Lots of stuff in WHM were failing but the server seemed fine, SMART was ok, nothing weird in logs.

    After banging my head against the wall I found that "/root/.cpanel/comet/channels" had almost a gigabyte of files in there that appear to be email related.

    After deleting some of these files suddenly we could write new files again..

    What is "comet" and why is it doing this ? Was this about using up all file handles and therefore not being able to write new files ? Can we just delete all of these files or will it break something ?

    What can we do to avoid this in the future ?

    Thank you.
     
  2. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    998
    Likes Received:
    10
    Trophy Points:
    18
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
  3. nyjimbo

    nyjimbo Well-Known Member

    Joined:
    Jan 25, 2003
    Messages:
    1,125
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    New York
    Thank you. It is inodes, after deleting a lot of stuff it shows:

    Code:
    root@xxxxx[~]# df -i
    Filesystem            Inodes   IUsed   IFree IUse% Mounted on
    /dev/sda6             262144  226897   35247   87% /
    tmpfs                 487769       1  487768    1% /dev/shm
    /dev/sdb1            18317312 1725052 16592260   10% /home2
    /dev/sda1              52416      46   52370    1% /boot
    /dev/sdc1            30531584     363 30531221    1% /backup
    /dev/sda8            5218304   47717 5170587    1% /home
    /dev/sda7             131072     194  130878    1% /tmp
    /dev/sda2            1638400  219199 1419201   14% /usr
    /dev/sda3            1638400   34844 1603556    3% /var
    root@xxxxx [~]#
    I am going to delete more so we don't see this again for a while.
     
  4. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    Comet (Comet (programming) - Wikipedia, the free encyclopedia) is a way of streaming information from the web server (in this case the cPanel & WHM daemon cpsrvd) and the browser. In cPanel & WHM its primary use is with the Email Queue Manager, and Email Delivery Reports.

    We have an internal case (59137) regarding the problem you describe. I'll speak to some of our developers about prioritizing a fix for this. In the meantime I recommend having something monitor /root/.cpanel/comet, clearing out files once it reaches a certain threshold.
     
  5. R2B2

    R2B2 Member

    Joined:
    Jul 29, 2013
    Messages:
    13
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Thanks.

    Follow-up question -- I'm on a server that's used over 1.7M Inodes inside of /root/.comet. :(

    Running /usr/local/cpanel/bin/purge_dead_comet_files only deleted about 140K files.

    Is it ok to delete everything inside of /root/.cpanel/comet/channels, and subscriptions?
     
  6. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    653
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    There is an additional case open (#88721) to further improve the handling of comet datastore files. You can monitor the cPanel 11.42 change log for this case number:

    cPanel 11.42 - Change Log

    Manually deleting the files is not recommended unless you are at or close to 100% inode usage and need the additional space.

    Thank you.
     
Loading...

Share This Page