The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Receiving e-mail about drive being full, but it is not?

Discussion in 'E-mail Discussions' started by noimad1, Jan 22, 2007.

  1. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    I keep getting an email from one of my dedicated server clients that says:

    Drive Critical: /hda3 (/) is 99% full

    However, hda3 is not full. I've done a du -h on the drive, and only 14gB out of 80GB are being used. Also, when I go to "show disk usage" in the WHM, it shows it 99% full?

    Anyone had this problem before? It is concerning me, even though I know the drive is not full.

    Thanks,
    Damion
     
  2. xerophyte

    xerophyte Well-Known Member

    Joined:
    Mar 16, 2003
    Messages:
    216
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    post your cpanel version and df -h command output
     
  3. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16

    Ok, here we go:

    cpanel version: WHM 10.8.0 cPanel 10.9.0-R57

    df -h output:
    /dev/hda3 74G 69G 1.2G 99% /
    /dev/hda1 99M 11M 83M 12% /boot
    none 475M 0 475M 0% /dev/shm
    /usr/tmpDSK 485M 11M 449M 3% /tmp
    /tmp 485M 11M 449M 3% /var/tmp



    now, when I do a du -h from the / directory, the very last entry says "12GB ." I would assume that adds up all of the totals from the du -h command? Could it be missing something?
     
  4. xerophyte

    xerophyte Well-Known Member

    Joined:
    Mar 16, 2003
    Messages:
    216
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    Code:
    man du 
    type
    Code:
     du -csh / 
    Summarize disk usage of each FILE, recursively for directories.

    Code:
    -c, --total
                  produce a grand total
    
    -h, --human-readable
                  print sizes in human readable format (e.g., 1K 234M 2G)
    
    -s, --summarize
                  display only a total for each argument
    
    please always check the man doc for the command it can save lots of your time and ours

    hope that helps
     
  5. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    Very sorry, guess I should have read the man page. I thought du -h * would check all files and give me a total, which i believe it did.

    Here is the output of du -csh:

    12G /
    12G total


    So why is df reporting full?
     
  6. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    Ok, this got worse. Today, the customers sites wouldn't load, and when I tried to get into cpanel it said the drive is completely full. I had to remove a few files to get it running.

    However, here is the issue. I am pretty sure only 12GB of the drive is actually being used based on the reports I've ran. So why is cpanel showing the drive to be full? What could be going on here?
     
  7. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    But you posted your df output earlier and it clearly showed the disk being 99% full. So where is the problem? Is df now showing below 90% used? If not, then you need to clear down space on that partition. You cannot compare the output of du and df as they du doesn't take into account deleted files that are still open (this usually happens if you clear down log files and have forgotten to restart the daemons using them as they will continue to write to the deleted file).
     
  8. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    So would all I need to do is restart some of the daemons? The df currently shows 99% being used. I just don't know how to fix that. I never really cleared anything on this server, the df just keeps showing full. So if it is indeed full, then I would like to find out where the files are that are taking up the space. The only way I know how to do that is using du? Is there another command I can use to find files that might have been deleted, but not showing up in the du?
     
    #8 noimad1, Jan 28, 2007
    Last edited: Jan 28, 2007
  9. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    This is what my datacenter said when I opened a ticket with them:

    "I looks like the filesystem is corrupted. The best way to fix this is to bring the server down and do a full FSCK on the / filesystem."

    Does that sound like something that needs to be done in this case? If there is something I can try without having to bring the server down I would rather try it....
     
  10. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    You can find out if there are processes running with deleted files open using lsof, e.g.:

    lsof / | grep deleted

    If that doesn't reveal anything then an FSCK is the way to go - but make sure you have secured backups incase the FSCK blows the disk.
     
  11. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16

    lsof / | grep deleted did it. Thank you so much!

    There was a process open that was using some old files....I killed those processes, and now the drive is back to normal...
     
  12. pphillips

    pphillips Well-Known Member

    Joined:
    Nov 14, 2003
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    6
    I am having a similar problem, though df and du report the drive has plenty of space left.

    From cPanel:
    Drive Critical: md0 (/) is 92% full

    df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/md0 70G 29G 37G 44% /
    /dev/sda1 99M 17M 78M 18% /boot
    none 1.5G 0 1.5G 0% /dev/shm
    /usr/tmpDSK 485M 12M 448M 3% /tmp
    /tmp 485M 12M 448M 3% /var/tmp

    du -csh /
    du: `/proc/27025/task': No such file or directory
    du: `/proc/27025/fd': No such file or directory
    31G /
    31G total


    The server has been rebooted several times since these messages started, but they continue to come, and although the server's disk usage has not grown, the amount of disk space used in these emails continues to grow. Any idea?

    Thanks!
     
  13. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16


    Did you try typing lsof / | grep deleted to see if there are any processes running? do that. Type:

    lsof / | grep deleted

    Then kill any processes listed. See if that changes anything.
     
  14. pphillips

    pphillips Well-Known Member

    Joined:
    Nov 14, 2003
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    6
    I tried that, same result. What commands does cpanel issue to determine the drive usage? Everything I try tells me the hard drive usage is well below what cPanel reports it to be.
     
  15. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    I'd say it used the "df" command, but I'm not sure. It would be interesting to see what happened to your disk space utilization figures after a reboot... that may fix your problems. If it does, you had the deleted-file-still-open problem (that's what the lsof check above is for).
     
  16. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    He noted that he already rebooted the server several times.

    It would be rather odd if it's due to using a raid mount for the root partition....
     
  17. pphillips

    pphillips Well-Known Member

    Joined:
    Nov 14, 2003
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    6
    I thought of that too, when i doubt, reboot.. but It has been rebooted at least half a dozen times since the messages started coming from cPanel, and they continue. df is consistently reporting around 40% disk usage, while the cPanel emails consistently report between 81 and 92% usage. Very odd.
     
  18. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    The function uses df -P to gernate the stats. Could you post the output from that command?
     
  19. pphillips

    pphillips Well-Known Member

    Joined:
    Nov 14, 2003
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    6
    df -P
    Filesystem 1024-blocks Used Available Capacity Mounted on
    /dev/md0 72666728 28285772 40689628 42% /
    /dev/sda1 101086 16725 79142 18% /boot
    none 1557528 0 1557528 0% /dev/shm
    /usr/tmpDSK 495844 19043 451201 5% /tmp
    /tmp 495844 19043 451201 5% /var/tmp

    This morning's cPanel email at 2:10am:

    Drive Warning: md0 (/) is 82% full

    One thing comes to mind,.. daily backups. The server creates about 15GB of backup files and FTP's them to our backup server every morning. The backups usually get FTP'd over around 6am, and these messages are sent around 2am. Is it probable that the backup process is consuming around 30GB of space while making 15GB worth of backup files?
     
  20. cPanelKenneth

    cPanelKenneth cPanel Development
    Staff Member

    Joined:
    Apr 7, 2006
    Messages:
    4,458
    Likes Received:
    22
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    The backup suggestion is very plausible.

    Looks like something for our documentation after I verify.
     
Loading...

Share This Page