The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

quotas and disk space usage problem

Discussion in 'General Discussion' started by ouki, Mar 25, 2016.

  1. ouki

    ouki Member

    Joined:
    Feb 23, 2006
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Hi,

    My server was reporting quota and disk space usage correctly, until yesterday when I had the crazy idea to run /scripts/fixquotas.

    Since then, the disk space usage by all hosted accounts is reported by WHM either 0 or with very low values, that are untrue.

    I searched the Internet and this forum for advice and I implemented all relevant instructions, failing to solve the problem.

    During the last 20 hours, I ran /scripts/fixquotas at least three times, without having any success.

    So, as last resort, I am kindly asking for your help.
    What can I do to fix it.

    Thank you for your help.


    MY SYSTEM INFO
    Code:
    grep '' /etc/redhat-release /usr/local/cpanel/version /var/cpanel/envtype ; grep CPANEL= /etc/cpupdate.conf ; httpd -v ; php -v ; mysql -V
    
    /etc/redhat-release:CentOS release 5.11 (Final)
    /usr/local/cpanel/version:11.52.4.2
    /var/cpanel/envtype:standard
    CPANEL=11.52
    Server version: Apache/2.2.29 (Unix)
    Server built:   Apr 25 2015 00:55:09
    Cpanel::Easy::Apache v3.28.8 rev9999
    PHP 5.4.40 (cli) (built: Apr 25 2015 01:00:10)
    Copyright (c) 1997-2014 The PHP Group
    Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
        with the ionCube PHP Loader v4.6.1, Copyright (c) 2002-2014, by ionCube Ltd., and
        with Zend Guard Loader v3.3, Copyright (c) 1998-2013, by Zend Technologies
    mysql  Ver 14.14 Distrib 5.5.47, for Linux (x86_64) using readline 5.1
    /etc/fstab
    Code:
    LABEL=/ /       ext3    usrjquota=quota.user,jqfmt=vfsv0        1       1
    LABEL=/home     /home   ext3    usrjquota=quota.user,jqfmt=vfsv0        1       2
    LABEL=/var      /var    ext3    usrjquota=quota.user,jqfmt=vfsv0        1       2
    LABEL=/usr      /usr    ext3    usrjquota=quota.user,jqfmt=vfsv0        1       2
    LABEL=/tmp              /tmp                    ext3    defaults        1 2
    LABEL=/boot             /boot                   ext3    defaults        1 2
    tmpfs                   /dev/shm                tmpfs   defaults        0 0
    devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
    sysfs                   /sys                    sysfs   defaults        0 0
    proc                    /proc                   proc    defaults        0 0
    LABEL=SWAP-sda7         swap                    swap    defaults        0 0
    /tmp             /var/tmp                    ext3    defaults,bind,noauto        0 0
    //10.0.5.5/IS-12571     /nas                    cifs    username=IS-12571,password=ms4u5Pmxwb 1 1
    /scripts/fixquotas output
    Code:
    Installing Default Quota Databases......Done
    journaled quota support: kernel supports, user space tools supports (available)
    Warning virtual file systems are mounted. Quotas may be counted as double
    for users who are currently logged in. Please have jailshell users logout
    before running quotacheck in the future!
    All jailed users will be logged out in 60 seconds!
    Broadcast message from root (Fri Mar 25 09:41:14 2016):
    Warning! The system is about to perform quota maintenance.
    All users will be logged out in 60 seconds.
    PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your
    account.
    jailshell: no process killed
    checking out /tmp/backup
    checking out /backup
    checking out /backup
    Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process.
    / (already configured quotas = 1).
    /home (already configured quotas = 1).
    /var (already configured quotas = 1).
    /usr (already configured quotas = 1).
    /tmp (already configured quotas = 0).
    /boot (already configured quotas = 0).
    Updating Quota Files......
            quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 2965 directories and 435752 files
            quotacheck: Old file not found.
            quotacheck: Scanning /dev/sda8 [/home] done
            quotacheck: Something weird happened while scanning. Error 2133571361
            quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 1767 directories and 24100 files
            quotacheck: Old file not found.
            quotacheck: Scanning /dev/sda6 [/usr] done
            quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 14003 directories and 147670 files
            quotacheck: Old file not found.
            done
    
    ....Done
    Quotas have been enabled and updated.
    Reading package NNNNNN
    Resetting quota for XXXXX to 1500 M
    edquota: Quota file not found or has wrong format.
    Reading package NNNNNN
    Resetting quota for XXXXX to 600 M
    edquota: Quota file not found or has wrong format.
    etc..
    
     
  2. vanessa

    vanessa Well-Known Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    817
    Likes Received:
    22
    Trophy Points:
    18
    Location:
    Virginia Beach, VA
    cPanel Access Level:
    DataCenter Provider
    Try moving the aquota.* files out of / and /home, then run:

    /scripts/initquotas
    /scripts/fixquotas
     
  3. ouki

    ouki Member

    Joined:
    Feb 23, 2006
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Hi Vanessa,
    Thank you for your reply!

    There are no aquota* files inside / and /home.

    Inside /home there is only a quota.group file.
    I rename it to /home/__quota.group.

    Inside / there is only a quota.user file.
    I tried to rename it to /__quota.user but system doesn't allow me to do it.

    So I renamed /home/__quota.group back to /home/quota.group.

    After that, I haven't tried to run the two commands that you suggested.
     
  4. ouki

    ouki Member

    Joined:
    Feb 23, 2006
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Hi Vanessa,

    I managed to move quota files out of / and /home directories (the i attribute was set).

    A) I ran /scripts/initquotas
    Now WHM shows quota as unlimited and disk used with low values (from 0 to a few MB which is incorrect).
    In / there is a quota.user.new file
    in /home there is no quota file

    /scripts/initquotas outputed:
    Code:
    journaled quota support: kernel supports, user space tools supports (available)
    Warning virtual file systems are mounted. Quotas may be counted as double
    for users who are currently logged in. Please have jailshell users logout
    before running quotacheck in the future!
    
    All jailed users will be logged out in 60 seconds!
    
    
    Broadcast message from root (Sat Mar 26 14:10:34 2016):
    
    Warning! The system is about to perform quota maintenance.
    All users will be logged out in 60 seconds.
    PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your
    account.
    jailshell: no process killed
    checking out /tmp/backup
    checking out /backup
    checking out /backup
    Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process.
    / (already configured quotas = 1).
    /home (already configured quotas = 1).
    /var (already configured quotas = 1).
    /usr (already configured quotas = 1).
    /tmp (already configured quotas = 0).
    /boot (already configured quotas = 0).
    Updating Quota Files......
            quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 2966 directories and 435787 files
            quotacheck: Cannot find checked quota file for users on /dev/root!
            quotacheck: Scanning /dev/sda8 [/home] done
            quotacheck: Something weird happened while scanning. Error 2133571361
            quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 1764 directories and 24133 files
            quotacheck: Old file not found.
            quotacheck: Scanning /dev/sda6 [/usr] done
            quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 14003 directories and 147691 files
            quotacheck: Old file not found.
            done
    
    ....Done
    Quotas have been enabled and updated.
    B) Then I ran /scripts/fixquotas
    Now WHM shows quota correctly, but still disk used with low values (from 0 to a few MB which is incorrect).
    In / there is a quota.user.new file
    in /home there is no quota file

    /scripts/fixquotas outputed:
    Code:
    Installing Default Quota Databases......Done
    journaled quota support: kernel supports, user space tools supports (available)
    Warning virtual file systems are mounted. Quotas may be counted as double
    for users who are currently logged in. Please have jailshell users logout
    before running quotacheck in the future!
    
    All jailed users will be logged out in 60 seconds!
    
    
    Broadcast message from root (Sat Mar 26 14:23:30 2016):
    
    Warning! The system is about to perform quota maintenance.
    All users will be logged out in 60 seconds.
    PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your
    account.
    jailshell: no process killed
    checking out /tmp/backup
    checking out /backup
    checking out /backup
    Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process.
    / (already configured quotas = 1).
    /home (already configured quotas = 1).
    /var (already configured quotas = 1).
    /usr (already configured quotas = 1).
    /tmp (already configured quotas = 0).
    /boot (already configured quotas = 0).
    Updating Quota Files......
            quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 2966 directories and 435788 files
            quotacheck: Can't create new quotafile //quota.user.new: File exists
            quotacheck: Cannot initialize IO on new quotafile: File exists
            quotacheck: Scanning /dev/sda8 [/home] done
            quotacheck: Something weird happened while scanning. Error 2133571361
            quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 1765 directories and 24089 files
            quotacheck: Old file not found.
            quotacheck: Scanning /dev/sda6 [/usr] done
            quotacheck: Cannot stat old user quota file: No such file or directory
            quotacheck: Old group file not found. Usage will not be substracted.
            quotacheck: Checked 14003 directories and 147666 files
            quotacheck: Old file not found.
            done
    
    ....Done
    Quotas have been enabled and updated.
    Reading package PACKAGE
    Resetting quota for USERNAME to 1500 M
    edquota: Quota file not found or has wrong format.
    Reading package PACKAGE
    Resetting quota for USERNAME to 600 M
    edquota: Quota file not found or has wrong format.
    ...
    ...
    ...
    C) The situation is the same as it was at the beginning.
    Per account disk usage is reported incorrectly.

    Also, please note the "Something weird happened while scanning. Error 2133571361" message, while running both commands.

    Thank you!
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,832
    Likes Received:
    672
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    Could you review /var/log/messages at the corresponding time this error message was generated to see if you notice any specific output? I've seen instances where this stems from a failing hard drive.

    Thank you.
     
  6. ouki

    ouki Member

    Joined:
    Feb 23, 2006
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Hello,

    Thank you for your reply!

    I searched /var/log/messages for "error" and found several messages like the following:
    kernel: [14859083.169308] Buffer I/O error on device sda8, logical block 207389274
    kernel: [14772771.647017] Buffer I/O error on device sda8, logical block 207389308
    kernel: [14859083.169101] sd 0:0:0:0: SCSI error: return code = 0x08100002
    kernel: [14681695.865730] php[18675]: segfault at 00002b80cd721479 rip 000000000072d910 rsp 00007fff536cf1a0 error 4

    This server has 2 SATA disks using hardware RAID 1.
    RAID monitoring (provided by data center), shows disks as healthy.
     
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,832
    Likes Received:
    672
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You may want to search for the specific time stamps associated with the /scripts/fixquotas commands as opposed to only searching for the "error" term so that you can determine if there are any specific messages associated with that particular quota error.

    Thank you.
     
  8. ouki

    ouki Member

    Joined:
    Feb 23, 2006
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Hello,

    I just checked logs around the time i've last ran initquotas and fixquotas commands. I didn't find something special, or I am missing it because I am not sure what to look for.
    Could you please tell me what to look for?

    Thank you!
     
  9. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,832
    Likes Received:
    672
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You are looking for error messages that suggest a damaged hard drive. EX:

    Code:
    Buffer I/O error
    You can also review /var/log/dmesg for additional output, or utilize a utility such as smartctl:

    Linux utility for Disk health Monitoring

    Thank you.
     
  10. ouki

    ouki Member

    Joined:
    Feb 23, 2006
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Hello,

    Thank you for your reply.

    You are right! Indeed I found the following messages in /var/log/messages:

    Code:
    kernel: [14468436.849848] sd 0:0:0:0: Unhandled sense code
    kernel: [14468436.849852] sd 0:0:0:0: SCSI error: return code = 0x08100002
    kernel: [14468436.849854] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
    kernel: [14468436.849927] sda: Current: sense key: Hardware Error
    kernel: [14468436.849929]     Add. Sense: Internal target failure
    kernel: [14468436.849934]
    kernel: [14468436.849996] Buffer I/O error on device sda8, logical block 207389274
    kernel: [14468436.850067] Buffer I/O error on device sda8, logical block 207389275
    kernel: [14468436.850134] Buffer I/O error on device sda8, logical block 207389276
    kernel: [14468436.850203] Buffer I/O error on device sda8, logical block 207389277
    kernel: [14468436.850270] Buffer I/O error on device sda8, logical block 207389278
    kernel: [14468436.850342] Buffer I/O error on device sda8, logical block 207389279
    kernel: [14468436.850412] Buffer I/O error on device sda8, logical block 207389280
    kernel: [14468436.850479] Buffer I/O error on device sda8, logical block 207389281
    kernel: [14468436.850548] Buffer I/O error on device sda8, logical block 207389282
    kernel: [14468436.850624] Buffer I/O error on device sda8, logical block 207389283
    kernel: [14468436.952903] sd 0:0:0:0: Unhandled sense code
    kernel: [14468436.952905] sd 0:0:0:0: SCSI error: return code = 0x08100002
    kernel: [14468436.952908] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
    kernel: [14468436.952975] sda: Current: sense key: Hardware Error
    kernel: [14468436.952977]     Add. Sense: Internal target failure
    kernel: [14468436.952980]
    kernel: [14468437.037448] sd 0:0:0:0: Unhandled sense code
    kernel: [14468437.037451] sd 0:0:0:0: SCSI error: return code = 0x08100002
    kernel: [14468437.037454] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
    kernel: [14468437.037525] sda: Current: sense key: Hardware Error
    kernel: [14468437.037527]     Add. Sense: Internal target failure
    kernel: [14468437.037531]
    kernel: [14468437.088533] sd 0:0:0:0: Unhandled sense code
    kernel: [14468437.088536] sd 0:0:0:0: SCSI error: return code = 0x08100002
    kernel: [14468437.088538] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
    kernel: [14468437.088606] sda: Current: sense key: Hardware Error
    kernel: [14468437.088608]     Add. Sense: Internal target failure
    I opened a ticket with the data center and I am awaiting for their reply.
    I hope there is a way to replace disks (RAID 1) without losing any data.

    Thank you!
     
  11. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,832
    Likes Received:
    672
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
  12. ouki

    ouki Member

    Joined:
    Feb 23, 2006
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Thank you both you and vanessa for your support!
    I am on a discussion with technicians in data center about the best method to deal with this situation.
     
Loading...

Share This Page