quotas and disk space usage problem

ouki · Mar 25, 2016

Hi,

My server was reporting quota and disk space usage correctly, until yesterday when I had the crazy idea to run /scripts/fixquotas.

Since then, the disk space usage by all hosted accounts is reported by WHM either 0 or with very low values, that are untrue.

I searched the Internet and this forum for advice and I implemented all relevant instructions, failing to solve the problem.

During the last 20 hours, I ran /scripts/fixquotas at least three times, without having any success.

So, as last resort, I am kindly asking for your help.
What can I do to fix it.

Thank you for your help.

MY SYSTEM INFO

Code:

grep '' /etc/redhat-release /usr/local/cpanel/version /var/cpanel/envtype ; grep CPANEL= /etc/cpupdate.conf ; httpd -v ; php -v ; mysql -V

/etc/redhat-release:CentOS release 5.11 (Final)
/usr/local/cpanel/version:11.52.4.2
/var/cpanel/envtype:standard
CPANEL=11.52
Server version: Apache/2.2.29 (Unix)
Server built:   Apr 25 2015 00:55:09
Cpanel::Easy::Apache v3.28.8 rev9999
PHP 5.4.40 (cli) (built: Apr 25 2015 01:00:10)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
    with the ionCube PHP Loader v4.6.1, Copyright (c) 2002-2014, by ionCube Ltd., and
    with Zend Guard Loader v3.3, Copyright (c) 1998-2013, by Zend Technologies
mysql  Ver 14.14 Distrib 5.5.47, for Linux (x86_64) using readline 5.1

/etc/fstab

Code:

LABEL=/ /       ext3    usrjquota=quota.user,jqfmt=vfsv0        1       1
LABEL=/home     /home   ext3    usrjquota=quota.user,jqfmt=vfsv0        1       2
LABEL=/var      /var    ext3    usrjquota=quota.user,jqfmt=vfsv0        1       2
LABEL=/usr      /usr    ext3    usrjquota=quota.user,jqfmt=vfsv0        1       2
LABEL=/tmp              /tmp                    ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda7         swap                    swap    defaults        0 0
/tmp             /var/tmp                    ext3    defaults,bind,noauto        0 0
//10.0.5.5/IS-12571     /nas                    cifs    username=IS-12571,password=ms4u5Pmxwb 1 1

/scripts/fixquotas output

Code:

Installing Default Quota Databases......Done
journaled quota support: kernel supports, user space tools supports (available)
Warning virtual file systems are mounted. Quotas may be counted as double
for users who are currently logged in. Please have jailshell users logout
before running quotacheck in the future!
All jailed users will be logged out in 60 seconds!
Broadcast message from root (Fri Mar 25 09:41:14 2016):
Warning! The system is about to perform quota maintenance.
All users will be logged out in 60 seconds.
PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your
account.
jailshell: no process killed
checking out /tmp/backup
checking out /backup
checking out /backup
Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process.
/ (already configured quotas = 1).
/home (already configured quotas = 1).
/var (already configured quotas = 1).
/usr (already configured quotas = 1).
/tmp (already configured quotas = 0).
/boot (already configured quotas = 0).
Updating Quota Files......
        quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory
        quotacheck: Old group file not found. Usage will not be substracted.
        quotacheck: Checked 2965 directories and 435752 files
        quotacheck: Old file not found.
        quotacheck: Scanning /dev/sda8 [/home] done
        quotacheck: Something weird happened while scanning. Error 2133571361
        quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory
        quotacheck: Old group file not found. Usage will not be substracted.
        quotacheck: Checked 1767 directories and 24100 files
        quotacheck: Old file not found.
        quotacheck: Scanning /dev/sda6 [/usr] done
        quotacheck: Cannot stat old user quota file: No such file or directory
        quotacheck: Old group file not found. Usage will not be substracted.
        quotacheck: Checked 14003 directories and 147670 files
        quotacheck: Old file not found.
        done

....Done
Quotas have been enabled and updated.
Reading package NNNNNN
Resetting quota for XXXXX to 1500 M
edquota: Quota file not found or has wrong format.
Reading package NNNNNN
Resetting quota for XXXXX to 600 M
edquota: Quota file not found or has wrong format.
etc..

vanessa · Mar 25, 2016

Try moving the aquota.* files out of / and /home, then run:

/scripts/initquotas
/scripts/fixquotas

ouki · Mar 26, 2016

Hi Vanessa,
Thank you for your reply!

There are no aquota* files inside / and /home.

Inside /home there is only a quota.group file.
I rename it to /home/__quota.group.

Inside / there is only a quota.user file.
I tried to rename it to /__quota.user but system doesn't allow me to do it.

So I renamed /home/__quota.group back to /home/quota.group.

After that, I haven't tried to run the two commands that you suggested.

ouki · Mar 26, 2016

Hi Vanessa,

I managed to move quota files out of / and /home directories (the i attribute was set).

A) I ran /scripts/initquotas
Now WHM shows quota as unlimited and disk used with low values (from 0 to a few MB which is incorrect).
In / there is a quota.user.new file
in /home there is no quota file

/scripts/initquotas outputed:

Code:

journaled quota support: kernel supports, user space tools supports (available)
Warning virtual file systems are mounted. Quotas may be counted as double
for users who are currently logged in. Please have jailshell users logout
before running quotacheck in the future!

All jailed users will be logged out in 60 seconds!

Broadcast message from root (Sat Mar 26 14:10:34 2016):

Warning! The system is about to perform quota maintenance.
All users will be logged out in 60 seconds.
PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your
account.
jailshell: no process killed
checking out /tmp/backup
checking out /backup
checking out /backup
Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process.
/ (already configured quotas = 1).
/home (already configured quotas = 1).
/var (already configured quotas = 1).
/usr (already configured quotas = 1).
/tmp (already configured quotas = 0).
/boot (already configured quotas = 0).
Updating Quota Files......
quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory
quotacheck: Old group file not found. Usage will not be substracted.
quotacheck: Checked 2966 directories and 435787 files
quotacheck: Cannot find checked quota file for users on /dev/root!
quotacheck: Scanning /dev/sda8 [/home] done
quotacheck: Something weird happened while scanning. Error 2133571361
quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory
quotacheck: Old group file not found. Usage will not be substracted.
quotacheck: Checked 1764 directories and 24133 files
quotacheck: Old file not found.
quotacheck: Scanning /dev/sda6 [/usr] done
quotacheck: Cannot stat old user quota file: No such file or directory
quotacheck: Old group file not found. Usage will not be substracted.
quotacheck: Checked 14003 directories and 147691 files
quotacheck: Old file not found.
done

....Done
Quotas have been enabled and updated.

B) Then I ran /scripts/fixquotas
Now WHM shows quota correctly, but still disk used with low values (from 0 to a few MB which is incorrect).
In / there is a quota.user.new file
in /home there is no quota file

/scripts/fixquotas outputed:

Code:

Installing Default Quota Databases......Done
journaled quota support: kernel supports, user space tools supports (available)
Warning virtual file systems are mounted. Quotas may be counted as double
for users who are currently logged in. Please have jailshell users logout
before running quotacheck in the future!

All jailed users will be logged out in 60 seconds!


Broadcast message from root (Sat Mar 26 14:23:30 2016):

Warning! The system is about to perform quota maintenance.
All users will be logged out in 60 seconds.
PLEASE Do not log back in for 30 minutes, or you may inadvertantly disable your
account.
jailshell: no process killed
checking out /tmp/backup
checking out /backup
checking out /backup
Warning : Your system does not have a separate filesystem for backups. This may cause performance degradation during the backup process.
/ (already configured quotas = 1).
/home (already configured quotas = 1).
/var (already configured quotas = 1).
/usr (already configured quotas = 1).
/tmp (already configured quotas = 0).
/boot (already configured quotas = 0).
Updating Quota Files......
        quotacheck: Scanning /dev/root [/] quotacheck: Cannot stat old user quota file: No such file or directory
        quotacheck: Old group file not found. Usage will not be substracted.
        quotacheck: Checked 2966 directories and 435788 files
        quotacheck: Can't create new quotafile //quota.user.new: File exists
        quotacheck: Cannot initialize IO on new quotafile: File exists
        quotacheck: Scanning /dev/sda8 [/home] done
        quotacheck: Something weird happened while scanning. Error 2133571361
        quotacheck: Scanning /dev/sda2 [/var] quotacheck: Cannot stat old user quota file: No such file or directory
        quotacheck: Old group file not found. Usage will not be substracted.
        quotacheck: Checked 1765 directories and 24089 files
        quotacheck: Old file not found.
        quotacheck: Scanning /dev/sda6 [/usr] done
        quotacheck: Cannot stat old user quota file: No such file or directory
        quotacheck: Old group file not found. Usage will not be substracted.
        quotacheck: Checked 14003 directories and 147666 files
        quotacheck: Old file not found.
        done

....Done
Quotas have been enabled and updated.
Reading package PACKAGE
Resetting quota for USERNAME to 1500 M
edquota: Quota file not found or has wrong format.
Reading package PACKAGE
Resetting quota for USERNAME to 600 M
edquota: Quota file not found or has wrong format.
...
...
...

C) The situation is the same as it was at the beginning.
Per account disk usage is reported incorrectly.

Also, please note the "Something weird happened while scanning. Error 2133571361" message, while running both commands.

Thank you!

cPanelMichael · Mar 28, 2016

ouki said: ↑

Also, please note the "Something weird happened while scanning. Error 2133571361" message, while running both commands.
Click to expand...

Hello

Could you review /var/log/messages at the corresponding time this error message was generated to see if you notice any specific output? I've seen instances where this stems from a failing hard drive.

Thank you.

ouki · Mar 31, 2016

Hello,

Thank you for your reply!

I searched /var/log/messages for "error" and found several messages like the following:
kernel: [14859083.169308] Buffer I/O error on device sda8, logical block 207389274
kernel: [14772771.647017] Buffer I/O error on device sda8, logical block 207389308
kernel: [14859083.169101] sd 0:0:0:0: SCSI error: return code = 0x08100002
kernel: [14681695.865730] php[18675]: segfault at 00002b80cd721479 rip 000000000072d910 rsp 00007fff536cf1a0 error 4

This server has 2 SATA disks using hardware RAID 1.
RAID monitoring (provided by data center), shows disks as healthy.

cPanelMichael · Mar 31, 2016

You may want to search for the specific time stamps associated with the /scripts/fixquotas commands as opposed to only searching for the "error" term so that you can determine if there are any specific messages associated with that particular quota error.

Thank you.

ouki · Mar 31, 2016

Hello,

I just checked logs around the time i've last ran initquotas and fixquotas commands. I didn't find something special, or I am missing it because I am not sure what to look for.
Could you please tell me what to look for?

Thank you!

cPanelMichael · Mar 31, 2016

You are looking for error messages that suggest a damaged hard drive. EX:
Code:
Buffer I/O error
You can also review /var/log/dmesg for additional output, or utilize a utility such as smartctl:

Linux utility for Disk health Monitoring

Thank you.

ouki · Mar 31, 2016

Hello,

Thank you for your reply.

You are right! Indeed I found the following messages in /var/log/messages:

Code:

kernel: [14468436.849848] sd 0:0:0:0: Unhandled sense code
kernel: [14468436.849852] sd 0:0:0:0: SCSI error: return code = 0x08100002
kernel: [14468436.849854] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
kernel: [14468436.849927] sda: Current: sense key: Hardware Error
kernel: [14468436.849929]     Add. Sense: Internal target failure
kernel: [14468436.849934]
kernel: [14468436.849996] Buffer I/O error on device sda8, logical block 207389274
kernel: [14468436.850067] Buffer I/O error on device sda8, logical block 207389275
kernel: [14468436.850134] Buffer I/O error on device sda8, logical block 207389276
kernel: [14468436.850203] Buffer I/O error on device sda8, logical block 207389277
kernel: [14468436.850270] Buffer I/O error on device sda8, logical block 207389278
kernel: [14468436.850342] Buffer I/O error on device sda8, logical block 207389279
kernel: [14468436.850412] Buffer I/O error on device sda8, logical block 207389280
kernel: [14468436.850479] Buffer I/O error on device sda8, logical block 207389281
kernel: [14468436.850548] Buffer I/O error on device sda8, logical block 207389282
kernel: [14468436.850624] Buffer I/O error on device sda8, logical block 207389283
kernel: [14468436.952903] sd 0:0:0:0: Unhandled sense code
kernel: [14468436.952905] sd 0:0:0:0: SCSI error: return code = 0x08100002
kernel: [14468436.952908] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
kernel: [14468436.952975] sda: Current: sense key: Hardware Error
kernel: [14468436.952977]     Add. Sense: Internal target failure
kernel: [14468436.952980]
kernel: [14468437.037448] sd 0:0:0:0: Unhandled sense code
kernel: [14468437.037451] sd 0:0:0:0: SCSI error: return code = 0x08100002
kernel: [14468437.037454] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
kernel: [14468437.037525] sda: Current: sense key: Hardware Error
kernel: [14468437.037527]     Add. Sense: Internal target failure
kernel: [14468437.037531]
kernel: [14468437.088533] sd 0:0:0:0: Unhandled sense code
kernel: [14468437.088536] sd 0:0:0:0: SCSI error: return code = 0x08100002
kernel: [14468437.088538] Result: hostbyte=invalid driverbyte=DRIVER_SENSE,SUGGEST_OK
kernel: [14468437.088606] sda: Current: sense key: Hardware Error
kernel: [14468437.088608]     Add. Sense: Internal target failure

I opened a ticket with the data center and I am awaiting for their reply.
I hope there is a way to replace disks (RAID 1) without losing any data.

Thank you!

cPanelMichael · Mar 31, 2016

Thank you for updating us with the outcome. You should be able to backup or transfer your accounts as long as it's just quotas that are affected at this point:

Transfer Tool - Documentation - cPanel Documentation

Thank you.

ouki · Mar 31, 2016

Thank you both you and vanessa for your support!
I am on a discussion with technicians in data center about the best method to deal with this situation.

Forums

The Community Forums

Interact with an entire community of cPanel & WHM users!

quotas and disk space usage problem

ouki Member

vanessa Well-Known Member
PartnerNOC

ouki Member

ouki Member

cPanelMichael Forums Analyst
Staff Member

ouki Member

cPanelMichael Forums Analyst
Staff Member

ouki Member

cPanelMichael Forums Analyst
Staff Member

ouki Member

cPanelMichael Forums Analyst
Staff Member

ouki Member

Quotas - Accounts Created With Unlimited Disk Space (XFS only?)

Email quotas after mailbox conversion

/scripts/cleanquotas fails during upcp

Safe to close terminal if /scripts/fixquotas is running?

Mailbox compression & quotas

Share This Page

Useful Searches

The Community Forums

Interact with an entire community of cPanel & WHM users!

quotas and disk space usage problem

ouki Member

vanessa Well-Known Member PartnerNOC

ouki Member

ouki Member

cPanelMichael Forums Analyst Staff Member

ouki Member

cPanelMichael Forums Analyst Staff Member

ouki Member

cPanelMichael Forums Analyst Staff Member

ouki Member

cPanelMichael Forums Analyst Staff Member

ouki Member

Quotas - Accounts Created With Unlimited Disk Space (XFS only?)

Email quotas after mailbox conversion

/scripts/cleanquotas fails during upcp

Safe to close terminal if /scripts/fixquotas is running?

Mailbox compression & quotas

Share This Page

vanessa Well-Known Member
PartnerNOC

cPanelMichael Forums Analyst
Staff Member

cPanelMichael Forums Analyst
Staff Member

cPanelMichael Forums Analyst
Staff Member

cPanelMichael Forums Analyst
Staff Member