cPanel wrong disk usage report

[thesufi]

Hello,

I am having some unusual trouble with one of my cPanel accounts. I have a reseller account from a provider, where I have some 8-9GB email hosted under one cPanel account.
On the quick stats on cPanel home page (first image), it shows Disk Usage is 8.03 GB of 11.72 GB (account is limited to 12000MB).

However, in "cPanel > Disk Usage", it shows "34,122.91 MB total disk space used." (second image).


This problem is mostly with email accounts and their wrong disk usage data. When I sort email accounts by "Storage used" from high to low (third image), counting only first page of 5 pages of disk usage manually gives around 14.3 GB usage, that is after leaving "Default" account from the counting, which is 15.82 GB itself!



Now, I have contacted with my hosting provider, who after contacting with cPanel told me that (quoted from my hosting provider support ticket reply) "this is intentional / known behavior and unfortunately not something they consider a bug" - here "they" = cPanel.

My hosting provider also quoted from cPanel:
"If you convert your mailboxes to the maildir format, you may notice a discrepancy between reported and actual disk usage for the default email account. This occurs because a maildir mailbox adds the total default email account usage to the INBOX directory usage of every other email address on the cPanel account. It does not add the subfolders for any other mailbox."

When I have asked my hosting provider about any recent conversion to maildir format, they have mentioned that they have always used "maildir", and this is what my hosting provider have mentioned:
"We've always ran maildir because if we did not then the ability to restore individual email messages would no longer be possible. cPanel now has a case open CPANEL-26525 to improve the reporting of the disk usage pf email so it's not misleading/incorrect but there is no estimate on when this will be fixed."

My hosting provider gave me a SSH command (du -h --max-depth=1 mail/) to check the disk usage of mail directory. Unfortunately, for my end user, who is using this cPanel account for only email purpose, is not usable and/or not user friendly.

Now, I need a solution for this problem. Since my hosting provider could not solve this, even after contacting cPanel support, I have no other option but to open a thread here. I am paying my hosting provider, who is paying cPanel, for a end product, which we all expect bug free/usable at it's best. I understand that there will be some bugs in a software. But what I am facing here is not any minor bug, and is kind of unusable for an finished product, in my opinion.

If my hosting provider is following this thread, to them: I am not here to complain, but to look for a solution.

I am sorry for the long post, and if my post is not clear or doesn't make any sense. And thanks for reading it in full.

I am unable to edit the original post. So here are the missing information:
cPanel Version: 78.0.21, running on CLOUDLINUX 7.6 kvm

 

[cPanelMichael]

Hello @thesufi,

The issue you have described looks to relate to internal case CPANEL-26448. You can find more information about this case on the following forums thread:

In Progress - [CPANEL-26448] Default Email Account Showing Incorrect Usage

Let me know if this helps.

Thank you.

 

[thesufi]

Hello @cPanelMichael

Thanks for your reply. I have checked the CPANEL-26448 and your provided thread. However, I can confirm that it doesn't matched with my scenario entirely. Only default email disk usage part matches. On my case, every email accounts are providing false disk usage data. That's the major problem. If it was only "default" account, I could have easily avoid it. But every email accounts are showing wrong disk usage data. Kindly read my post again and match with my attached images.

Thanks.

 

[cPanelMichael]

Hello @thesufi,

We'd need to take a look at your system to see why the disk space reported for your virtual email accounts are inaccurate. Can you ask your hosting provider to open a support ticket with us on your behalf so we can take a closer look? You can ask them for the ticket number they receive from us and post it here once they let you know what it is.

Thank you.

 

[cPanelMichael]

Hello @thesufi,

To update, the disk space usage discrepancy for the individual email accounts is associated with Compress Messages feature documented on the link below:

Mailserver Configuration - Version 78 Documentation - cPanel Documentation

This is related to the following notice in the cPanel UI:

Email account storage may occupy less space on the disk if you use compression or hard-link optimizations designed to save space. Email account storage does not include the metadata that the system uses to store email.

The following feature request is open to provide a supported method of compressing existing email (this would address the issue you have noticed):

Ability to compress existing email

I encourage you to vote and add feedback to this feature request if this is something you'd like to see added to cPanel & WHM in the future.

Thank you.

 

[thesufi]

Hello @cPanelMichael,

Yes, this seems like the problem. This feature request has been there for around 10 months now. So, I don't think this is going to see daylight anytime soon. Any possible workaround? I can deploy a quick VPS and do some work, if there is any hope to fix this! Because the disk usage data end user is getting is very misleading and causing problem.

Thanks.

 

[mathx]

Some of the statements here are incorrect. We're on a ZFS filesystem which compresses automatically, and we see a 7775MB report for the user, however because of the email reported usage it's putting the user over 11,000MB.

I compared du -xm . | sort -n | tail -1 (which reports 7775MB of actual disk) vs

$find . -ls -type f | awk '{a+=$7} END {print a/1000/1000;}'
8171.7

you can see this isnt very far off, vs the "email disk usage of 11,600MB" in email accounts reported under the disk usage tab of the user's cpanel panel.

Note what i've done here is tallied the actual LISTED disk usage from ls of the file, which is the correct file size uncompressed. So compression isnt the issue here specifically. Something is completely wrong in the disk usage calculation for email.

Where is the script that cpanel uses to calculate this so that I may investigate and see what it is doing internally, and possibly modify it for our uses?

 

[cPanelMichael]

Any possible workaround? I can deploy a quick VPS and do some work, if there is any hope to fix this! Because the disk usage data end user is getting is very misleading and causing problem.

The workaround noted on the following feature request comments remains applicable:

Ability to compress existing email comment 1
Ability to compress existing email comment 2

Where is the script that cpanel uses to calculate this so that I may investigate and see what it is doing internally, and possibly modify it for our uses?

Here's the path to the script that's used to generate the maildirsize files:

/usr/local/cpanel/scripts/generate_maildirsize

Feel free to open a support ticket if you'd like us to take a closer look and investigate where the discrepancy is coming from.

Thank you.

 

[whipworks]

Hi @cPanelMichael,

We are also having the same quota issue not being read right. Can we also open a ticket? We are on an LXC container on a ZFS backend storage subsystem. Thanks!

 

[cPanelMichael]

Hello @whipworks,

Yes, feel free to open a support ticket and post the ticket number here. Thanks!

 

[mathx]

I work with @whipworks, so it was the same issue. How was this solved so I know what to do for our other systems?

 

[cPanelMichael]

I work with @whipworks, so it was the same issue. How was this solved so I know what to do for our other systems?

Can you share the ticket number that was opened? Or, are you referring to the ticket opened by the thread creator?

Thank you.

 

[whipworks]

Can you share the ticket number that was opened? Or, are you referring to the ticket opened by the thread creator?

Thank you.

Ticket #12315117

 

[cPanelMichael]

Ticket #12315117

The issue reported in this ticket was determined to relate to case CPANEL-26448:

Pending Publication - [CPANEL-26448] Default Email Account Showing Incorrect Usage

The workaround provided in the support ticket was to execute the /scripts/generate_maildirsize command and move aside the email account cache file from the affected user's /home/username/.cpanel/ directory. EX:

/scripts/generate_maildirsize --confirm --allaccounts
mv -v /home/username/.cpanel/email_accounts.json /home/username/.cpanel/backup.email_accounts.json.backup1

Thank you.