PPNSteve

Well-Known Member
Mar 13, 2003
414
6
168
Somewhere in Ilex Forest
cPanel Access Level
Root Administrator
Twitter
Code:
Disk usage warning

The filesystem “/usr” mounted at “/usr” reached “critical” status because you currently use 99.74% of its available inodes.

[B]Notification Type: [/B]critical 

[B]Server:[/B] crz01.[redacted]

[B]Primary IP Address:[/B]

[redacted]

[B]Filesystem: [/B]/usr

[B]Mount Point: [/B]/usr

[B]Percent Used: [/B]99.74%

[B]Inode Usage:[/B]
Used: 1,045,922
Available: 2,654
Total: 1,048,576

[B]ChkServd Version: [/B]17.0
and so on.. now how does a default cPanel install (on centOS 6.x) with a single site running on it max out inodes in /usr?

More importantly, how can I fix it?
 
Last edited by a moderator:

Eminds

Well-Known Member
Nov 10, 2016
319
32
28
India
cPanel Access Level
Root Administrator
/usr folder has sub folders which contains log files and important cpanel stuff. Check the size of the folders accordingly with du -sh * command and it will give you an idea of what is consuming the disk space.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,226
463
mod_sec audit logs was the cause.. one per day in a folder since it was installed/enabled. removed all but the latest and went from 100% inode usage to 17% in /usr
Hello,

Is this system using EasyApache 3 or EasyApache 4? Can you provide an example of how the directory structure looked so we can verify the logs were being rotated?

Thank you.
 

PPNSteve

Well-Known Member
Mar 13, 2003
414
6
168
Somewhere in Ilex Forest
cPanel Access Level
Root Administrator
Twitter
Hi,

Yeah, Its using EA 3.

here is a copy/pasta from my ssh shell session:
Code:
[email protected] [~]# cd  /usr/local/apache/logs/modsec_audit
[email protected] [/usr/local/apache/logs/modsec_audit]# ls -al
total 64
drwx-wx-wt 5 root     root      4096 Dec 19 07:08 ./
drwxr-xr-x 4 root     root      4096 Dec 19 12:00 ../
drwxr-x--- 3      504      515  4096 Sep 15  2015 crzaccou/
-rw-r--r-- 1      504      515     1 Sep 15  2015 crzaccou.offset
drwxr-x--- 3      505      516  4096 Sep 15  2015 crzadmn/
-rw-r--r-- 1      505      516     1 Sep 15  2015 crzadmn.offset
-rw-r--r-- 1 crzscore crzscore     1 Nov 21  2016 crzscore.offset
drwxr-x--- 4 nobody   nobody   32768 Dec 19 02:05 nobody/
[email protected] [/usr/local/apache/logs/modsec_audit]# cd crzadmn/
[email protected] [/usr/local/apache/logs/modsec_audit/crzadmn]# ls -al
total 12
drwxr-x--- 3  505  516 4096 Sep 15  2015 ./
drwx-wx-wt 5 root root 4096 Dec 19 07:08 ../
drwxr-x--- 4  505  516 4096 Sep 15  2015 20150915/
[email protected] [/usr/local/apache/logs/modsec_audit/crzadmn]# cd ../nobody
[email protected] [/usr/local/apache/logs/modsec_audit/nobody]# ls -al
total 56
drwxr-x---   4 nobody nobody 32768 Dec 19 02:05 ./
drwx-wx-wt   5 root   root    4096 Dec 19 07:08 ../
drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 20171218/
drwxr-x---  83 nobody nobody  4096 Dec 19 07:05 20171219/
[email protected] [/usr/local/apache/logs/modsec_audit/nobody]# cd 20171218/
[email protected] [/usr/local/apache/logs/modsec_audit/nobody/20171218]# ls -al
total 1188
drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 ./
drwxr-x---   4 nobody nobody 32768 Dec 19 02:05 ../
drwxr-x---   2 nobody nobody  4096 Dec 18 00:00 20171218-0000/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:05 20171218-0005/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:10 20171218-0010/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:15 20171218-0015/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:19 20171218-0019/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:20 20171218-0020/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:25 20171218-0025/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:30 20171218-0030/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:35 20171218-0035/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:40 20171218-0040/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:45 20171218-0045/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:49 20171218-0049/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:50 20171218-0050/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:55 20171218-0055/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:00 20171218-0100/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:05 20171218-0105/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:09 20171218-0109/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:10 20171218-0110/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:15 20171218-0115/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:20 20171218-0120/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:25 20171218-0125/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:30 20171218-0130/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:35 20171218-0135/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:40 20171218-0140/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:45 20171218-0145/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:50 20171218-0150/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:51 20171218-0151/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:55 20171218-0155/
[shortened]
drwxr-x---   2 nobody nobody  4096 Dec 18 16:30 20171218-1630/
drwxr-x---   2 nobody nobody  4096 Dec 18 16:34 20171218-1634/
drwxr-x---   2 nobody nobody  4096 Dec 18 16:35 20171218-1635/
drwxr-x---   2 nobody nobody  4096 Dec 18 16:40 20171218-1640/
drwxr-x---   2 nobody nobody  4096 Dec 18 16:45 20171218-1645/
[email protected] [/usr/local/apache/logs/modsec_audit/nobody/20171218]# cd 20171218-1645/
[email protected] [/usr/local/apache/logs/modsec_audit/nobody/20171218/20171218-1645]# ls -al
total 20
drwxr-x---   2 nobody nobody  4096 Dec 18 16:45 ./
drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 ../
-rw-r-----   1 nobody nobody  2080 Dec 18 16:45 20171218-164501-Wjg23UpWraoAAAEufK0AAAAD
[email protected] [/usr/local/apache/logs/modsec_audit/nobody/20171218/20171218-1645]#
etc...

note that this is under the 'nobody' apache account. the end user account(s) don't seem to be having the issue.. (are rotated)
it does that for every day in its dir.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,904
2,226
463
Hello,

Those are non-fatal errors, and are displayed to note a difference in the EA4 profile compared to the EA3 profile:

SOLVED - Migrate EA3 to EA4 issues

The primary notice I see relates to suPHP, as it is not compatible with the mpm_itk or the mod_ruid2 Apache module. Thus, the EasyApache 4 conversion script is letting you know it will not install suPHP since you are using Mod_Ruid2. You can review the following documents to determine which PHP handlers and MPM options are supported:

Multi-Processing Modules - MPMs - EasyApache 4 - cPanel Documentation
PHP Handlers - EasyApache 4 - cPanel Documentation

Let us know of any additional questions.

Thank you.
 

Macs R We

Active Member
Mar 17, 2015
26
3
53
Arizona
cPanel Access Level
Root Administrator
The OP looks satisfied, so I hope I'm not hijacking this thread... I have a similar problem:

The filesystem “/var/lib/snapd/snap/emacs/966” mounted at “/var/lib/snapd/snap/emacs/966” reached “critical” status because you currently use 100% of its available inodes.

I've verified this is indeed a mountpoint, though why and what all the consequences of this are are lost on me.

I don't believe any files (like logs) get added to this hierarchy at runtime, I think it's just the result of my snap installation of emacs. I've run the du -sh over it and the big components seem to be runtime libraries and lisp.

What, if anything, do I do about this?
 

Macs R We

Active Member
Mar 17, 2015
26
3
53
Arizona
cPanel Access Level
Root Administrator
@Macs R We - that's definitely an odd mount path. You may want to speak with your host to see why that is setup like that, as I haven't come across such a path before.
My hosting provider isn't going to know what's going on. I am the admin, and I had a fresh machine recently on which the Emacs was gone, so I did a web search for installing Emacs on CentOS. I found instructions from what seemed to be a reputable source (Install GNU Emacs on CentOS using the Snap Store | Snapcraft) for doing it with a service called snap. So I just ran the two or three commands on the webpage, and that's what I ended up with. It works great as being able to use Emacs, but about a month later I'm getting these messages, and I have no idea what's going on under the hood here. I didn't customize anything, the installation was pretty simple and looked a lot like services I had used before on other UNIX platforms.
 
Last edited:

ffeingol

Well-Known Member
PartnerNOC
Nov 9, 2001
667
232
343
cPanel Access Level
DataCenter Provider
I don't know if it's going to help much, but can you cd into /var/lib/snapd/snap/emacs/966 (so you are in that mount point) and then run:

Code:
df -i .
That at least will show you how many inodes are available and how many are used.
 

Macs R We

Active Member
Mar 17, 2015
26
3
53
Arizona
cPanel Access Level
Root Administrator
I don't know if it's going to help much, but can you cd into /var/lib/snapd/snap/emacs/966 (so you are in that mount point) and then run:

Code:
df -i .
That at least will show you how many inodes are available and how many are used.
Thanks for the reference.

Code:
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop1      24398 24398     0  100% /var/lib/snapd/snap/emacs/966
I have no idea what that is, so I stat-ed it:

Code:
  File: âloop1â
  Size: 0             Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d    Inode: 12362       Links: 1     Device type: 7,1
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2020-12-01 20:35:54.284724168 -0700
Modify: 2020-12-01 20:35:35.720000000 -0700
Change: 2020-12-01 20:35:35.720000000 -0700
Birth: -
I'm out of my depth here, especially since I did not set up this VM.
 

Macs R We

Active Member
Mar 17, 2015
26
3
53
Arizona
cPanel Access Level
Root Administrator
The snapcraft folks explained this 100% to my satisfaction (below). So I can either find a control in WHM to tell it not to run that particular check on that filesystem, or at worst, just ignore the warning messages.

Snaps are distributed as compressed squashfs filesystems, rather than being extracted, they are mounted at runtime, this applies to all snaps and not just the Emacs snap.
This is similar to the approach AppImage takes, and certain application installers on MacOS. It provides some benefits, and to some people some disadvantages in that it appears in ‘lsblk’ output and similar, and similarly this is why your software is concerned that it sees a filesystem that is 100 percent full.
But it’s supposed to be 100 percent full anyway, because being less than fully at capacity would just be wasted space, the automatic updates will replace the mountpoint rather than add to it, so theres nothing to worry about.
Tl:;Dr it’s working as designed, you can ignore the error in this case and if you can exclude a directory from being checked, safely silence it by excluding /var/lib/snapd/snap