Max out inodes issue

[PPNSteve]

Disk usage warning

The filesystem “/usr” mounted at “/usr” reached “critical” status because you currently use 99.74% of its available inodes.

[B]Notification Type: [/B]critical 

[B]Server:[/B] crz01.[redacted]

[B]Primary IP Address:[/B]

[redacted]

[B]Filesystem: [/B]/usr

[B]Mount Point: [/B]/usr

[B]Percent Used: [/B]99.74%

[B]Inode Usage:[/B]
Used: 1,045,922
Available: 2,654
Total: 1,048,576

[B]ChkServd Version: [/B]17.0

and so on.. now how does a default cPanel install (on centOS 6.x) with a single site running on it max out inodes in /usr?

More importantly, how can I fix it?

 

[Eminds]

/usr folder has sub folders which contains log files and important cpanel stuff. Check the size of the folders accordingly with du -sh * command and it will give you an idea of what is consuming the disk space.

 

[PPNSteve]

mod_sec audit logs was the cause.. one per day in a folder since it was installed/enabled. removed all but the latest and went from 100% inode usage to 17% in /usr

 

[cPanelMichael]

mod_sec audit logs was the cause.. one per day in a folder since it was installed/enabled. removed all but the latest and went from 100% inode usage to 17% in /usr

Hello,

Is this system using EasyApache 3 or EasyApache 4? Can you provide an example of how the directory structure looked so we can verify the logs were being rotated?

Thank you.

 

[PPNSteve]

Hi,

Yeah, Its using EA 3.

here is a copy/pasta from my ssh shell session:

root@crz01 [~]# cd  /usr/local/apache/logs/modsec_audit
root@crz01 [/usr/local/apache/logs/modsec_audit]# ls -al
total 64
drwx-wx-wt 5 root     root      4096 Dec 19 07:08 ./
drwxr-xr-x 4 root     root      4096 Dec 19 12:00 ../
drwxr-x--- 3      504      515  4096 Sep 15  2015 crzaccou/
-rw-r--r-- 1      504      515     1 Sep 15  2015 crzaccou.offset
drwxr-x--- 3      505      516  4096 Sep 15  2015 crzadmn/
-rw-r--r-- 1      505      516     1 Sep 15  2015 crzadmn.offset
-rw-r--r-- 1 crzscore crzscore     1 Nov 21  2016 crzscore.offset
drwxr-x--- 4 nobody   nobody   32768 Dec 19 02:05 nobody/
root@crz01 [/usr/local/apache/logs/modsec_audit]# cd crzadmn/
root@crz01 [/usr/local/apache/logs/modsec_audit/crzadmn]# ls -al
total 12
drwxr-x--- 3  505  516 4096 Sep 15  2015 ./
drwx-wx-wt 5 root root 4096 Dec 19 07:08 ../
drwxr-x--- 4  505  516 4096 Sep 15  2015 20150915/
root@crz01 [/usr/local/apache/logs/modsec_audit/crzadmn]# cd ../nobody
root@crz01 [/usr/local/apache/logs/modsec_audit/nobody]# ls -al
total 56
drwxr-x---   4 nobody nobody 32768 Dec 19 02:05 ./
drwx-wx-wt   5 root   root    4096 Dec 19 07:08 ../
drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 20171218/
drwxr-x---  83 nobody nobody  4096 Dec 19 07:05 20171219/
root@crz01 [/usr/local/apache/logs/modsec_audit/nobody]# cd 20171218/
root@crz01 [/usr/local/apache/logs/modsec_audit/nobody/20171218]# ls -al
total 1188
drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 ./
drwxr-x---   4 nobody nobody 32768 Dec 19 02:05 ../
drwxr-x---   2 nobody nobody  4096 Dec 18 00:00 20171218-0000/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:05 20171218-0005/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:10 20171218-0010/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:15 20171218-0015/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:19 20171218-0019/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:20 20171218-0020/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:25 20171218-0025/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:30 20171218-0030/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:35 20171218-0035/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:40 20171218-0040/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:45 20171218-0045/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:49 20171218-0049/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:50 20171218-0050/
drwxr-x---   2 nobody nobody  4096 Dec 18 00:55 20171218-0055/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:00 20171218-0100/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:05 20171218-0105/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:09 20171218-0109/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:10 20171218-0110/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:15 20171218-0115/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:20 20171218-0120/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:25 20171218-0125/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:30 20171218-0130/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:35 20171218-0135/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:40 20171218-0140/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:45 20171218-0145/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:50 20171218-0150/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:51 20171218-0151/
drwxr-x---   2 nobody nobody  4096 Dec 18 01:55 20171218-0155/
[shortened]
drwxr-x---   2 nobody nobody  4096 Dec 18 16:30 20171218-1630/
drwxr-x---   2 nobody nobody  4096 Dec 18 16:34 20171218-1634/
drwxr-x---   2 nobody nobody  4096 Dec 18 16:35 20171218-1635/
drwxr-x---   2 nobody nobody  4096 Dec 18 16:40 20171218-1640/
drwxr-x---   2 nobody nobody  4096 Dec 18 16:45 20171218-1645/
root@crz01 [/usr/local/apache/logs/modsec_audit/nobody/20171218]# cd 20171218-1645/
root@crz01 [/usr/local/apache/logs/modsec_audit/nobody/20171218/20171218-1645]# ls -al
total 20
drwxr-x---   2 nobody nobody  4096 Dec 18 16:45 ./
drwxr-x--- 287 nobody nobody 12288 Dec 18 16:45 ../
-rw-r-----   1 nobody nobody  2080 Dec 18 16:45 20171218-164501-Wjg23UpWraoAAAEufK0AAAAD
root@crz01 [/usr/local/apache/logs/modsec_audit/nobody/20171218/20171218-1645]#

etc...

note that this is under the 'nobody' apache account. the end user account(s) don't seem to be having the issue.. (are rotated)
it does that for every day in its dir.

 

[cPanelMichael]

Hello,

It's possible this is isolated to EasyApache 3. Is there anything in-particular we can do to help you convert the system to EasyApache 4? Please see:

EasyApache 3: It's been a long road, but it will be time to say goodbye soon.

Thank you.

 

[PPNSteve]

Not sure.. getting a bunch of warnings when the pre-flight checks are run:
/snag.gy/yP6nGK.jpg

safe to ignore or ??

 

[cPanelMichael]

Hello,

Those are non-fatal errors, and are displayed to note a difference in the EA4 profile compared to the EA3 profile:

SOLVED - Migrate EA3 to EA4 issues

The primary notice I see relates to suPHP, as it is not compatible with the mpm_itk or the mod_ruid2 Apache module. Thus, the EasyApache 4 conversion script is letting you know it will not install suPHP since you are using Mod_Ruid2. You can review the following documents to determine which PHP handlers and MPM options are supported:

Multi-Processing Modules - MPMs - EasyApache 4 - cPanel Documentation
PHP Handlers - EasyApache 4 - cPanel Documentation

Let us know of any additional questions.

Thank you.

 

[PPNSteve]

well looks like that server migrated just fine (so far) then again it's only hosting a single site.
Will update if anything strange happens or if that inode / log issue returns.

 

[Macs R We]

The OP looks satisfied, so I hope I'm not hijacking this thread... I have a similar problem:

The filesystem “/var/lib/snapd/snap/emacs/966” mounted at “/var/lib/snapd/snap/emacs/966” reached “critical” status because you currently use 100% of its available inodes.

I've verified this is indeed a mountpoint, though why and what all the consequences of this are are lost on me.

I don't believe any files (like logs) get added to this hierarchy at runtime, I think it's just the result of my snap installation of emacs. I've run the du -sh over it and the big components seem to be runtime libraries and lisp.

What, if anything, do I do about this?

 

[cPRex]

@Macs R We - that's definitely an odd mount path. You may want to speak with your host to see why that is setup like that, as I haven't come across such a path before.

 

[Macs R We]

@Macs R We - that's definitely an odd mount path. You may want to speak with your host to see why that is setup like that, as I haven't come across such a path before.

My hosting provider isn't going to know what's going on. I am the admin, and I had a fresh machine recently on which the Emacs was gone, so I did a web search for installing Emacs on CentOS. I found instructions from what seemed to be a reputable source (Install GNU Emacs on CentOS using the Snap Store | Snapcraft) for doing it with a service called snap. So I just ran the two or three commands on the webpage, and that's what I ended up with. It works great as being able to use Emacs, but about a month later I'm getting these messages, and I have no idea what's going on under the hood here. I didn't customize anything, the installation was pretty simple and looked a lot like services I had used before on other UNIX platforms.

 

[ffeingol]

I don't know if it's going to help much, but can you cd into /var/lib/snapd/snap/emacs/966 (so you are in that mount point) and then run:

df -i .

That at least will show you how many inodes are available and how many are used.

 

[Macs R We]

I don't know if it's going to help much, but can you cd into /var/lib/snapd/snap/emacs/966 (so you are in that mount point) and then run:

df -i .

That at least will show you how many inodes are available and how many are used.

Thanks for the reference.

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop1      24398 24398     0  100% /var/lib/snapd/snap/emacs/966

I have no idea what that is, so I stat-ed it:

  File: âloop1â
  Size: 0             Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d    Inode: 12362       Links: 1     Device type: 7,1
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2020-12-01 20:35:54.284724168 -0700
Modify: 2020-12-01 20:35:35.720000000 -0700
Change: 2020-12-01 20:35:35.720000000 -0700
Birth: -

I'm out of my depth here, especially since I did not set up this VM.

 

[cPRex]

That's definitely something to speak with the host about. I have no idea why someone would want or need a separate partition for emacs tools.

 

[Macs R We]

That's definitely something to speak with the host about. I have no idea why someone would want or need a separate partition for emacs tools.

Well, the host didn't have any input to this as far as I know. It must have been done by the Snap installation tool. I guess I'll ask there.

 

[Macs R We]

The snapcraft folks explained this 100% to my satisfaction (below). So I can either find a control in WHM to tell it not to run that particular check on that filesystem, or at worst, just ignore the warning messages.

Snaps are distributed as compressed squashfs filesystems, rather than being extracted, they are mounted at runtime, this applies to all snaps and not just the Emacs snap.
This is similar to the approach AppImage takes, and certain application installers on MacOS. It provides some benefits, and to some people some disadvantages in that it appears in ‘lsblk’ output and similar, and similarly this is why your software is concerned that it sees a filesystem that is 100 percent full.
But it’s supposed to be 100 percent full anyway, because being less than fully at capacity would just be wasted space, the automatic updates will replace the mountpoint rather than add to it, so theres nothing to worry about.
Tl:;Dr it’s working as designed, you can ignore the error in this case and if you can exclude a directory from being checked, safely silence it by excluding /var/lib/snapd/snap

 

[Macs R We]

This is definitely what's going on here... but I can't find any configuration tweaker in WHM to tell the system not to scan those pathnames. Is there one?

 

[cPRex]

There isn't an option to exclude a partition just yet, but we actually have a feature request open for your situation here:

https://features.cpanel.net/topic/disk-warning-exclude

I'd recommend adding your vote to that page as I just did a few minutes ago.

 

[Macs R We]

Thanks, I just did this.