Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!
cPanelResources

Tutorial Auditd - The Linux Auditing System

Learn about the auditd utility.

  1. cPanelResources
    Introduction

    The auditd utility can be an extremely valuable tool for monitoring what's happening on your Linux server. It is designed to integrate with the kernel and monitor system calls that the system makes.

    In this guide we will learn how to check if auditd is installed, install it if it is not, check to make sure the daemon is running, create a simple audit rule, and check the logs to see what our example rule detected.

    Note: Auditd requires access to the kernel which is not available in containers such as Virtuozzo.

    You can install the package but attempts to start it will fail due to not having direct kernel access:

    Code:
    # systemctl start auditd.service
    Job for auditd.service failed because the control process exited with error code. See "systemctl status auditd.service" and "journalctl -xe" for details.
    
    The systemd journal will show:
    Code:
    Oct 17 08:37:07 <HOSTNAME> auditd[4819]: Cannot change priority (Operation not permitted)
    Oct 17 08:37:07 <HOSTNAME> auditd[4819]: The audit daemon is exiting.
    Oct 17 08:37:07 <HOSTNAME> systemd[1]: auditd.service: control process exited, code=exited status=1
    Oct 17 08:37:07 <HOSTNAME> systemd[1]: Failed to start Security Auditing Service.
    

    Checking for, and installing, auditd

    Your cPanel server should already have a couple audit-libs* packages installed, but it will not have the actual audit package that provides the daemon that we need.

    A default installation should look like this:
    Code:
    #rpm -qa | grep audit
    audit-libs-2.8.1-3.el7_5.1.x86_64
    audit-libs-python-2.8.1-3.el7_5.1.x86_64
    
    Since it is not installed on this server, let's install it:
    Code:
    # yum install audit
    Loaded plugins: fastestmirror, universal-hooks
    Loading mirror speeds from cached hostfile
     * EA4: 208.74.121.35
     * cpanel-addons-production-feed: 208.74.121.35
     * epel: fedora-epel.mirror.lstn.net
     * remi-safe: repo1.dal.innoscale.net
    Resolving Dependencies
    --> Running transaction check
    ---> Package audit.x86_64 0:2.8.1-3.el7_5.1 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    =============================================================================================================================================================================
     Package                               Arch                                    Version                                          Repository                               Size
    =============================================================================================================================================================================
    Installing:
     audit                                 x86_64                                  2.8.1-3.el7_5.1                                  updates                                 247 k
    
    Transaction Summary
    =============================================================================================================================================================================
    Install  1 Package
    
    Total download size: 247 k
    Installed size: 621 k
    Is this ok [y/d/N]: y
    Downloading packages:
    audit-2.8.1-3.el7_5.1.x86_64.rpm                                                                                                                      | 247 kB  00:00:00
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : audit-2.8.1-3.el7_5.1.x86_64                                                                                                                              1/1
      Verifying  : audit-2.8.1-3.el7_5.1.x86_64                                                                                                                              1/1
    
    Installed:
      audit.x86_64 0:2.8.1-3.el7_5.1
    
    Complete!
    
    Now we can see the actual audit package:
    Code:
    # rpm -qa | grep audit
    audit-libs-2.8.1-3.el7_5.1.x86_64
    audit-2.8.1-3.el7_5.1.x86_64
    audit-libs-python-2.8.1-3.el7_5.1.x86_64
    
    This will install the daemon process and it will be enabled to start on server reboot, but it will not be running:
    Code:
    # systemctl status auditd.service
    ● auditd.service - Security Auditing Service
       Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
       Active: inactive (dead)
         Docs: man:auditd(8)
               https://github.com/linux-audit/audit-documentation
    
    So let's start it so it will be running when we create our rules:
    Code:
    # systemctl start auditd.service
    # systemctl status auditd.service
    ● auditd.service - Security Auditing Service
       Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
       Active: active (running) since Wed 2018-10-17 08:34:59 CDT; 1s ago
         Docs: man:auditd(8)
               https://github.com/linux-audit/audit-documentation
      Process: 27739 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUCCESS)
      Process: 27729 ExecStart=/sbin/auditd (code=exited, status=0/SUCCESS)
     Main PID: 27731 (auditd)
       CGroup: /system.slice/auditd.service
               └─27731 /sbin/auditd
    
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: lost 0
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog 0
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: enabled 1
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: failure 1
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: pid 27731
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: rate_limit 0
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog_limit 8192
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: lost 0
    Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog 1
    Oct 17 08:34:59 <HOSTNAME> systemd[1]: Started Security Auditing Service.
    
    Adding rules

    Now that we have started the service, the 'auditctl' command can be used to list/create/remove rules:
    Code:
    # auditctl -l
    No rules
    

    So let's create our first rule, which takes the format of:
    Code:
    # auditctl -w PathToFile -p Permissions -k SearchKey
    
    Let's add a rule to watch /etc/passwd for writes(w) or attribute changes(a) and set the search key to 'passwd' to make it easy to find the entries regarding the /etc/passwd file being changed:
    Code:
    # auditctl -w /etc/passwd -p wa -k passwd
    
    Now we can see the rule with auditctl and within the audit.rules file:
    Code:
    # auditctl -l
    -w /etc/passwd -p wa -k passwd
    
    Note: This rule will exist as long as the server is not rebooted. If you wish to make the rule permanent, you need to add it to the /etc/audit/rules.d/audit.rules file. This is how it will look by default:
    Code:
    # cat /etc/audit/rules.d/audit.rules
    ## First rule - delete all
    -D
    
    ## Increase the buffers to survive stress events.
    ## Make this bigger for busy systems
    -b 8192
    
    ## Set failure mode to syslog
    -f 1
    
    To add our example rule, simply add it to the bottom of the file in the exact same way that it appears when running 'auditctl -l':
    Code:
    # cat /etc/audit/rules.d/audit.rules
    ## First rule - delete all
    -D
    
    ## Increase the buffers to survive stress events.
    ## Make this bigger for busy systems
    -b 8192
    
    ## Set failure mode to syslog
    -f 1
    
    -w /etc/passwd -p wa -k passwd
    

    Searching audit logs

    The 'ausearch' command is used to search the audit logs, so we'll use the following to search for the "passwd" search key:
    Code:
    # ausearch -k passwd
    ----
    time->Wed Oct 17 08:45:23 2018
    type=CONFIG_CHANGE msg=audit(1539783923.295:77): auid=0 ses=51297 op=add_rule key="passwd" list=4 res=1
    
    That entry is showing the rule was added and it is currently the only entry since /etc/passwd hasn't been modified since we created this rule.

    Now let's add a cPanel account to the server so we can see how that operation modifies /etc/passwd:
    Code:
    # ausearch -i -k passwd
    ----
    type=CONFIG_CHANGE msg=audit(10/17/2018 08:45:23.295:77) : auid=root ses=51297 op=add_rule key=passwd list=exit res=yes
    ----
    type=PROCTITLE msg=audit(10/17/2018 09:04:15.572:155) : proctitle=whostmgr5 - wwwacct
    type=PATH msg=audit(10/17/2018 09:04:15.572:155) : item=1 name=/etc/passwd inode=1308128 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
    type=PATH msg=audit(10/17/2018 09:04:15.572:155) : item=0 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
    type=CWD msg=audit(10/17/2018 09:04:15.572:155) :  cwd=/usr/local/cpanel/whostmgr/docroot
    type=SYSCALL msg=audit(10/17/2018 09:04:15.572:155) : arch=x86_64 syscall=open success=yes exit=10 a0=0x4ffa230 a1=O_RDWR|O_CREAT a2=0644 a3=0x7f9899e7cedd items=2 ppid=32113 pid=32209 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=whostmgr5 - www exe=/usr/local/cpanel/whostmgr/bin/whostmgr5 key=passwd
    ----
    type=CONFIG_CHANGE msg=audit(10/17/2018 09:04:15.573:156) : auid=unset ses=unset op=updated_rules path=/etc/passwd key=passwd list=exit res=yes
    ----
    type=PROCTITLE msg=audit(10/17/2018 09:04:15.573:157) : proctitle=whostmgr5 - wwwacct
    type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=4 name=/etc/passwd inode=1308152 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
    type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=3 name=/etc/passwd inode=1308128 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
    type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=2 name=/etc/passwd.work.d21f6ce1 inode=1308152 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
    type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=1 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
    type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=0 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
    type=CWD msg=audit(10/17/2018 09:04:15.573:157) :  cwd=/usr/local/cpanel/whostmgr/docroot
    type=SYSCALL msg=audit(10/17/2018 09:04:15.573:157) : arch=x86_64 syscall=rename success=yes exit=0 a0=0x580a770 a1=0x6a583f a2=0x7f989a106980 a3=0x3 items=5 ppid=32113 pid=32209 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=whostmgr5 - www exe=/usr/local/cpanel/whostmgr/bin/whostmgr5 key=passwd
    
    The "-i" will print the actual named system call, rather than the number, making it far easier to know what system calls are taking place. The entries we're most concerned with here are the second and last entries. Within those, specifically we're looking mostly at the "syscall" and "exe" values.

    The second entry shows that /etc/passwd was opened using the "open" system call ("syscall=open") to the "whostmgr5" executable file ("exe=/usr/local/cpanel/whostmgr/bin/whostmgr5")

    A system call is how a program interacts with the kernel, so these are important to look at because they tell us whether a program is requesting to, for example, open a file, write to a file, read from a file, etc.

    The last entry then shows that there was a "rename" call also by the same WHM process, and looking a little above that, you can see that a temporary file had been created "name=/etc/passwd.work.d21f6ce1" then it was renamed to /etc/passwd, thus replacing the old file with the new one.

    Knowing that this change was done by the WHM process, one could then check cPanel's access log (/usr/local/cpanel/logs/access_log) to find the IP address and username that made the request that caused this file to be written to:
    Code:
    10.7.65.81 - root [10/17/2018:14:03:52 -0000] "POST /cpsess3914673904/scripts5/wwwacct HTTP/1.1" 200 0 "https://<HOSTNAME>:2087/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6)
    AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "s" "-" 2087
    
    (The time difference comes from the access_log being written to in UTC and the audit logs using the system's timezone which is set to US/Central time so there's a 5 hour difference)

    So we now know that on the morning of October 17th, someone at 10.7.65.81 logged into WHM as root and made a POST request to wwwacct which modified the /etc/passwd file. In this specific instance, one could then reference the /var/cpanel/accounting.log file and see the account that was created:
    Code:
    Wed Oct 17 09:04:16 2018:CREATE:root:root:auditdexample.tld:172.16.0.93:auditdexample
    
    Deleting rules

    Lastly, once you've gathered whatever information you're after with your audit rule, you should delete this watch rule using the same format that was used to add it, except replacing -w with -W:
    Code:
    # auditctl -W /etc/passwd -p wa -k passwd
    # auditctl -l
    No rules
    
    The above command will remove the rule from the current running audit session. You will also need to remove the entry from /etc/audit/rules.d/audit.rules if you added it there.

    It is important to delete the rules once they've served their purpose, as audit logs can potentially grow very large in size if they're left to run for extended periods of time.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice