What is a Kernel Level Audit Trail?

Monday, July 11, 2011

Jamie Adams


Many security experts believe that audit trails and logs are the key to identifying an ongoing cyber attack as well as providing forensic data following an attack.

However, few people understand how audit records are generated or the difference between a kernel level audit trail and an application event log.

It is critical to configure auditing and logging mechanisms to capture the right data in an optimal manner and to safeguard the data to prevent it from being modified.

Like most operating systems, the main component of Linux is its kernel. It acts as a bridge between software applications and hardware and as such is responsible for managing the system's resources. Generically speaking, these resources include the central processing units (CPU), memory, and any Input/Output (I/O) devices present in the computer.

Running software applications are considered processes by the kernel. Generally, a run-time library sits between applications and the operating system such as libc and glibc. The application calls functions in these libraries – which act as wrappers to system calls. Ultimately, a system call (syscall) is how a process requests a service from the kernel that it does not normally have permission to run.

When we talk about kernel level audit trails, we are referring to records of system calls and file system watches. The kernel has the ability to intercept every system call (see linux/kernel/auditsc.c) and optionally record them to a file system using an auditing daemon (auditd). This auditing daemon can be configured with a set of rules on specifically what to record.

There are plenty of examples on the Internet of how to implement auditing rules but I want to show you a simple example:

# /sbin/auditctl -w /etc/hosts -p rwa
# /sbin/auditctl -l
LIST_RULES: exit,always watch=/etc/hosts perm=rwa

# /bin/grep test /etc/hosts
# /sbin/ausearch -f /etc/hosts

time->Mon Jul 11 09:53:28 2011

type=PATH msg=audit(1310392408.506:36): item=0 name="/etc/hosts"
inode=4197780 dev=08:02 mode=0100644 ouid=0 ogid=0 rdev=00:00

type=CWD msg=audit(1310392408.506:36):  cwd="/var/log/audit"

type=SYSCALL msg=audit(1310392408.506:36): arch=c000003e
syscall=2 success=yes exit=3 a0=7fff2ce9471d a1=0 a2=61f768
a3=7fff2ce92a20 items=1 ppid=20478 pid=21013 auid=1000 uid=0
gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1
ses=1 comm="grep" exe="/bin/grep"

In the above example, I set an audit rule to watch the /etc/hosts file. Then I used the grep utility to search the contents of the file for the word “test.” Finally, I searched the audit trail for entries pertaining to /etc/hosts. As you can see, it creates quite a detailed audit record.

Many security guidelines recommend that you establish watches on administration utilities. Two such command line utilities, /bin/chmod and /bin/chown are used to change a file's mode and ownership respectively.

I find it more beneficial to establish a rule to monitor the underlying system calls: chmod and chown. Keeping in mind, the aforementioned utilities use the system calls: chmod and chown. Since other utilities and programs can make similar calls, it is imperative we capture the system calls themselves rather than just the utilities.

Some guidelines go overboard in my opinion. They want you to establish a rule to record every successful file opening. Here is an example rule:

/sbin/auditctl -a exit,always -F arch=b64 -S open -F success=0

The problem I have with this rule is that the audit daemon now records every file that is opened including every shared library than an application is dynamically linked to.

When it comes to application event logs, they tend to be much more useful. If programmed correctly, applications can produce meaningful event logs which tend to be mapped to business process flow.

For example, if a Java-based web application is running on a Tomcat servlet container, the process is usually owned by a non-privileged user account. Thus kernel level audit trails of the Tomcat process accessing report files can be ambiguous.

On the other hand, Java application's event logs are far more useful if they record the identification and authentication of an internal user, the user's request to view the report, and then the report delivered to the user.

In summary, there are a couple of things to remember when configuring system auditing:

  •  First and foremost, you must balance what you need to collect with your ability to collect it. In other words, ensure you have the system resources (storage and CPU power) to generate the required audit trail without adversely impacting application performance.
  •  Consolidate your rules where possible. The system call rules are loaded into a matching engine that intercepts each syscall that all programs on the system makes. Therefore it is very important to only use syscall rules when you have to since these affect performance. The more rules, the bigger the performance hit. You can help the performance, though, by combining syscalls into one rule whenever possible.
  •  Ensure applications are configured to record the necessary events. Additionally, be sure that in-house developed applications provide plenty of details in their event logs.
  •  Protect audit trails and event logs. It is imperative that audit records are not sent over unsecure channels, unauthorized users are denied access, and audits can not be modified. 
Cross-posted from the Security Blanket Technical Blog
Possibly Related Articles:
Operating Systems
Information Security
Operating Systems Linux Security Audits Event Logging Network Security Monitoring Kernal Level Audit
Post Rating I Like this!
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.