Track file changes using auditd

Most of Linux distributions comes with Linux Auditing System that makes it possible to track file changes, file accesses as well as system calls. It’s pretty useful functionality for sysadmins who wish to know who and when accessed and/or changed sensitive files like /etc/passwd, /etc/sudoers or others.

Daemon auditd that usually runs in background and starts after reboot by default logs those events into /var/log/audit.log file (or into other file if different syslog facility is specified). The common usage is to list all files which should be watched and search auditd’s logs from time to time. For example, I prefer to track any file changes into /etc/passwd, reading/writing of /etc/sudoers, executing of /bin/some/binary or just everything (read, write, attributes changes, executing) for my /very/important/file.

In order to configure that you’ll need two commands: auditctl and ausearch. First one is for configuring auditd daemon (e.g. setting a watch on a file), second one is for searching auditd logs (it’s possible to use grep against /var/log/audit.log too but ausearch command makes this task easier).

Install and start Linux Auditing System

If it happened that auditd daemon isn’t installed in your system then you can fix this by one of below commands:

sudo apt-get install audit

or

sudo yum install audit

The next step is to make sure that auditd is running, if command ps ax | grep [a]udit shows nothing then start auditd using command:

/etc/init.d/auditd start

As soon as auditd daemon is started we can start configuring it for tracking file changes using auditctl command.

Make auditd to log all file changes

auditctl -w /etc/passwd -k passwd-ra -p ra

This command will add a rule for auditd daemon to monitor file /etc/passwd file (see option -w /etc/passwd) for reading or changing the atributes (see option -p ra, where r is for read, a is for attribute). Also this command specifies filter key (-k passwd-ra) that will uniquely identify auditd records in its logs files.

Now let’s test this rule: optput the last 20 lines of /etc/passwd file and then search audit log for corresponding records

tail /etc/passwd

and then

[root@test artemn]# ausearch -k passwd-ra
—-
time->Wed Jul 4 15:17:14 2012
type=CONFIG_CHANGE msg=audit(1341407834.821:207310): auid=500 ses=23783 op=”add rule” key=”passwd-ra” list=4 res=1
—-
time->Wed Jul 4 15:17:20 2012
type=PATH msg=audit(1341407840.181:207311): item=0 name=”/etc/passwd” inode=31982841 dev=09:02 mode=0100644 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1341407840.181:207311): cwd=”/home/artemn”
type=SYSCALL msg=audit(1341407840.181:207311): arch=c000003e syscall=2 success=yes exit=3 a0=7fffecd41817 a1=0 a2=0 a3=7fffecd40b40 items=1 ppid=642502 pid=521288 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=23783 comm=”tail” exe=”/usr/bin/tail” key=”passwd-ra”

As you can see the output of second command shows that auditd has one record for filter key ‘passwd-ra’, it shows that root user (uid=0 gid=0) has read file /etc/passwd using command tail (comm=”tail” exe=”/usr/bin/tail”) at July 4, 2012 (time->Wed Jul 4 15:17:20 2012).

Utility ausearch is pretty powerful so I recommend to read output of man ausearch, in the meantime here are some useful examples:

ausearch -x /bin/grep
ausearch -x rm

This approach allows to scan auditd records for certain executable, e.g. if you’d like to see if any of watched files was deleted (or not) using command rm then you should use second command of above two.

This one will show you all records for certain UID (username).

ausearch -ui 1000


Category: Linux | Comments Off on Track file changes using auditd

Grub Fallback: Boot good kernel if new one crashes

It’s hard to believe but I didn’t know about Grub fallback feature. So every time when I needed to reboot remote server into a new kernel I had to test it on local server to make sure it won’t panic on remote unit. And if kernel panic still happened I had to ask somebody who has physical access to the server to reboot the hardware choose proper kernel in Grub. It’s all boring and not healthful – it’s much better to use Grub’s native fallback feature.

Grub is default boot loader in most Linux distributions today, at least major distros like Centos/Fedora/RedHat, Debian/Ubuntu/Mint, Arch use Grub. This makes it possible to use Grub fallback feature just out of the box. Here is example scenario.

There is remote server hosted in New Zealand and you (sitting in Denmark) have access to it over the network only (no console server). In this case you cannot afford that the new kernel makes server unreachable, e.g. if new kernel crash during boot it won’t load network interface drivers so your Linux box won’t appear online until somebody reboots it into workable kernel. Thankfully Grub can be configured to try loading new kernel once and if it fails Grub will load another kernel according to configuration. You can see my example grub.conf below:

default=saved
timeout=5
splashimage=(hd0,1)/boot/grub/splash.xpm.gz
hiddenmenu
fallback 0 1
title Fedora OpenVZ (2.6.32-042stab053.5)
root (hd0,1)
kernel /boot/vmlinuz-2.6.32-042stab053.5 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet crashkernel=auto
initrd /boot/initramfs-2.6.32-042stab053.5.img
savedefault fallback
title Fedora (2.6.35.12-88.fc14.i686)
root (hd0,1)
kernel /boot/vmlinuz-2.6.35.12-88.fc14.i686 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet
initrd /boot/initramfs-2.6.35.12-88.fc14.i686.img
savedefault fallback

According to this configuration Grub will try to load ‘Fedora OpenVZ’ kernel once and if it fails system will be loaded into good ‘Fedora’ kernel. If ‘Fedora OpenVZ’ loads well you’ll be able to reach the server over the network after reboot. Notice lines ‘default=saved’ and ‘savedefault fallback’ which are mandatory to make fallback feature working.

Alternative way

I’ve heard that official Grub fallback feature may work incorrectly on RHEL5 (and Centos 5) so there is elegant workaround (found here):

1. Add param ‘panic=5′ to your new kernel line so it looks like below:

title Fedora OpenVZ (2.6.32-042stab053.5)
root (hd0,1)
kernel /boot/vmlinuz-2.6.32-042stab053.5 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet crashkernel=auto panic=5
initrd /boot/initramfs-2.6.32-042stab053.5.img

This param will make crashed kernel to reboot itself in 5 seconds.

2. Point default Grub param to good kernel, e.g. ‘default=0′.

3. Type in the following commands (good kernel appears in grub.conf first and new kernel is second one):

# grub

Category: Linux | Comments Off on Grub Fallback: Boot good kernel if new one crashes

Add physical NIC to XenServer

If you add new physical network interface to the hardware that runs XenServer it won’t appear in XenCenter by default.

In order to attach it to VMs or change its settings you’ll need to type in a few commands to XenServer’s CLI.

1. Connect XenServer via SSH using root rights:

ssh root@192.168.10.1 -v

2. Make sure that new NIC is attached to hardware and detected by Linux, in below command’s output you can see there are three Ethernet controllers (the last one was just attached to hardware):

[root@localhost ~]# lspci | grep -i ethernet
10:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
1e:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5723 Gigabit Ethernet PCIe (rev 10)
30:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)

As you can see this NIC isn’t shown in XenCenter and below command doesn’t show its UID among detected interfaces:

root@localhost ~]# xe pif-list
uuid ( RO) : 095abcc1-4d64-7925-200f-a91d558ec872
device ( RO): eth1
currently-attached ( RO): true
VLAN ( RO): -1
network-uuid ( RO): 9da74476-ffcb-6824-25ad-62d46f34e252

Category: Linux | Comments Off on Add physical NIC to XenServer