Creating new LV and moving files from old location
Recently I had some disk issues on my server and unfortunately it went to load average above 20 (which was killing for this machine). It forced me to power down the machine completely and to investigate it further. (It turned out that issue was with space on /home partition — sharing file system between two disks turned to be not-so-good-idea after all).
Nevertheless, during those perturbations I found out, that I haven’t got fs dedicated for /var/log — it’s good practice and I decided, that it needs to be done. First I checked how much space is used in /var/log at the moment:
root@dziobak:~# du -h /var/log 52K /var/log/exim4 5.9M /var/log/nmon 4.0K /var/log/samba/cores/nmbd 4.0K /var/log/samba/cores/smbd 12K /var/log/samba/cores 652K /var/log/samba 88K /var/log/apt 4.0K /var/log/mysql 12M /var/log/installer/cdebconf 13M /var/log/installer 48K /var/log/unattended-upgrades 4.0K /var/log/iptraf 4.0K /var/log/news 4.0K /var/log/sysstat 12K /var/log/fsck 4.0K /var/log/ntpstats 2.3M /var/log/nginx 39M /var/log
Not so much, so I looked where I can move those files for a moment:
root@dziobak:~# df -PTh | grep /dev/mapper /dev/mapper/dziobak-root ext4 322M 178M 128M 59% / /dev/mapper/dziobak-home ext4 169G 148G 14G 92% /home /dev/mapper/dziobak-tmp ext4 368M 11M 339M 3% /tmp /dev/mapper/dziobak-usr ext4 8.3G 1.0G 6.9G 13% /usr /dev/mapper/dziobak-var ext4 2.8G 620M 2.1G 24% /var
/tmp sounds good. Last thing to check before performing any action — is there any application that is currently using /var/log folder:
root@dziobak:~# lsof /var/log/* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 1165 root 1w REG 254,3 483188 4039 /var/log/kern.log rsyslogd 1165 root 2w REG 254,3 381980 4099 /var/log/messages rsyslogd 1165 root 5w REG 254,3 404756 42 /var/log/syslog rsyslogd 1165 root 6w REG 254,3 55716 4072 /var/log/debug rsyslogd 1165 root 7w REG 254,3 2403297 4032 /var/log/daemon.log rsyslogd 1165 root 8w REG 254,3 251603 4051 /var/log/auth.log nmon 2202 root cwd DIR 254,3 4096 2948 /var/log/nmon
There are two (rsyslog and nmon), so I will need to stop them before move. But firstly I will prepare new LV dedicated to /var/log:
root@dziobak:~# lvcreate -L 1G --name var-log dziobak Logical volume "var-log" created
Where “-L 1G” is size of LV, “—name var-log” is name of LV and “dziobak” is name VG. Now it’s time to create some file system — I was struggling whether to choose ext2 or ext4. Finally I decided to go for ext4, but I believe it’s just a matter of taste:
root@dziobak:~# mkfs.ext4 /dev/dziobak/var-log mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 65536 inodes, 262144 blocks 13107 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 24 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.
Simple. OK, now it’s time to stop those applications, that are using /var/log, so we can move logs to some temporary place:
root@dziobak:~# service rsyslog stop Stopping enhanced syslogd: rsyslogd. root@dziobak:~# lsof /var/log/* COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nmon 2202 root cwd DIR 254,3 4096 2948 /var/log/nmon root@dziobak:~# kill -9 2202 root@dziobak:~# lsof /var/log/* root@dziobak:~#
Done. Let’s move those logs to /tmp then:
root@dziobak:~# tar -cvvzf /tmp/var-log.tar.gz /var/log *** Some long output ;) *** root@dziobak:~# ls -lh /tmp/var-log.tar.gz -rw-r--r-- 1 root root 10095778 Sep 27 17:10 /tmp/var-log.tar.gz root@dziobak:~# rm -rf /var/log
OK, now it’s time to attach new LV and bring back old log files:
root@dziobak:~# mkdir /var/log root@dziobak:~# mount /dev/mapper/dziobak-var--log /var/log root@dziobak:~# cd / root@dziobak:/# tar xvvzf /tmp/var-log.tar.gz
Last step is to add new file system to /etc/fstab:
/dev/mapper/dziobak-var--log /var/log ext4 defaults 0 2
That’s it (:
