Using LVM snapshots for backup and restore of filesystems

This post really is about using LVM (Logical Volume Manager; an abstraction layer for disk devices) snapshots. A snapshot is a frozen image of a logical volume, which simply means “filesystem”. It’s not really “frozen”, LVM2 snapshots are read/write by default. But you can freeze a filesystem in time with a LVM snapshot.

The background of this really is Exadata (computing node) and upgrading, but has nothing unique to Exadata. So don’t let this bother you. But the idea of using LVM snapshots popped up when dealing with Exadata computing nodes and upgrades.

First of all: LVM is in development, which mean different Linux versions have different LVM options available to them. I am using the Exadata X2 Linux version: RHEL/OL 5u7 x86_64. I guess OL6 has more and more advanced features inside LVM, but with X2, OL5u7 is what I have to use. So the steps in this blogpost are done with this version. Any comments are welcome!

Second: if you want to experiment with this: most people allocate all space in the volume group to logical volumes upfront. A snapshot is a COW (Copy on Write) copy of a logical volume. This means a snapshot starts off with zero extra bytes (source and snapshot are equal), and grows as the source gets modified. This means you need to have free/available space in the volume group to facilitate the snapshot.

Then there is another caveat: the ‘/boot’ filesystem cannot be in LVM, so is a normal partition on most systems (also on Exadata). This means snapshots do not help if you want a backup of that filesystem. You need to use another trick.

Okay, here we go: you have a large modification upcoming and want to be able to restore your system to this moment in time.

1. Backup /boot filesystem

[root@localhost ~]# df /boot 
Filesystem           1K-blocks  Used Available Use%  
Mounted on /dev/sda1    248895 24037    212008  11% /boot
[root@localhost ~]# umount /boot
[root@localhost ~]# dd if=/dev/sda1 of=dev_sda1_backup
[root@localhost ~]# mount /boot

2. Create snapshot of logical volumes
In this example it’s only the root filesystem (which is a bit special, because this filesystem is set in grub.conf for the bootloader, and in /etc/fstab).

[root@localhost ~]# lvdisplay -v /dev/vg00/lv_root 
Using logical volume(s) on command line 
/dev/hdc: open failed: No medium found 
--- Logical volume --- 
LV Name /dev/vg00/lv_root 
VG Name vg00 
LV UUID wutwln-ffdB-QRlg-1LgL-XqKB-glvn-OpCowW 
LV Write Access read/write 
LV Status available # open 1 
LV Size 3.91 GB 
Current LE 125 
Segments 1 
Allocation inherit 
Read ahead sectors auto 
- currently set to 256 
Block device 253:0 

lvdisplay displays the properties of the logical volume to be snapshotted, I use this to spot the number of LE’s.

[root@localhost ~]# lvcreate -l 125 -s /dev/vg00/lv_root -n lv_root_snap 
/dev/hdc: open failed: No medium found 
Logical volume "lv_root_snap" created

The system now has been backed up in a way that we can revert to this situation.

Next would be to do the O/S upgrade, software upgrade or whatever. It goes horribly wrong, and you need to restore the system to the previous situation.

1. Rename current logical volume
In this case, I rename my lv_root logical volume to lv_root_old:

[root@localhost ~]# lvrename /dev/vg00/lv_root /dev/vg00/lv_root_old
/dev/hdc: open failed: No medium found 
Renamed "lv_root" to "lv_root_old" in volume group "vg00"

The logical volume (which we currently use) has been renamed.

2. Create new lv_root
This is the logical volume we are going to use to put the snapshot in.

[root@localhost ~]# lvcreate -l 125 -n lv_root /dev/vg00
Logical volume "lv_root" created

3. Populate the new lv_root with the snapshot contents

[root@localhost ~]# dd if=/dev/vg00/lv_root_snap of=/dev/vg00/lv_root 	
8192000+0 records in 
8192000+0 records out 
4194304000 bytes (4.2 GB) copied, 93.0858 seconds, 45.1 MB/s

4. Restore the /boot filesystem

[root@localhost ~]# umount /boot
[root@localhost ~]# dd if=dev_sda1_backup of=/dev/sda1
[root@localhost ~]# mount /boot

5. Reboot

[root@localhost ~]# shutdown -r now

This sequence of events enabled me to restore my system to a post-modification situation. Of course you should test this very thoroughly for your own situation, but this offers an elegant way, which has little external dependencies.

Post restore/cleaning up:
When the system is reverted to its old situation, we are left with a logical volume and a snapshot which probably are not of use anymore. These can be cleaned up the following way:

[root@localhost ~]# lvremove /dev/vg00/lv_root_old 
Do you really want to remove active logical volume lv_root_snap? [y/n]: y
Logical volume "lv_root_snap" successfully removed 
Do you really want to remove active logical volume lv_root_old? [y/n]: y 
Logical volume "lv_root_old" successfully removed
  1. great tutorial! I was just wondering what are the differences between using this method and using lvconvert –merge ?

    Thank you

  2. John said:

    Nice, I messed up system following your advice. You should tell folks to consider changing UUID of the either original or the snapshot, otherwise wrong one may be mounted, since both have same UUID and snapshot is being mounted by default (instead of origin), what would make you have corrupted snapshot. Same behavior seems to occur on Ubuntu and OpenSUSE, but this behavior may change in the future to the opposite, thus changing UUID should be a good practice.

  3. joe said:

    Well, dd’ing an LVM snapshot volume is useless because the snapshot volume only contains the COW copies from changes to the original LV! (If there were no changes to the original root LVM, then the snapshot would essentially be empty. So you want me to copy a COW repository, which potentially contains only possible changes back as an original LV? What are you thinking? I tried this experiment on a few servers and as expected, fails miserably. I believe this is one of those situations where somebody just publishes b.s. without really testing it or knowing what they are talking about.

    • Joe, your writing style is a bit bullying, are you aware of that?

      I did test this method, but the lack of details in your reply makes it hard to reply.

      A summary of this post is:
      A snapshot contains a pre-upgrade state.
      If the upgrade fails, the lv and snapshot differ, because the lv is post-upgrade.
      I am aware what a snapshot is, but it’s about the state of the filesystem in the snapshot.
      The current lv is renamed, and the snapshot contents are copied from the snapshot to a new lv.

      A potential problem is the snapshot contains a mounted filesystem.

      If you are able to produce a reply with technical content (beyond ‘doesn’t work’) I am happy to discuss this with you.

      • Joe White said:

        “The current lv is renamed, and the snapshot contents are copied from the snapshot to a new lv”

        Did you actually even check this works? I doubt it. The “snapshot contents” are only the changed files; not the whole filesystem.

        And the “new lv” is completely blank.

        As joe pointed out, you are advising people to copy only the changed files to a blank volume and then boot from that volume. This will end in disaster. You should’ve checked this works before posting it.

      • Joe, let me return the question: did you try this (carefully done the steps without any deviation or own interpretation)?

        Just to be on the same page with you: an LVM snapshot is an image of the logical volume it has been made of. The LVM things work on the LVM level, so it has no concept of a filesystem (so please do not mention filesystem when talking about LVM concepts). The snapshot and the logical volume share the same blocks. When either the snapshot or the logical volume block is changed, the changed block is copied into a new block, and that block is linked for that logical volume image, so changing something in the logical volume doesn’t change the snapshot (or vice versa). But both represent a full logical volume (so: a full filesystem).

        If you look careful at the blogpost, you see that a whole logical volume (the snapshot) is dd’ed (this means copied using the operating system ‘dd’ utility) into the new logical volume.

        And, to answer your original question, I’ve run through this procedure multiple times. This works.

  4. jarshvor said:

    /boot can be inside an LV. Mine is. Im not sure right now if i had to do anything fancy for this to work, but I dont think so.
    YMMV depending on your distro, but I think its fairly standard atm.

    • jarshvor said:

      I think it was grub2 that enabled me to do this??

  5. Thank you for this post. I really would like to know how to backup all data on my LVM partition root. I thought that clonezilla would work but it did not. Any suggestions about lvm compatible cloning software. I’m running Linux Mint 15 Olivia KDE 64 bit. Thanks in advance.

  6. sivagnanam palanivel said:

    excellent notes…………

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: