Home>
Technology>
Migrating from LVM to iSCSI with (almost) No Downtime

Migrating from LVM to iSCSI with (almost) No Downtime

In our datacenter, we maintain a cluster of Ubuntu servers running KVM virtualization backed by LVM storage. There are times where we’d like to move a virtual server from one host to another, but the LVM-backed storage makes this a huge chore.

In order to facilitate easier migration of virtual machines, I had to find a way to move a bunch of virtual machines from the LVM-backed storage to a centralized iSCSI cluster. I could have simply used the command-line tool “dd” to move the machines, but that would require each machine to be shut down for the entire dd process. I needed little downtime (like, 2-3 minutes max per virtual machine).

I was able to use a lvmsync to facilitate the migration. I’m going to document my step-by-step process here, and I’ll include some details here about how, exactly, I accomplished the migration.

Create iSCSI Target and LUNs

On the iSCSI server, I created a single new iSCSI Target and a new LUN for each virtual machine I planned to move. The steps here vary greatly depending on the iSCSI server you’re using, so I’ll leave this to be an exercise for the reader.

View LUNs from Virtual Machine Host

I found that iscsiadm wasn’t installed on my physical host server, so I installed it by executing:

sudo apt-get install open-iscsi

I verified I could see the LUNs from my first physical host by executing:

/sbin/iscsiadm --mode node --portal 192.168.0.5:3260 --login

The output of the above command listed each Target available on the iSCSI host:

Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-809:iscsi.testtarget.c06c41, portal: 192.168.0.5,3260]
Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-809:iscsi.testingagain.c06c41, portal: 192.168.0.5,3260]
Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41, portal: 192.168.0.5,3260]

I didn’t want to stay logged in to all of these Targets, so I logged out of them with:

/sbin/iscsiadm --mode node --portal 192.168.0.5:3260 --logout

Add Storage Pool to LibVirt

On the physical host machine, I added the iSCSI server to LibVirt by creating a new LibVirt “Storage Pool”. I first created a text file called “iscsi.xml” with the following contents. The device path was copied from the output from iscsiadm above.

<pool type='iscsi'>
  <name>vms</name>
  <source>
    <host name='192.168.0.5'/>
    <device path='iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41'/>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
  </target>
</pool>

List Volumes in Storage Pool

Using virsh on my host machine, I listed all the LUNs (called “Volumes” in LibVirt) by executing:

sudo virsh vol-list vms

The output was:

Name Path
 -----------------------------------------
 11.0.0.0 /dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-0
 11.0.0.1 /dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-1

Syncing with lvmsync

First, I installed lvmsync on my machine. I copied it to /usr/local/sbin, and then logged out and in so that it would be available in my PATH.

lvmsync has three basic steps:

Create an LVM snapshot
Use dd to copy the snapshot to the new drive
Use lvmsync to do a final sync (with the virtual machine powered off)

I created the LVM snapshot of the volume that I wanted to migrate:

lvcreate -s -n vm-hard-drive-snapshot -L 10G /dev/volgroup/vm-hard-drive

I used dd to copy the data from the snapshot to the new iSCSI target:

dd if=/dev/volgroup/vm-hard-drive bs=1M | pv -ptrb | dd of=/dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-1

I shutdown the virtual machine and then used lvmsync to sync the final hard disk changes to the iSCSI target:

lvmsync /dev/volgroup/vm-hard-drive-snapshot /dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-1

Modifying Virtual Machine to use iSCSI instead of LVM

Once all the data is migrated, the final step is to configure the virtual machine to use the iSCSI disk instead of the local LVM volume. To do this, use virsh edit to modify the XML for the domain.

virsh edit vm

Locate the following stanza:

<disk type='file' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/dev/volgroup/vm-hard-drive'/>
  <target dev='hda' bus='ide'/>
 </disk>

Change the <source> stanza to point to the new iSCSI target:

<disk type='file' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-1'/>
  <target dev='hda' bus='ide'/>
</disk>

Start the VM

Start the virtual machine, and you’re all set!

About Travis Austin