In our datacenter, we maintain a cluster of Ubuntu servers running KVM virtualization backed by LVM storage. There are times where we’d like to move a virtual server from one host to another, but the LVM-backed storage makes this a huge chore.
In order to facilitate easier migration of virtual machines, I had to find a way to move a bunch of virtual machines from the LVM-backed storage to a centralized iSCSI cluster. I could have simply used the command-line tool “dd” to move the machines, but that would require each machine to be shut down for the entire dd process. I needed little downtime (like, 2-3 minutes max per virtual machine).
I was able to use a lvmsync to facilitate the migration. I’m going to document my step-by-step process here, and I’ll include some details here about how, exactly, I accomplished the migration.
On the iSCSI server, I created a single new iSCSI Target and a new LUN for each virtual machine I planned to move. The steps here vary greatly depending on the iSCSI server you’re using, so I’ll leave this to be an exercise for the reader.
I found that iscsiadm wasn’t installed on my physical host server, so I installed it by executing:
sudo apt-get install open-iscsi
I verified I could see the LUNs from my first physical host by executing:
/sbin/iscsiadm --mode node --portal 192.168.0.5:3260 --login
The output of the above command listed each Target available on the iSCSI host:
Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-809:iscsi.testtarget.c06c41, portal: 192.168.0.5,3260] Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-809:iscsi.testingagain.c06c41, portal: 192.168.0.5,3260] Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41, portal: 192.168.0.5,3260]
I didn’t want to stay logged in to all of these Targets, so I logged out of them with:
/sbin/iscsiadm --mode node --portal 192.168.0.5:3260 --logout
On the physical host machine, I added the iSCSI server to LibVirt by creating a new LibVirt “Storage Pool”. I first created a text file called “iscsi.xml” with the following contents. The device path was copied from the output from iscsiadm above.
<pool type='iscsi'> <name>vms</name> <source> <host name='192.168.0.5'/> <device path='iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41'/> </source> <target> <path>/dev/disk/by-path</path> </target> </pool>
Using virsh on my host machine, I listed all the LUNs (called “Volumes” in LibVirt) by executing:
sudo virsh vol-list vms
The output was:
Name Path ----------------------------------------- 220.127.116.11 /dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-0 18.104.22.168 /dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-1
First, I installed lvmsync on my machine. I copied it to /usr/local/sbin, and then logged out and in so that it would be available in my PATH.
lvmsync has three basic steps:
I created the LVM snapshot of the volume that I wanted to migrate:
lvcreate -s -n vm-hard-drive-snapshot -L 10G /dev/volgroup/vm-hard-drive
I used dd to copy the data from the snapshot to the new iSCSI target:
dd if=/dev/volgroup/vm-hard-drive bs=1M | pv -ptrb | dd of=/dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-1
I shutdown the virtual machine and then used lvmsync to sync the final hard disk changes to the iSCSI target:
lvmsync /dev/volgroup/vm-hard-drive-snapshot /dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-1
Once all the data is migrated, the final step is to configure the virtual machine to use the iSCSI disk instead of the local LVM volume. To do this, use virsh edit to modify the XML for the domain.
virsh edit vm
Locate the following stanza:
<disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/volgroup/vm-hard-drive'/> <target dev='hda' bus='ide'/> </disk>
Change the <source> stanza to point to the new iSCSI target:
<disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/disk/by-path/ip-192.168.0.5:3260-iscsi-iqn.2004-04.com.qnap:ts-809:iscsi.vms.c06c41-lun-1'/> <target dev='hda' bus='ide'/> </disk>
Start the virtual machine, and you’re all set!
I found some excellent information related to creating the LibVirt Storage Pool and configuring the virtual machine to see the remote Volume over at Daniel Berrangé’s blog.
lvmsync has some great information on their tool.
Please log in again. The login page will open in a new tab. After logging in you can close it and return to this page.