Clone a node/computer using KA method

1. CLONING WILL ERASE ALL CLIENT NODES DATA !

2. Clone a computer over the network

2.1. KA method

2.2. HOW it works

2.2.1. Steps
2.2.2. Needed files

2.3. Step 1: PXE, TFTP, DHCPD services

2.3.1. PXE parameters on server
2.3.2. TFTP server
2.3.3. PXE configuration
2.3.4. DHCPD configuration

3. Setup a node as a golden node

3.1. The rescue.sqfs file

3.1.1. ka-d.sh
3.1.2. replication.conf
3.1.3. fdisk_to_desc
3.1.4. gen_modprobe_conf.pl
3.1.5. ka-d-client
3.1.6. ka-d-server
3.1.7. ka_replication.sh
3.1.8. store_log.sh
3.1.9. bootable_flag.sh
3.1.10. make_initrd_grub
3.1.11. make_initrd_lilo
3.1.12. prepare_node.sh
3.1.13. send_status.pl
3.1.14. status_node.pl

4. The golden node, KA server

5. KA client node

5.1. PXE server (kamethod)
5.2. Stage1 KA method, node waiting stage2
5.3. Stage2, the duplication process
5.4. Prepare the node
5.5. PXE server to local boot

6. Step by step from scratch KA duplication

6.1. Golden node side
6.2. KA client side
6.3. Post duplication process

1. CLONING WILL ERASE ALL CLIENT NODES DATA !

!! USE WITH CARE !!

2. Clone a computer over the network

Goal of duplication is to easily deploy a computer over network without taking care of numbers of computer. In this documentation, we call golden node the node we want to clone. We can duplicate SCSI or IDE hard drive, and duplication support multiple filesystem (reiserfs, ext2, ext3, ext4, xfs, jfs). This method came from a very old project called CLIC, and was used under IGGI project, all Mandrake Clustering products, and now it is used under XtreemOS project. Now it should be available in 2010 spring, and all futur product. -

WARNING: all data on client nodes will be ERASED ! We duplicate partitions of HDD's golden node, and the process will do an fdisk command on the client node, so ALL YOUR DATA will be erased on client nodes.

2.1. KA method

With KA method you can quickly duplicate a node using a desc file describing partitions. KA method only duplicate data on partitions, so if you have 80go HDD disk, and only 10go on it, KA only duplicates 10go, and not the whole disk. KA method doesn't not support RAID software. @@ -11,15 +11,15 @@ Drawbacks:

KA method doesn't support RAID software (use dolly to do that)
all data on client nodes are erased
you need a PXE, DHDCP and TFTP server
you must re-create same partition table as the golden node (even if size can differ)
even if it has been tested, it's still an experimental method
cloning script are old, and need a full rewrite
now it's only works with the Mandriva installer (need to patch it to support a KA method)
if a node crash while doing a duplication, the duplication process stop (or became very unstable)
using fdisk to erase and re-format the HDD is not a good way to proceed
UUID support is not really done (fstab use old /dec/sdX)
you can only clone Linux filesystems (if you want to duplicate another kinf of FS, it's up to you to modify the scripts)
of course various other things !

2.2. HOW it works

2.2.1. Steps

2.2. HOW it works

2.2.1. Steps

The clone process works in three steps -

PXE boot to retrieve stage1: the computer boot on PXE mode, retrieve vmlinuz and an initrd image. The computer is in stage1 mode, and is able to get the stage2 throug KA. Network is up.
get stage2: the computer gets the stage2 with the KA method. The stage2 contains all necessary tools to recognize your hardware (the most important things is to detect your HDD and your network card), and all necessary tools/scripts to finalize the cloning process.
Duplication process: the computer auto-probes needed modules to be able to access the HDD. A basic log server is launched on the client node to be able to run command and get status of the KA duplication process. The computer reconfigure the modprobe.conf and restore the booloader (grub or lilo)

2.2.2. Needed files

PXE boot to retrieve stage1: the computer boot on PXE mode, retrieve vmlinuz and an initrd image. The computer is in stage1 mode, and is able to get the stage2 throug KA. Network is up.
get stage2: the computer gets the stage2 with the KA method. The stage2 contains all necessary tools to recognize your hardware (the most important things is to detect your HDD and your network card), and all necessary tools/scripts to finalize the cloning process.
Duplication process: the computer auto-probes needed modules to be able to access the HDD. A basic log server is launched on the client node to be able to run command and get status of the KA duplication process. The computer reconfigure the modprobe.conf and restore the booloader (grub or lilo)

2.2.2. Needed files

All needed files are available in Mandriva Linux cooker.

install/stage2/rescue.sqfs: this is the stage2 file with all needed files to detect and probe modules, and launch the third step of the duplication process. This file will be used on the golden node.
isolinux/alt0/vmlinuz: linux kernel, needed in the /var/lib/tftpboot/X86PC/linux/images/ directory of the PXE server
isolinux/alt0/all.rdz: stage1 and all needed modules and tools.

2.3. Step 1: PXE, TFTP, DHCPD services

To easily clone a computer node, we use PXE technology to boot a kernel, and an initrd image wich contains all needed modules for network and media storage. Documentation about PXE can be found here: PXE doc. Please, keep in mind setting such services can DISTURB your current network architecture. -

2.3.1. PXE parameters on server

Mandriva Linux installer supports various methods to install a computer. With PXE configuration file you can specify wich method you want to use to install your node, or add a specific option at boot prompt. Edit your default PXE configuration file to add your custom entry (/var/lib/tftpboot/X86PC/linux/pxelinux.cfg/default).

@@ -40,7 +40,7 @@ label kamethod
 	  At boot prompt no you can boot:
 	

 	  
DEFAULT local: default boot will be local one, change it with the name of a LABEL
local: boot local
kamethod: automatic mode, get stage2 through KA. Network interface is set to eth0. Auto setup the network with DHCP, and use the KA technology to launch the replication method.

-

2.3.2. TFTP server

TFTP server should be activated in /etc/xinetd.d/tftp file, and the xinetd service started.

@@ -57,7 +57,7 @@ service tftp
 	    cps= 100 2
 	    flags= IPv4
 }

2.3.3. PXE configuration

 # which interface to use
 interface=eth0
@@ -97,7 +97,7 @@ tftpdbase=/
 # domain=guibland.com
 domain=

2.3.4. DHCPD configuration

IE of an /etc/dhcpd.conf configuration file. Change IPADDR_TFTP with the IP address of the TFTP serrver, and the NET value. Don't forget to adjust the domain-name and the domain-name-servers.

@@ -183,7 +183,7 @@ subnet NET.0 netmask 255.255.255.0 {
   range NET.30 NET.40;
   }
 }

3. Setup a node as a golden node

3.1. The rescue.sqfs file

3. Setup a node as a golden node

3.1. The rescue.sqfs file

You need the rescue disk (wich contains the /ka directory), Just extract this file, and copy all directory in /mnt/ka.

@@ -197,7 +197,7 @@ bin/ dev/ etc/ ka/ lib/ modules/ proc/ sbin/ squashfs-root/ tmp/ usr/

Go in the /mnt/ka/ka directory, and see all new files available. All those files are needed to do a KA duplication process. We will explain now the rule of each of them. You can modify all them, those files will be copied in the directory /tmp/stage2 of the client node of the duplication process (second step). -

3.1.1. ka-d.sh

This is the master script to declare a node as a golden node. This script takes a lot of arguments. This script should be run on the host wich have the /mnt/ka directory.

@@ -213,38 +213,38 @@ bin/  dev/  etc/  ka/  lib/  modules/  proc/  sbin/  squashfs-root/  tmp/  usr/
     -r 'grub|lilo' : choose the bootloader (you can add mkinitrd options)
 
     ie: ka-d.sh -n 3 -p sda /tmp/desc -X 'sdb|sdc' -r 'grub --with=ata_piix --with=piix'

3.1.2. replication.conf

This file contains all variables needed by other scripts. It also tries to get information like IP address. -

3.1.3. fdisk_to_desc

This script generate the description table of the hard drive disk in the /tmp/desc file. This file must follow some rules: one line per partition, with two fields : type of partition and size in megabytes. The type can be linux, swap, extended. Other types can be obtained by appending their hexadecimal number to 'type'. For example linux is the same as type83. The size is either a number of megabytes, or the keyword fill (to take all available space). The logical partitions must have the logical keyword. Do a man ka-d for more help. -

3.1.4. gen_modprobe_conf.pl

This script creates a basic output like the content of the/etc/modprobe.conf file. Drawbacks this file must be updated for each new modules available in the kernel (based on the kernel/list_modules.pm file). -

3.1.5. ka-d-client

The ka-d-client binary file is used to get stage2 with the KA method, and after get the whole system. The important argument is the -s session name. A KA can only connect to a specific session (getstage2, kainstall ...). The code source is available in the ka-deploy SRPM. -

3.1.6. ka-d-server

The ka-d-server binary file is used to be a KA golden node server. Like the ka-d-client the session arguments is an important parameter (-s session_name). The session name will be getstage2 to retrieve the stage2 (after the PXE boot) and will be kainstall1 at duplication process step. If you want to do more than one duplication process of nodes at the same time, you should synchronize the ka_sesion name between the server and the client. The code source is available in the ka-deploy SRPM. -

3.1.7. ka_replication.sh

Script launched on the KA client (after getting stage2 and probing modules), to do the full process of the Ka duplication. This script call other scripts to prepare the node (prepare_node.sh), configure the bootloader (make_initrd_grub or make_initrd_lilo). -

3.1.8. store_log.sh

Basic script to store the log of the KA duplication process on an FTP server. Adjust to feet your need, and uncomment the line #store_log.sh in the /mnt/ka/ka/ka_replication.sh file. -

3.1.9. bootable_flag.sh

Script to set bootable an HDD using fdisk. First arg must be the HDD device. -

3.1.10. make_initrd_grub

- Restore and reload the Grub bootloader in the /mnt/disk directory. It's a very basic script, and perhaps use the restore_bootloader of the Mandriva Linux Rescue should be a better idea.

3.1.11. make_initrd_lilo

3.1.10. make_initrd_grub

+ Restore and reload the Grub bootloader in the /mnt/disk directory. It's a very basic script, and perhaps use the restore_bootloader of the Mandriva Linux Rescue should be a better idea.

3.1.11. make_initrd_lilo

Restore and reload the lilo bootloader in the /mnt/disk directory. Again it's a very basic script, perhaps we should use the restore_bootloader of the Mandriva Linux Rescue. -

3.1.12. prepare_node.sh

This script remove in the futur system the old network's udev rules, old dhcp cache files, launch the script gen_modprobe_conf.pl to regenerate an up to date /etc/modprobe.conf in the new system, and launch the script to restore the bootloader. If you want to do more action on the installed, system, you can modify this script. -

3.1.13. send_status.pl

Very basic perl script to open the port 12345, and paste the content of the /tmp/ka* file. It also permit the execution of commands on node, if user send a message from the golden node with the exec prefix. -

3.1.14. status_node.pl

Script to connect to a client node, first arg must be the IP address of the node. You can run command on the node with the exec prefix. -

4. The golden node, KA server

Now, it is time to build a description of the node partitions. You can use the script /mnt/ka/ka/fdisk_to_desc as root user, or your favorite text editor, you can write a file like this one:

@@ -262,7 +262,6 @@ logical linux fill

Type the following to start the ka replication server as root user on the golden node:

-      <screen>
 [root@node40 ka]# ./ka-d.sh -n 1 -p sda /root/desc -X sdb -r "grub --with=jfs --with=ata_piix"
 takembr =
 desc = sda /root/desc
@@ -289,13 +288,13 @@ Socket 5 on port 30764 on node40.guibland.com ready.
       
-r "grub --with=jfs --with=ata_piix": use grub bootloader and --with=jfs --with=piix mkinitrd option in the chrooted system after the KA deploiement
-n nb_nodes: specify how many nodes are clients
-p sda desc: specify the name of the hdd
-x /tmp: exclude /tmp directory
-X sdb: exclude sdb hdd for the duplication

     

       Now the golden node is waiting for clients nodes to start replication.
-

5. KA client node

5.1. PXE server (kamethod)

5. KA client node

5.1. PXE server (kamethod)

We have to configure the PXE to boot by default on kamethod. To do this just edit /var/lib/tftpboot/X86PC/linux/pxelinux.cfg/default and set DEFAULT to kamethod:

DEFAULT kamethod

So, next time a node boots, the PXE server will force the node to boot using the kamethod entry. -

5.2. Stage1 KA method, node waiting stage2

Now, you boot all remaining nodes. The replication process will start once all nodes are up and waiting on the KA screen. @@ -304,7 +303,7 @@ Socket 5 on port 30764 on node40.guibland.com ready. server the message Can't reach a valid KA server will appear. Each node will try five times to reach the KA server, after that the node will reboot. As the node boots on kamethod, it will retry until it finds it. -

5.3. Stage2, the duplication process

Once all the nodes have found the KA server, the first duplication process will start. This step duplicates the stage2 from the /mnt/ka directory @@ -315,7 +314,7 @@ Socket 5 on port 30764 on node40.guibland.com ready. Nodes will rewrite their partition table, then format their filesystems (ReiserFs, XFS, ext2/3/4, JFS). All new partitions will be mounted in the /mnt/disk directory. Then, the drive duplication process will begin. On a fast Ethernet switch you can reach speeds of 10MBytes/sec. -

5.4. Prepare the node

At the end of the duplication process, each node will chroot its partitions and rebuild its /boot/initrd.img, and /etc/modprobe.conf files. @@ -323,12 +322,12 @@ Socket 5 on port 30764 on node40.guibland.com ready. SCSI drives and adjusting its network card driver. Before rebooting, each node reinstalls lilo/grub. All your node are now ready, and are clone of master node. -

5.5. PXE server to local boot

Don't forget to change the default PXE boot to local so node after replication will boot localy. -

6. Step by step from scratch KA duplication

We will use a PIV 3gz box as golden node, with a SATA hard drive, and an Intel 82540EM Gigabit Ethernet Controller card. This golden box will be the: PXE, DHCPD, TFTP server. Client nodes are

6. Step by step from scratch KA duplication

We will use a PIV 3gz box as golden node, with a SATA hard drive, and an Intel 82540EM Gigabit Ethernet Controller card. This golden box will be the: PXE, DHCPD, TFTP server. Client nodes are

basic PIV 2.8gz, with a Realtek Semiconductor 8139 network card, and a IDE hard drive disk
PE2650 dual XEON 2.4gz, SCSI Hard Drive disk, and NetXtreme BCM5701 Gigabit Ethernet cards

Both nodes are configured to boot on their network card.

6.1. Golden node side

Both nodes are configured to boot on their network card.

6.1. Golden node side

Prepapre the golden node, install all needed tools.

@@ -444,7 +443,6 @@ guibland.com
        valid_lft forever preferred_lft forever
 
 [root@localhost ~]# vi /etc/pxe.conf 
------------
 # which interface to use
 interface=eth0
 default_address=10.0.1.42
@@ -482,7 +480,6 @@ tftpdbase=/
 
 # domain name
 domain=guibland.com
----------------
 
 [root@localhost ~]# vi /etc/xinetd.d/tftp 
 service tftp
@@ -951,7 +948,7 @@ Total data sent = 792 Megs, in 25445 packets
 Transfer time = 69.904 seconds, throughput = 11.343 Mbytes/second
 The pipeline was emptied in 4.002 seconds

6.2. KA client side

To get the log of the client node, launch /mnt/ka/ka/status_node.pl IPADD on the golden node.

 [root@node42 ka]# status_node.pl 10.0.1.111
@@ -1188,7 +1185,7 @@ Total data received = 788 Megs (10.796 Mbytes/sec); BUF :0M End of data flow
 10.0.1.111> ftp: connect: Connection refused
 10.0.1.111> Local directory now /tmp
 10.0.1.111> Not connected.
-

6.3. Post duplication process

Now client nodes reboots, so we have to switch PXE to a local boot.

6.3. Post duplication process

Now client nodes reboots, so we have to switch PXE to a local boot.

 [root@node42 ka]# vi /var/lib/tftpboot/X86PC/linux/pxelinux.cfg/default 
 DEFAULT local
-- 
cgit v1.2.1