1 of 8

Buildbot Setup for Virtual Machines - Additional Steps

This section documents some additions that have been made after the initial setup

Install Cassandra on Fulltest VMs

CassandraSE is no longer actively being developed and has been removed in MariaDB 10.6. See MDEV-23024.

Here are the steps I took to install Cassandra on the Fulltest VMs.

backed up the fulltest VMs with:

rsync -avP /kvm/vms/*fulltest* host:/destination/path/

boot the amd64 fulltest VM:

in the VM, install Cassandra:

in the VM, launch the cassandra-cli program and test the Cassandra installation:

Output of the above:

in the VM, shut it down:

Do steps 2-6 for vm-precise-i386-fulltest.qcow2. The output of the testing step was:

on the other build hosts, rsync the files over:

_{This page is licensed: CC BY-SA / Gnu FDL}

Install cmake on build VMs

Requires cmake. Install cmake on all -build VMs (and other Unix-like machines) with:

wget http://www.cmake.org/files/v2.8/cmake-2.8.8.tar.gz
tar -zxvf cmake-2.8.8.tar.gz
cd cmake-2.8.8;./configure
make
sudo make install

_{This page is licensed: CC BY-SA / Gnu FDL}

Installing correct libraries for PAM and readline

Some additional/alternative libraries needs to be installed to handle readline and PAM correctly.

The newer libreadline is GPLv3 and so not compatible with the MariaDB/MySQL GPLv2 license. The PAM libraries are needed for the PAM plugin.

On the Centos and RHEL -build VMs, install the pam-devel package:

sudo yum install pam-devel

On all the Debian/Ubuntu -build virtual machines, install libpam0g-dev:

sudo apt-get install libpam0g-dev

On debian6/maverick/natty, install libreadline5-dev (replacing any libreadline6-dev already there):

sudo apt-get install libreadline5-dev

On oneiric (and any newer, eg. Debian 7 or Ubuntu 12.04), the package is called libreadline-gplv2-dev:

sudo apt-get install libreadline-gplv2-dev

_{This page is licensed: CC BY-SA / Gnu FDL}

Installing the Boost library needed for the OQGraph storage engine

The OQGraph storage engine needs a newer version of Boost that what is available on (most) distributions. The version installed is 1.42.0, available from.

The boost library is installed in all the builder virtual machine images with the following single command:

for i in "vm-hardy-amd64-build qemu64" "vm-hardy-i386-build qemu32,-nx" \
        "vm-intrepid-amd64-build qemu64" "vm-intrepid-i386-build qemu32,-nx" \
        "vm-karmic-amd64-build qemu64" "vm-karmic-i386-build qemu32,-nx" \
        "vm-jaunty-amd64-build qemu64" "vm-jaunty-i386-deb-build qemu32,-nx" \
        "vm-lucid-amd64-build qemu64" "vm-lucid-i386-build qemu32,-nx" \
        "vm-maverick-amd64-build qemu64" "vm-maverick-i386-build qemu32,-nx" \
        "vm-natty-amd64-build qemu64" "vm-natty-i386-build qemu64" \
        "vm-oneiric-amd64-build qemu64" "vm-oneiric-i386-build qemu64" \
        "vm-debian5-amd64-build qemu64" "vm-debian5-i386-build qemu32,-nx" \
        "vm-debian4-amd64-build qemu64 --netdev=e1000" "vm-debian4-i386-build qemu32,-nx --netdev=e1000" \
        "vm-centos5-i386-build qemu32,-nx" "vm-centos5-amd64-build qemu64" \
        "vm-hardy-amd64-build qemu64" "vm-hardy-i386-build qemu32,-nx" \
        "vm-jaunty-i386-deb-tarbake qemu32,-nx" ; do \
  set $i; \
  runvm -m 512 --smp=1 --port=2200 --user=buildbot --cpu=$2 $3 $1.qcow2 \
    "= scp -P 2200 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no /kvm/boost_1_42_0.tar.gz buildbot@localhost:/dev/shm/" \
    "sudo mkdir -p /usr/local/src /usr/local/include" \
    "cd /usr/local/src && sudo tar zxf /dev/shm/boost_1_42_0.tar.gz" \
    "cd /usr/local/include && sudo ln -s ../src/boost_1_42_0/boost ." ; \
done

Upgrade Boost to 1.49

To upgrade boost on the VMs to 1.49 I performed the following steps:

I initially tried to upgrade the VMs using a script like the one at the top of the page which was used to install boost 1.42, but I ran into issues with getting it to work on all of the VMs (it worked on some, but not on others). So I ended up using the steps below.

Copy the VM (keeping the original as a backup, in case something goes wrong):

Start the VM:

Copy the boost tar.gz file over to the VM:

ssh into the VM:

once inside the VM, perform the following steps:

_{This page is licensed: CC BY-SA / Gnu FDL}

Installing VM images for testing .deb upgrade between versions

This step creates virtual machine images used to do an important additional upgrade test for .debs. Each virtual machine is pre-installed with an older version of MariaDB, and the test tries upgrading this version to the newly build one (to check that dependencies etc. work out correctly).

This step may not be easily possible to replicate exactly for new or updated images. For example, when we add a new platform/distro, we will not have the same old version of MariaDB available for installation on the new planform. Or if needing to re-create an image in the future, the original old .debs used before may no longer be available. However, this is not a big problem, as we can just use whatever version of MariaDB is available. In fact, while we cannot reasonably test every possible upgrade combination between MariaDB versions, it is still useful to test upgrades from different versions on different platforms, to increase coverage a bit.

The bulk of the images were installed with the following two loops, using packages from the OurDelta repository for , which was the one available from there at the time of installation.

for i in "vm-hardy-amd64-install qemu64 hardy" "vm-hardy-i386-install qemu32,-nx hardy" \
          "vm-intrepid-amd64-install qemu64 intrepid" "vm-intrepid-i386-install qemu32,-nx intrepid" \
          "vm-karmic-amd64-install qemu64 karmic" "vm-karmic-i386-install qemu32,-nx karmic" \
          "vm-jaunty-amd64-install qemu64 jaunty" "vm-jaunty-i386-deb-install qemu32,-nx jaunty" \
          "vm-debian5-amd64-install qemu64 lenny" "vm-debian5-i386-install qemu32,-nx lenny" ; do \
  set $i; \
  runvm -b  $1.qcow2 -m 512 --smp=1 --port=2200 --user=buildbot --cpu=$2 $(echo $1 | sed -e 's/-install/-upgrade2/').qcow2 \
    "wget -O- http://ourdelta.org/deb/ourdelta.gpg | sudo apt-key add -" \
    "sudo sh -c \"echo 'deb http://mirror.ourdelta.org/deb $3 mariadb-ourdelta' > /etc/apt/sources.list.d/ourdelta.list\"" \
    "sudo apt-get update || true" \
    "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y mariadb-server mariadb-test libmariadbclient-dev || true" \
    'mysql -uroot -prootpass -e "create database mytest; use mytest; create table t(a int primary key); insert into t values (1); select * from t"' \
    "sudo rm /etc/apt/sources.list.d/ourdelta.list" \
    "sudo apt-get update || true" \
    "sudo DEBIAN_FRONTEND=noninteractive apt-get upgrade -y || true" ; \
done

The Ubuntu 10.04 "lucid" images were installed manually (as no packages for lucid were available from OurDelta at the time of installation):

Create and boot 64-bit lucid upgrade image:

qemu-img create -b vm-lucid-amd64-install.qcow2 -f qcow2 vm-lucid-amd64-upgrade2.qcow2 
kvm -m 512 -hda vm-lucid-amd64-upgrade2.qcow2 -redir 'tcp:2200::22' -boot c -smp 1 -cpu qemu64 -net nic,model=virtio -net user -nographic

Create directory inside the image:

Copy in packages to be installed from outside:

Install the MariaDB packages, remove package dir, and upgrade to latest security fixes:

Now the same, but for 32-bit lucid:

Create directory inside the image:

Copy in packages to be installed from outside:

Install the MariaDB packages, remove package dir, and upgrade to latest security fixes:

The Ubuntu 10.10 "maverick" images were likewise installed manually:

Create and boot 64-bit maverick upgrade image:

Same for 32-bit maverick:

For Ubuntu 11.04 "natty", mariadb packages were installed from the repository for the previous version.

64-bit Ubuntu natty:

32-bit Ubuntu natty:

For Ubuntu 11.10 "oneiric". The steps performed are listed at: (in the VMs for Upgrade Testing section)

_{This page is licensed: CC BY-SA / Gnu FDL}

Resizing a Virtual Machine Image

Some KVM images end up not having enough space on them. In such cases, it is preferable to increase the size of the VM rather than to just delete an image and rebuild it from scratch. The steps outlined below document what was done to increase the size of the Red Hat 5 x86 VM and should be able to be easily adapted to other VMs, should they need the same treatment in the future.

Make a copy of the VM to work on (we don't want to change the original):

Using the qemu-img command, resize the image:

Not all versions of qemu-img can resize VMs.

Update Debian 4 mirrors for Buildbot VMs

Debian 4 has become so old that the apt repository has been moved out of the main Debian 4 mirror servers, and into the archive of old versions. This needs to be fixed by pointing the Debian 4 images to a different mirror:

64-bit:

kvm -m 512 -hda vm-debian4-amd64-install.qcow2 -redir 'tcp:2200::22' -boot c -smp 1 -cpu qemu64 -net nic,model=e1000 -net user -nographic

sudo vi /etc/apt/sources.list
# replace http://ftp.dk.debian.org/debian/ with http://ftp.de.debian.org/archive/debian/

32-bit:

kvm -m 512 -hda vm-debian4-i386-install.qcow2 -redir 'tcp:2200::22' -boot c -smp 1 -cpu qemu64 -net nic,model=e1000 -net user -nographic

sudo vi /etc/apt/sources.list
# replace http://ftp.dk.debian.org/debian/ with http://ftp.de.debian.org/archive/debian/

After that, it is necessary to re-do from scratch the -update and -update2 debian4 images (as these are built on top of the -install images).

_{This page is licensed: CC BY-SA / Gnu FDL}

Installing VM images for testing .deb upgrade between versions

The bulk of the images were installed with the following two loops, using packages from the OurDelta repository for , which was the one available from there at the time of installation.

for i in "vm-hardy-amd64-install qemu64 hardy" "vm-hardy-i386-install qemu32,-nx hardy" \
          "vm-intrepid-amd64-install qemu64 intrepid" "vm-intrepid-i386-install qemu32,-nx intrepid" \
          "vm-karmic-amd64-install qemu64 karmic" "vm-karmic-i386-install qemu32,-nx karmic" \
          "vm-jaunty-amd64-install qemu64 jaunty" "vm-jaunty-i386-deb-install qemu32,-nx jaunty" \
          "vm-debian5-amd64-install qemu64 lenny" "vm-debian5-i386-install qemu32,-nx lenny" ; do \
  set $i; \
  runvm -b  $1.qcow2 -m 512 --smp=1 --port=2200 --user=buildbot --cpu=$2 $(echo $1 | sed -e 's/-install/-upgrade2/').qcow2 \
    "wget -O- http://ourdelta.org/deb/ourdelta.gpg | sudo apt-key add -" \
    "sudo sh -c \"echo 'deb http://mirror.ourdelta.org/deb $3 mariadb-ourdelta' > /etc/apt/sources.list.d/ourdelta.list\"" \
    "sudo apt-get update || true" \
    "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y mariadb-server mariadb-test libmariadbclient-dev || true" \
    'mysql -uroot -prootpass -e "create database mytest; use mytest; create table t(a int primary key); insert into t values (1); select * from t"' \
    "sudo rm /etc/apt/sources.list.d/ourdelta.list" \
    "sudo apt-get update || true" \
    "sudo DEBIAN_FRONTEND=noninteractive apt-get upgrade -y || true" ; \
done

for i in "vm-debian4-amd64-install qemu64 etch /kvm/debian-40r8-amd64-netinst.iso" "vm-debian4-i386-install qemu32,-nx etch /kvm/debian-40r8-i386-netinst.iso" ; do \
  set $i; \
  runvm -b  $1.qcow2 -m 512 --smp=1 --port=2200 --user=buildbot --cpu=$2 --netdev=e1000 --kvm=-cdrom --kvm=$4 $(echo $1 | sed -e 's/-install/-upgrade2/').qcow2 \
    "wget -O- http://ourdelta.org/deb/ourdelta.gpg | sudo apt-key add -" \
    "sudo sh -c \"echo 'deb http://mirror.ourdelta.org/deb $3 mariadb-ourdelta' > /etc/apt/sources.list.d/ourdelta.list\"" \
    "sudo apt-get update || true" \
    "sudo sh -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y mariadb-server mariadb-test libmariadbclient-dev' || true" \
    'mysql -uroot -prootpass -e "create database mytest; use mytest; create table t(a int primary key); insert into t values (1); select * from t"' \
    "sudo rm /etc/apt/sources.list.d/ourdelta.list" \
    "sudo apt-get update || true" \
    "sudo sh -c 'DEBIAN_FRONTEND=noninteractive apt-get upgrade -y' || true" ; \
done

The Ubuntu 10.04 "lucid" images were installed manually (as no packages for lucid were available from OurDelta at the time of installation):

Create and boot 64-bit lucid upgrade image:

qemu-img create -b vm-lucid-amd64-install.qcow2 -f qcow2 vm-lucid-amd64-upgrade2.qcow2 
kvm -m 512 -hda vm-lucid-amd64-upgrade2.qcow2 -redir 'tcp:2200::22' -boot c -smp 1 -cpu qemu64 -net nic,model=virtio -net user -nographic

Create directory inside the image:

mkdir buildbot

Copy in packages to be installed from outside:

scp -P 2200 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -r /archive/pack/mariadb-5.1-knielsen/build-277/kvm-deb-lucid-amd64/debs buildbot@localhost:buildbot/

Install the MariaDB packages, remove package dir, and upgrade to latest security fixes:

Now the same, but for 32-bit lucid:

Create directory inside the image:

Copy in packages to be installed from outside:

Install the MariaDB packages, remove package dir, and upgrade to latest security fixes:

The Ubuntu 10.10 "maverick" images were likewise installed manually:

Create and boot 64-bit maverick upgrade image:

Same for 32-bit maverick:

For Ubuntu 11.04 "natty", mariadb packages were installed from the repository for the previous version.

64-bit Ubuntu natty:

32-bit Ubuntu natty:

For Ubuntu 11.10 "oneiric". The steps performed are listed at: (in the VMs for Upgrade Testing section)

_{This page is licensed: CC BY-SA / Gnu FDL}