Buildbot Setup for Windows
This is the recipe for setting up a MariaDB Buildbot slave on Windows:
- Prepare the development environment, including cygwin.
- Install Python version 2.6, 32 bit. Twisted does not work on 64 bit and builtbot hasn't been tested properly with Python version 3.
- Install pywin32. Make sure the version matches your Python version perfectly, and get the .exe file, not the zip file.
- Install Twisted 9.0, not the latest version. You find it under "older releases".
- Install buildbot: Get the zip file and unpack it. In an administrator shell, cd to the buildbot dir and run "python setup.py install". After that, the unpacked buildbot directory is no longer needed.
When this has been done, you run these commands:
cd <somewhere> mkdir buildbot cd buildbot bzr init-repo . buildbot create-slave --usepty=0 <slavedir> hasky.askmonty.org:9989 <slavename> <passwd>
You get <slavename> and <passwd> from one of the Monty Program people handling the buildbot system. <slavedir> can be whatever you want - <slavename> is a good choice.
Please edit the files <slavedir>\info\admin and <slavedir>\info\host with your information. The host information should include your Windows version, 32 or 64 bit, and your Visual Studio version.
You run the buildbot slave with this command:
buildbot start <somewhere>\buildbot\<slavedir>
When the buildbot is working, you can run it as a service instead of manually. See the instructions here. It's under the section "Windows Buildbot service setup". This document also has the generic Windows Buildbot installation documentation.
It's possible to install other versions than the exact version numbers mentioned above, but you just don't know how easy it will be. Most times, if you try newer versions of the software, you will end up in a dead end and have to uninstall everything again. This described set is known to work well together.
Disabling Windows pop-ups
In most (all?) Windows setups with Visual Studio, crash of some program causes Windows to pop up a dialog bog offering to debug it. This is called "Just In Time debugging", and it can be very annoying when buildbot runs a testsuite over an unstable tree and pops up the message all the time.
Approaches to disabling it:
- If you have a non-express edition of Visual Studio, go to Tools->Options->Debug->JIT debugging (or something similar, exact wording depends on the version) and disable the JIT .
- Here: http://www.techtut.com/Tutorial/Windows/44-Disabling-Windows-XP-crash-report.html they have the following suggestion:
- First of all, you need to go to "Control Panel" then click on "System" icon, choosing the "ADVANCED" tab. The second step of this process would be a simple click on the "ERROR REPORTING" button where you make sure to check the "DISABLE ERROR REPORTING" option and to uncheck the "BUT NOTIFY ME WHEN CRITICAL ERROR OCCUR" option.
Common Windows Buildbot Failures
- Test hangs after trying to call cdb to print a backtrace.
MTR attempts to call cdb
via the Perl backticks operator, which appears to be buggy on Windows. The workaround at this time is to start the test with --mysqld=--gdb
which will cause no core files to be created and thus will prevent cdb
from being called.
Buildbot exception The process cannot access the file because it is being used by another process
This is due to the fact that buildbot does not clean up any processes left over from a previous run, and those processes may hold locks on files that are needed for a new build to start. The current workaround is to use taskkill
as the first step in the build process.
factory.addStep(ShellCommand( command=["cmd", "/C", "taskkill /im cmake* /f /t || taskkill /im mysqld* /f /t || taskkill /im bzr* /f /t || taskkill /im perl* /f /t || exit 0"], alwaysRun=True))
Buildbot exception ShellCommand.failed: command failed: SIGKILL failed to kill process
This usually happens after retrying multiple failing tests multiple times. It appears that MTR's --retries=
option is not safe to use on Windows. The solution is to run the test with no --retries
.
Buildbot exception Connection to the other side was lost in a non-clean fashion.
This is a sympthom of intermittent network issues, which cause Buildbot to abort the current build altogether. The following workarounds are possible:
- In your
buildbot.tac
file, specify a lowerkeepalive
value, such as 60. - If you are running the Windows host inside VMWare, substitute the
e1000
network adapter withvmxnet3
- If your build host is behind a firewall, makes sure that the firewall does not time out any idle connections quickly. Configure at least a 24-hour timeout.
- If your build host is behind a firewall, consider disabling the built-in Windows Firewall in order to avoid one potential point of failure.
- Modify the Windows
KeepAliveTime
registry setting to a lower value, such as 60000 (equal to 60 seconds). For more information, see TechNet - Make sure the buildbot master does not experience prolonged bouts of 100% CPU activity, as this may prevent keepalives from working. If the buildbot master
twisted.log
says that data is frequently being loaded from on-disk pickles, increase thebuildCacheSize
in the master configuration file to be more than the number of builds per builder that the log file reports are being loaded.
Alternative Windows Buildbot setup (experimental)
In case the default Windows buildbot setup is not sufficiently reliable due to "connection lost in a non-clean fashion" errors, the following setup can be used instead. It runs the buildbot deamon on a linux host while doing the builds on Windows, thus working around Twisted issues on Windows.
Note that the procedure below _significantly_ degrades the overall security of your Windows host. It is strongly recommended that a properly-firewalled, standalone virtual machine is used.
- Set the Windows server to auto-login with the user the builds will run as. This will prevent any issues relating to running things as a service and the privilege considerations associated with services;
- Install FreeSSHd on Windows. The OpenSSHd/Cygwin combo is not likely to work.
- Do not run FreeSSHd it as a service, but rather as a startup console application for the user you are running the builds under. This allows FreeSSHd UI to properly operate;
- Generate a private/public key pair using PuTTYGen and place the public part in a properly named file in the key directory specified in the FreeSSHd GUI
- Using the FreeSSHd GUI, create a new user and set it to use key authentication;
- Set the SFTP home path to the directory where builds will take place.
- Disable UAC (User Access Control) (or set to lowest level) by using the Users Control Panel. Add your buildbot user to the Administrators group.
- Export the private key from PuTTYGen and copy it to a Linux host.
- Install buildbot on the Linux host
- The buildbot configuration file will then look something like:
f_win2008r2_i386_release.addStep(ShellCommand( command=["ssh", "-i", "/home/buildbot/keys/id_rsa", "buildbot@win2008r2-build", "cmd", "/C", "whoami"] ));
The following buildbot considerations apply under this setup:
- buildbot will not perform a proper cleanup of the build directory on the Windows host before each build (it will only clean up things on the linux side). So, a directory cleanup via rmdir must be placed as as an explicit first step in the build sequence;
- Bzr() can not be used to check out BZR trees, as it expects that the bzr checkout command will run locally. Instead, an explicit call to bzr checkout via a SSH command must be used;