Buildbot Setup for Windows
This is the recipe for setting up a MariaDB Buildbot slave on Windows:
- Prepare the development environment, including cygwin.
- Install Python version 2.6, 32 bit. Twisted does not work on 64 bit and builtbot hasn't been tested properly with Python version 3.
- Install pywin32. Make sure the version matches your Python version perfectly, and get the .exe file, not the zip file.
- Install Twisted 9.0, not the latest version. You find it under "older releases".
- Install buildbot: Get the zip file and unpack it. In an administrator shell, cd to the buildbot dir and run "python setup.py install". After that, the unpacked buildbot directory is no longer needed.
When this has been done, you run these commands:
cd <somewhere> mkdir buildbot cd buildbot bzr init-repo . buildbot create-slave --usepty=0 <slavedir> hasky.askmonty.org:9989 <slavename> <passwd>
You get <slavename> and <passwd> from one of the Monty Program people handling the buildbot system. <slavedir> can be whatever you want - <slavename> is a good choice.
Please edit the files <slavedir>\info\admin and <slavedir>\info\host with your information. The host information should include your Windows version, 32 or 64 bit, and your Visual Studio version.
You run the buildbot slave with this command:
buildbot start <somewhere>\buildbot\<slavedir>
When the buildbot is working, you can run it as a service instead of manually. See the instructions here. It's under the section "Windows Buildbot service setup". This document also has the generic Windows Buildbot installation documentation.
It's possible to install other versions than the exact version numbers mentioned above, but you just don't know how easy it will be. Most times, if you try newer versions of the software, you will end up in a dead end and have to uninstall everything again. This described set is known to work well together.
Disabling Windows pop-ups
In most (all?) Windows setups with Visual Studio, crash of some program causes Windows to pop up a dialog bog offering to debug it. This is called "Just In Time debugging", and it can be very annoying when buildbot runs a testsuite over an unstable tree and pops up the message all the time.
Approaches to disabling it:
- If you have a non-express edition of Visual Studio, go to Tools->Options->Debug->JIT debugging (or something similar, exact wording depends on the version) and disable the JIT .
- Here: http://www.techtut.com/Tutorial/Windows/44-Disabling-Windows-XP-crash-report.html they have the following suggestion:
- First of all, you need to go to "Control Panel" then click on "System" icon, choosing the "ADVANCED" tab. The second step of this process would be a simple click on the "ERROR REPORTING" button where you make sure to check the "DISABLE ERROR REPORTING" option and to uncheck the "BUT NOTIFY ME WHEN CRITICAL ERROR OCCUR" option.
Common Windows Buildbot Failures
- Test hangs after trying to call cdb to print a backtrace.
MTR attempts to call cdb
via the Perl backticks operator, which appears to be buggy on Windows. The workaround at this time is to start the test with --mysqld=--gdb
which will cause no core files to be created and thus will prevent cdb
from being called.
- Buildbot exception
This is the recipe for setting up a MariaDB Buildbot slave on Windows:
- Prepare the development environment, including cygwin.
- Install Python version 2.6, 32 bit. Twisted does not work on 64 bit and builtbot hasn't been tested properly with Python version 3.
- Install pywin32. Make sure the version matches your Python version perfectly, and get the .exe file, not the zip file.
- Install Twisted 9.0, not the latest version. You find it under "older releases".
- Install buildbot: Get the zip file and unpack it. In an administrator shell, cd to the buildbot dir and run "python setup.py install". After that, the unpacked buildbot directory is no longer needed.
When this has been done, you run these commands:
cd <somewhere> mkdir buildbot cd buildbot bzr init-repo . buildbot create-slave --usepty=0 <slavedir> hasky.askmonty.org:9989 <slavename> <passwd>
You get <slavename> and <passwd> from one of the Monty Program people handling the buildbot system. <slavedir> can be whatever you want - <slavename> is a good choice.
Please edit the files <slavedir>\info\admin and <slavedir>\info\host with your information. The host information should include your Windows version, 32 or 64 bit, and your Visual Studio version.
You run the buildbot slave with this command:
buildbot start <somewhere>\buildbot\<slavedir>
When the buildbot is working, you can run it as a service instead of manually. See the instructions here. It's under the section "Windows Buildbot service setup". This document also has the generic Windows Buildbot installation documentation.
It's possible to install other versions than the exact version numbers mentioned above, but you just don't know how easy it will be. Most times, if you try newer versions of the software, you will end up in a dead end and have to uninstall everything again. This described set is known to work well together.
Disabling Windows pop-ups
In most (all?) Windows setups with Visual Studio, crash of some program causes Windows to pop up a dialog bog offering to debug it. This is called "Just In Time debugging", and it can be very annoying when buildbot runs a testsuite over an unstable tree and pops up the message all the time.
Approaches to disabling it:
- If you have a non-express edition of Visual Studio, go to Tools->Options->Debug->JIT debugging (or something similar, exact wording depends on the version) and disable the JIT .
- Here: http://www.techtut.com/Tutorial/Windows/44-Disabling-Windows-XP-crash-report.html they have the following suggestion:
- First of all, you need to go to "Control Panel" then click on "System" icon, choosing the "ADVANCED" tab. The second step of this process would be a simple click on the "ERROR REPORTING" button where you make sure to check the "DISABLE ERROR REPORTING" option and to uncheck the "BUT NOTIFY ME WHEN CRITICAL ERROR OCCUR" option.
Common Windows Buildbot Failures
Test hangs after trying to call cdb to print a backtrace
MTR attempts to call cdb
via the Perl backticks operator, which appears to be buggy on Windows. The workaround at this time is to start the test with --mysqld=--gdb
which will cause no core files to be created and thus will prevent cdb
from being called.
Buildbot exception The process cannot access the file because it is being used by another process
This is due to the fact that buildbot does not clean up any processes left over from a previous run, and those processes may hold locks on files that are needed for a new build to start. The current workaround is to use taskkill
as the first step in the build process.
factory.addStep(ShellCommand( command=["cmd", "/C", "taskkill /im cmake* /f /t || taskkill /im mysqld* /f /t || taskkill /im bzr* /f /t || taskkill /im perl* /f /t || exit 0"], alwaysRun=True))
Buildbot exception ShellCommand.failed: command failed: SIGKILL failed to kill process
This usually happens after retrying multiple failing tests multiple times. It appears that MTR's --retries=
option is not safe to use on Windows. The solution is to run the test with no --retries
.
Buildbot exception Connection to the other side was lost in a non-clean fashion.
This is a sympthom of intermittent network issues, which cause Buildbot to abort the current build altogether. The following workarounds are possible:
- In your
buildbot.tac
file, specify a lowerkeepalive
value, such as 60. - If you are running the Windows host inside VMWare, substitute the
e1000
network adapter withvmxnet3
- If your build host is behind a firewall, makes sure that the firewall does not time out any idle connections quickly. Configure at least a 24-hour timeout.
- If your build host is behind a firewall, consider disabling the built-in Windows Firewall in order to avoid one potential point of failure.
- Modify the Windows
KeepAliveTime
registry setting to a lower value, such as 60000 (equal to 60 seconds). For more information, see TechNet - Make sure the buildbot master does not experience prolonged bouts of 100% CPU activity, as this may prevent keepalives from working. If the buildbot master
twisted.log
says that data is frequently being loaded from on-disk pickles, increase thebuildCacheSize
in the master configuration file to be more than the number of builds per builder that the log file reports are being loaded.