Autopkgtest - Defining tests for Debian packages ================================================ This document describes how autopkgtest interprets and executes tests found in Debian source packages. Overview -------- The source package provides a test metadata file ``debian/tests/control``. This enumerates the tests and specifies their dependencies and requirements for the testbed. It contains zero or more RFC822-style stanzas, along these lines: :: Tests: fred, bill, bongo Depends: pkg1, pkg2 [amd64] | pkg3 (>= 3) Restrictions: needs-root, breaks-testbed This example defines three tests, called ``fred``, ``bill`` and ``bongo``. The tests will be performed by executing ``debian/tests/fred``, ``debian/tests/bill``, etc. Each test program should, on success, exit with status 0 and print nothing to stderr; if a test exits nonzero, or prints to stderr, it is considered to have failed. The cwd of each test is guaranteed to be the root of the source package, which will have been unpacked but not built. *However* note that the tests must test the *installed* version of the program. Tests may not modify the source tree (and may not have write access to it). If the file to be executed has no execute bits set, ``chmod a+x`` is applied to it (this means that tests can be added in patches without the need for additional chmod; contrast this with debian/rules). During execution of the test, the environment variable ``$AUTOPKGTEST_TMP`` will point to a directory for the execution of this particular test, which starts empty and will be deleted afterwards (so there is no need for the test to clean up files left there). If tests want to create artifacts which are useful to attach to test results, such as additional log files or screenshots, they can put them into the directory specified by the ``$AUTOPKGTEST_ARTIFACTS`` environment variable. When using the ``--output-dir`` option, they will be copied into ``outputdir/artifacts/``. Tests must declare all applicable Restrictions - see below. The '#' character introduces a comment. Everything from '#' to the end of line will be entirely ignored. Examples -------- Simplest possible control file that installs all of the source package's binary packages and runs the standalone ``debian/tests/smoke`` test script as user, without any further limitations: :: Tests: smoke Control file for an inline test command that just calls a program ``foo-cli`` from package ``foo`` (which usually should be a binary of the source package that contains this test control) as root, and makes sure it exits with zero: :: Test-Command: foo-cli --list --verbose Depends: foo Restrictions: needs-root Control fields -------------- The fields which may appear in debian/tests/control stanzas are: Tests: name-of-test [, name-of-another-test ...] This field names the tests which are defined by this stanza, and map to executables/scripts in the test directory. All of the other fields in the same stanza apply to all of the named tests. Either this field or ``Test-Command:`` must be present. Test names are separated by comma and/or whitespace and should contain only characters which are legal in package names. It is permitted, but not encouraged, to use upper-case characters as well. Test-Command: shell command If your test only contains a shell command or two, or you want to re-use an existing upstream test executable and just need to wrap it with some command like ``dbus-launch`` or ``env``, you can use this field to specify the shell command directly. It will be run under ``bash -e``. This is mutually exclusive with the ``Tests:`` field. This is also useful for running the same script under different interpreters and/or with different dependencies, such as ``Test-Command: python debian/tests/mytest.py`` and ``Test-Command: python3 debian/tests/mytest.py``. Restrictions: restriction-name [, another-restriction-name ...] Declares some restrictions or problems with the tests defined in this stanza. Depending on the test environment capabilities, user requests, and so on, restrictions can cause tests to be skipped or can cause the test to be run in a different manner. Tests which declare unknown restrictions will be skipped. See below for the defined restrictions. Features: feature-name [, another-feature-name ...] Declares some additional capabilities or good properties of the tests defined in this stanza. Any unknown features declared will be completely ignored. See below for the defined features. Depends: dpkg dependency field syntax Declares that the specified packages must be installed for the test to go ahead. This supports all features of dpkg dependencies (see https://www.debian.org/doc/debian-policy/ch-relationships.html), plus the following extensions: ``@`` stands for the package(s) generated by the source package containing the tests; each dependency (strictly, or-clause, which may contain ``|``\ s but not commas) containing ``@`` is replicated once for each such binary package, with the binary package name substituted for each ``@`` (but normally ``@`` should occur only once and without a version restriction). ``@builddeps@`` will be replaced by the package's ``Build-Depends:``, ``Build-Depends-Indep:``, and ``build-essential``. This is useful if you have many build dependencies which are only necessary for running the test suite and you don't want to replicate them in the test ``Depends:``. However, please use this sparingly, as this can easily lead to missing binary package dependencies being overlooked if they get pulled in via build dependencies. If no Depends field is present, ``Depends: @`` is assumed. Note that the source tree's Build-Dependencies are *not* necessarily installed, and if you specify any Depends, no binary packages from the source are installed unless explicitly requested. Tests-Directory: path Replaces the path segment ``debian/tests`` in the filenames of the test programs with ``path``. I. e., the tests are run by executing ``built/source/tree/path/testname``. ``path`` must be a relative path and is interpreted starting from the root of the built source tree. This allows tests to live outside the debian/ metadata area, so that they can more palatably be shared with non-Debian distributions. Classes: class-1 [, class-2 ...] Most package tests should work in a minimal environment and are usually not hardware specific. However, some packages like the kernel, X.org, or graphics drivers should be tested on particular hardware, and also run on a set of different platforms rather than just a single virtual testbeds. This field can specify a list of abstract class names such as "desktop" or "graphics-driver". Consumers of autopkgtest can then map these class names to particular machines/platforms/policies. Unknown class names should be ignored. This is purely an informational field for autopkgtest itself and will be ignored. Any unknown fields will cause the whole stanza to be skipped. Defined restrictions -------------------- rw-build-tree The test(s) needs write access to the built source tree (so it may need to be copied first). Even with this restriction, the test is not allowed to make any change to the built source tree which (i) isn't cleaned up by debian/rules clean, (ii) affects the future results of any test, or (iii) affects binary packages produced by the build tree in the future. breaks-testbed The test, when run, is liable to break the testbed system. This includes causing data loss, causing services that the machine is running to malfunction, or permanently disabling services; it does not include causing services on the machine to temporarily fail. When this restriction is present the test will usually be skipped unless the testbed's virtualisation arrangements are sufficiently powerful, or alternatively if the user explicitly requests. needs-root The test script must be run as root. build-needed The tests need to be run from a built source tree. The test runner will build the source tree (honouring the source package's build dependencies), before running the tests. However, the tests are *not* entitled to assume that the source package's build dependencies will be installed when the test is run. Please use this considerately, as for large builds it unnecessarily builds the entire project when you only need a tiny subset (like the tests/ subdirectory). It is often possible to run ``make -C tests`` instead, or copy the test code to ``$AUTOPKGTEST_TMP`` and build it there with some custom commands. This cuts down the load on the Continuous Integration servers and also makes tests more robust as it prevents accidentally running them against the built source tree instead of the installed packages. allow-stderr Output to stderr is not considered a failure. This is useful for tests which write e. g. lots of logging to stderr. isolation-container The test wants to start services or open network TCP ports. This commonly fails in a simple chroot/schroot, so tests need to be run in their own container (e. g. autopkgtest-virt-lxc) or their own machine/VM (e. g. autopkgtest-virt-qemu or autopkgtest-virt-null). When running the test in a virtualization server which does not provide this (like autopkgtest-schroot) it will be skipped. isolation-machine The test wants to interact with the kernel, reboot the machine, or other things which fail in a simple schroot and even a container. Those tests need to be run in their own machine/VM (e. g. autopkgtest-virt-qemu or autopkgtest-virt-null). When running the test in a virtualization server which does not provide this it will be skipped. needs-reboot The test wants to reboot the machine using ``/tmp/autopkgtest-reboot`` (see below). needs-recommends Enable installation of recommended packages in apt for the test dependencies. This does not affect build dependencies. Defined features ---------------- There are no currently defined Features. Source package header --------------------- To allow test execution environments to discover packages which provide tests, their source packages need to have a ``Testsuite:`` header containing ``autopkgtest`` (or a value like ``autopkgtest-pkg-perl``, see below). Multiple values get comma separated, as usual in control files. This tag is added automatically by dpkg-source version 1.17.11 or later, so normally you don't need to worry about this field. Automatic test control file for known package types --------------------------------------------------- There are groups of similarly-structured packages for which the contents of ``debian/tests/control`` would be mostly identical, such as Perl or Ruby libraries. If ``debian/tests/control`` is absent, the ``autodep8`` tool can generate an automatic control file. If installed, ``autopkgtest`` will automatically use it; this can be disabled with the ``--no-auto-control`` option. Those packages do not have to provide ``debian/tests/``, but they should still include an appropriate source package header (``Testsuite: autopkgtest-pkg-perl`` or similar) so that they can be discovered in the archive. Reboot during a test -------------------- Some testbeds support rebooting; for those, the testbed will have a ``/tmp/autopkgtest-reboot`` command which tests can call to cause a reboot. **Do not** use ``reboot`` and similar commands directly without at least checking for the presence of that script! They will cause testbeds like ``null`` or ``schroot`` to reboot the entire host, and even for ``lxc`` or ``qemu`` it will just cause the test to fail as there is no state keeping to resume a test at the right position after reboot without further preparation (see below). The particular steps for a rebooting tests are: - The test calls ``/tmp/autopkgtest-reboot my_mark`` with a "mark" identifier. ``autopkgtest-reboot`` will cause the test to terminate (with ``SIGKILL``). - autopkgtest backs up the current state of the test source tree and any ``$AUTOPKGTEST_ARTIFACTS`` that were created so far, reboots the testbed, and restores the test source tree and artifacts. - The test gets run again, this time with a new environment variable ``$AUTOPKGTEST_REBOOT_MARK`` containing the argument to ``autopkgtest-reboot``, e. g. ``my_mark``. - The test needs to check ``$AUTOPKGTEST_REBOOT_MARK`` and jump to the appropriate point. A nonexisting variable means "start from the beginning". This example test will reboot the testbed two times in between: :: #!/bin/sh -e case "$AUTOPKGTEST_REBOOT_MARK" in "") echo "test beginning"; /tmp/autopkgtest-reboot mark1 ;; mark1) echo "test in mark1"; /tmp/autopkgtest-reboot mark2 ;; mark2) echo "test in mark2" ;; esac echo "test end" In some cases your test needs to do the reboot by itself, e. g. through kexec, or a reboot command that is hardcoded in the piece of software that you want to test. To support those, you need to call ``/tmp/autopkgtest-reboot-prepare my_mark`` at a point as close as possible to the reboot instead; this will merely save the state but not issue the actual reboot by itself. Note that all logs and artifacts from the time between calling ``autopkgtest-reboot-prepare`` and rebooting will be lost. Other than that, the usage is very similar to above. Example: :: #!/bin/sh if [ "$AUTOPKGTEST_REBOOT_MARK" = phase1 ]; then echo "continuing test after reboot" ls -l /var/post-request-action echo "end of test" else echo "beginning test" /tmp/autopkgtest-reboot-prepare phase1 touch /var/post-request-action reboot fi Network access -------------- autopkgtest needs access to the network at least for downloading test dependencies and possibly dist-upgrading testbeds. In environments with restricted internet access you need to set up an apt proxy and configure the testbed to use it. (Note that the standard tools like autopkgtest-build-lxc or mk-sbuild automatically use the apt proxy from the host system.) In general, tests are also allowed to access the internet. As this usually makes tests less reliable, this should be kept to a minimum; but for many packages their main purpose is to interact with remote web services and thus their testing should actually cover those too, to ensure that the distribution package keeps working with their corresponding web service. Debian's production CI infrastructure allows unrestricted network access, in Ubuntu's infrastructure access to sites other than `*.ubuntu.com` and `*.launchpad.net` happens via a proxy (limited to DNS and http/https). .. vim: ft=rst tw=72