Fixing timeouts running Android integration tests

The main problem with Android CI I had recently is that, after switching to Lollipop, the integration tests wouldn't run. Invoking androidConnectedTest gradle target always resulted in crashing with ShellCommandUnresponsiveException. Internet says that in such a case ou just need to set ADB_INSTALL_TIMEOUT. Set and then nothing. Sourcediving it is then !

A long while after that I got to this file: Device.java [Linking to master, here's the commit hash:1cb1a4c2976b99ae53d28d7f01d975232c85f990, as I don't seem to be able to find how to link to that hash directly] What do we see there ? That indeed ADB_INSTALL_TIMEOUT is being used:

static {
    String installTimeout = System.getenv("ADB_INSTALL_TIMEOUT");
    long time = 4;
    if (installTimeout != null) {
        try {
            time = Long.parseLong(installTimeout);
        } catch (NumberFormatException e) {
            // use default value
        }
    }
    INSTALL_TIMEOUT_MINUTES = time;
}

So far so good, ADB_INSTALL_TIMEOUT system variable seems to be respected when invoking package installation tools. Are the above the only methods that can install a package though ? Going further on that hunch we see that in addition to installing single packages there is a possibility of having a multi-package installation session.

public void installPackages(List<String> apkFilePaths, int timeOutInMs, boolean reinstall, String... extraArgs) throws InstallException {
assert(!apkFilePaths.isEmpty());
if (getApiLevel() < 21) {
        Log.w("Internal error : installPackages invoked with device < 21 for %s",Joiner.on(",").join(apkFilePaths));
        if (apkFilePaths.size() == 1) {
                installPackage(apkFilePaths.get(0), reinstall, extraArgs);
                return;
        }
        Log.e("Internal error : installPackages invoked with device < 21 for multiple APK : %s", Joiner.on(",").join(apkFilePaths));
        throw new InstallException("Internal error : installPackages invoked with device < 21 for multiple APK : " + Joiner.on(",").join(apkFilePaths));
}
[...]
String sessionId = createMultiInstallSession(apkFilePaths, extraArgsList, reinstall);

Aha ! No-Lollipop check here, with a fallback to the old method - we may be onto something ! Some lines pass and we can see an invocation of createMultiInstallSession. What's there ?

private String createMultiInstallSession(List<String> apkFileNames, @NonNull Collection<String> extraArgs, boolean reinstall) throws TimeoutException, AdbCommandRejectedException, ShellCommandUnresponsiveException, IOException {
[...]
String cmd = String.format("pm install-create %1$s -S %2$d", parameters.toString(), totalFileSize);
executeShellCommand(cmd, receiver, DdmPreferences.getTimeOut());
[...]

A different invocation of executeShellCommand, now using DdmPreferences.getTimeOut() as a timeout value source. Summarizing - this only happens if you install multiple applications for your androidConnectedTest and you are using android device to test on that has api version that is equal or greater to 21. That is all cool that we had this little Computer Science Investigation, but how to fix that - i.e. how to have proper timeouts for your installations ? Ideally from somewhere you configure and/or invoke your builds. It turns out that gradle supports invoking just enough Java for us to use there. In your gradle.build as the very first lines:

println "setting global timeout for apk installation to 10 minutes"
com.android.ddmlib.DdmPreferences.setTimeOut(600000)

android {
        compileSdkVersion compileSdk
        buildToolsVersion buildTools
[...]

That's it. Invoke your android tests with ADB_INSTALL_TIMEOUT env variable set AND have the DddPreference set in your gradle.build as in the example above and you should be golden. Happy droiding !

Tools: precision test indicator holder

This is the second part in the series of the tools I use. Tools that are surprisingly useful, tools that are not that obvious to find. Check out the first part here. Today: how to calibrate the CNC axis without actually cutting anything ? Use a test indicator ! How to hold the meter steady though, ? Attach it to the frame of your router using the power of magnets ! Sample item on Amazon here [affiliate link warning]

image0 image1 image2

Despite being attached to the frame by its back instead of the bottom it still holds beautifully.

Enabling USB 3.0 in already existing Virtualbox VMs

Just a quick note on how to get USB 3.0 in Virtualbox for VMs that were created with USB 1.1 support only. First, download VirtualBox Extension Pack from here. Install it. Then quit Virtualbox completely. Go to your directory that contains your virtual machine and edit .vbox file. Replace the whole <USBController> section with the following:

<USB>
    <Controllers>
        <Controller name="xHCI" type="XHCI"/>
    </Controllers>
    <DeviceFilters/>
</USB>

That's it, let me know if it works for you !

When life gives you temp space - make lemonade !

Some VPS providers, e.g. Azure (I know..) provide you with 2 disks for your VPSes. One, of very limited size, system disk, and the other one, spacy but with not guarantees that the data survives reboot. Basically it means that you can have a small VPS, with a small amount of RAM but large temp disk space. Why this could be useful ? Imagine tasks with lots of mem requirements but that not need to be extra fast, where swapping is allowed. Like complex nightly builds. Here is a set of super simple scripts I've come up with to quickly boot up a system, and then in the background add a new swap file on the temp drive there. The temp drive is assumed to be under /mnt.

root@someazurehost:~# cat /etc/rc.local
#!/bin/sh -e
set -v

# do not wait for swap to become online,
# proceed with the boot further,
# with swap being created in the background
/etc/make_and_enable_swap &

exit 0
root@someazurehost:~# cat /etc/make_and_enable_swap
#!/bin/sh
set -e
set -v
# create new 2GB swap file
dd if=/dev/zero of=/mnt/swap bs=1M count=2048
chmod 0600 /mnt/swap
mkswap /mnt/swap
swapon /mnt/swap

Don't forget to make /etc/make_and_enable_swap executable ! Do not add this swap file to fstab, as it is being read before rc.local, and this may certainly result in a boot failure, as the swap file would not be ready yet.

Containerized zombie spawner

Recently I was playing with a fully Dockerized setup of Jenkins at work and found a curious issue there. Whenever Jenkins was polling the git server the side effect was that it created a zombie ssh process. The issue is actually remediated by the Jenkins team now by explicitly using a tiny init system called ... tini, started as the main container's process instead of just starting Jenkins there. This tiny tini thing can properly adopt and reap the children. I was all like - wow, what a great blog entry is coming at me. I was planning to describe how zombies come to existence on Linux and why Docker should, in my opinion, provide an adopter-reaper by default and other very interesting things ! But then I found a really excellent article by the Phusion team here explaining all that and more. It is very good. You should read it. That is it. The end. Happy reaping !