Saturday, August 17, 2013

Improving the testability of the Nix iOS build facilities

In the previous blog post, I have described some new features of the Nix Android build environment. Apart from Android, the Nix iOS build environment has a few improvements as well, which are mostly related to testing.

Simulating multiple iOS versions


In the previous blog post about the Nix iOS build function, I have described how to install various iPhone simulator SDK versions by picking the preferences option from the 'Xcode' menu bar and selecting the 'Downloads' tab in the window, as shown in the screenshot below:


However, I did not implement any facilities in the simulator function that take these SDK versions into account yet.

Picking a specific SDK version can be done quite easily, by providing the -currentSDKRoot parameter to the iPhone simulator, such as:

$ "iPhone Simulator" -SimulateApplication build/HelloWorld \
  -SimulateDevice 'iPhone' -currentSDKRoot \
  "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform\
/Developer/SDKs/iPhoneSimulator6.1.sdk"

In the above case, we have configured the simulator to use the iPhone simulator 6.1 SDK. By changing this parameter, we can use a different version, such as version 5.1.

I have encapsulated the above command-line invocation in the xccodeenv.simulateApp {} Nix function and the paths to the iPhone simulator SDKs in the xcodewrapper. The SDK version can be configured by providing the sdkVersion parameter (which defaults to 6.1):

{xcodeenv, helloworld, device}:

xcodeenv.simulateApp {
  name = "HelloWorld";
  app = helloworld;
  inherit device;
  sdkVersion = "6.1";
}

This parameter makes it possible to spawn a simulator instance running a specific iOS version.

The described facilities earlier are only for simulating apps. To build an app for a specific iOS revision, the iOS target version must be changed inside the Xcode project settings.

Building signed Apps for release


Apart from the simulator, we may also want to deploy apps to real devices, such as an iPhone, iPad or iPod, either directly or through the App store. In order to do that, we require permission from Apple by signing the apps that we have just built.

As described in the previous blog post, we require a certificate that contains a developer's or company's identity and a mobile provisioning file describing which (groups of) apps of a certain company/developer can be run of which (groups of) devices. These files must be obtained through Apple's Dev Center.

Because of these restrictions, it's hard to test the trivial "Hello World" example case I have developed in the previous blog post on a real device, since it contains a dummy app and company name.

To alleviate these problems, I have created a script that "renames" the example app into a different app, so that an existing developer or company certificate and a mobile provisioning profile of a different app can be used.

By setting the rename parameter to true when calling the composition expression of the example case, we can automatically generate jobs building IPAs and xcarchives for the "renamed" app:

import ./nix-xcodeenvtests/deployment {
  rename = true;
  newName = "MyVeryCoolApp";
  newId = "coolapp";
  newCompanyName = "My Cool Company";
  newDomain = "com.coolcompany";
  ipaCertificateFile = ./adhoc.p12;
  ipaCodeSignIdentity = "iPhone Distribution: My Cool Company";
  ipaCertificatePassword = "";
  ipaProvisioningProfile = ./adhoc.mobileprovision;
  xcArchiveCertificateFile = ./distribution.p12;
  xcArchiveCodeSignIdentity = "iPhone Distribution: My Cool Company";
  xcArchiveCertificatePassword = "";
  xcArchiveProvisioningProfile = ./distribution.mobileprovision;
}

In the above expression, we changed the name of the app into: "MyVeryCoolApp" and the company name into: "My Cool Company". Moreover, we provide a certificate and mobile provisioning file for the IPA job (which can be deployed to a device directly) and the xcarchive job (which can be used to deploy to the App store).

By running the following command-line instruction:

$ nix-build -A renamedPkgs.testapp_ipa

We can build an IPA allowing us to deploy the test app on a real device. Of course, to do this yourself, the company name and app names should correspond to the ones defined in your certificate and mobile provisioning profile.

Moreover, as with the Android environment in the previous blog post, we can also use Hydra to provide parameters to the composition expression and to retrieve all the resulting artifacts:


Conclusion


In this blog post, I have described some improvements to the Nix iOS facilities. We are now also capable of configuring the SDK versions for the iPhone simulator, and we can build the trivial testcase to run on a real device in addition to the simulator. This is useful for testing the Nix iOS build function.

Thursday, August 8, 2013

Some improvements to the Nix Android build environment

Some time ago, I have packaged the Android SDK and some of its plugins in Nix, which was quite a challenge. Moreover, I have developed two Nix functions that can be used to automatically build Android apps and spawn emulator instances.

Meanwhile, I have implemented some major updates since the last blog post. Besides upgrading to a newer Android SDK version, we now also support a bunch of new interesting features.

Android SDK packaging updates


Currently, I have packaged the 22.05 version of the Android SDK (which is the latest version at the time writing this blog post). To package it, I have followed the same recipe as described in the previous blog post, with some additions:

  • Newer versions of the Android SDK include x86-64 versions of the emulator executables. I had to patch/wrap these executables to allow them to find the x86-64 libraries they depend on.
  • The newer Android SDK has a new sub package called build-tools. Apparently, this sub package contains some tools that were formerly part of platform-tools, such as debugging libraries.
  • The GUI front-end of the android utility did not work in the last blog post, which I have fixed by wrapping it, so that it can find its dependencies, such as GTK+.
  • There are also x86 and MIPS application binary interface (ABI) system images, but they reside in external package repositories. I have adapted the Nix expression generator to include these as well.
  • I have adapted the build and emulation Nix functions to take both API-level and ABI versions into account
  • I have implemented some facilities to improve the performance of the emulator instances

Packaging build-tools


The build-tools package contains various tools and libraries that used to belong to platform-tools. It turns out that some of these files, such as the debugging libraries, are API-level specific. The purpose of the build-tools package is to provide API-level specific build tools. Currently, the Android SDK supports two versions: 17 and 18.01 that can be installed next to each other.

I have packaged the 18.01 version in a straight forward manner. The only thing I needed to do is inspecting the dependencies of the executables to the paths to these libraries to the executables' RPATH.

For example, I did this to patch aapt:

patchelf --set-rpath ${stdenv.gcc.libc}/lib:${stdenv.gcc.gcc}/lib:${stdenv.zlib}/lib aapt

Every version of the build-tools package is symlinked into the Android SDK basedir as follows: build-tools/android-<version>

Fixing the Android GUI front-end


If the android tool is called without any parameters, it should show a GUI that allows somebody to download plugins or to configure AVDs. However, the GUI did not work so far. Invoking 'android' gave me the following error message:

$ android
Exception in thread "main" java.lang.UnsatisfiedLinkError:
no swt-pi-gtk-3550 or swt-pi-gtk in swt.library.path,
java.library.path or the jar file
        at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
        at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
        at org.eclipse.swt.internal.gtk.OS.(Unknown Source)
        at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
        at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
        at org.eclipse.swt.widgets.Display.(Unknown Source)
        at com.android.sdkmanager.Main.showSdkManagerWindow(Main.java:346)
        at com.android.sdkmanager.Main.doAction(Main.java:334)
        at com.android.sdkmanager.Main.run(Main.java:120)
        at com.android.sdkmanager.Main.main(Main.java:103)

First, I thought that some file belonging to SWT was missing, but after manually digging through all Android files I couldn't find it and I gave up. However, after using strace, I discovered that it's actually trying to load the GTK+ library and some other libraries:

$ strace -f android
...
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk.so", 0x7fe4e0cffb20) = -1 ENOENT (No such file or directory)
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk-3550.so", {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk-3550.so", {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] open("/tmp/swtlib-64/libswt-pi-gtk-3550.so", O_RDONLY|O_CLOEXEC) = 26
[pid  8169] read(26, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\234\3\0\0\0\0\0"..., 832) = 832
[pid  8169] fstat(26, {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] mmap(NULL, 1503944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 26, 0) = 0x7fe4b5d7e000
[pid  8169] mprotect(0x7fe4b5dea000, 1044480, PROT_NONE) = 0
[pid  8169] mmap(0x7fe4b5ee9000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 26, 0x6b000) = 0x7fe4b5ee9000
[pid  8169] mmap(0x7fe4b5eec000, 4808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe4b5eec000
[pid  8169] close(26)                   = 0
[pid  8169] open("/run/opengl-driver/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  8169] open("/run/opengl-driver-32/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  8169] open("/nix/store/zm4bhsm8lprkzvrjgqr0klfkvr21als4-glibc-2.17/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...

To allow GTK+ and the other missing libraries to be found, I have wrapped the android utility:

wrapProgram `pwd`/android \
   --prefix PATH : ${jdk}/bin \
   --prefix LD_LIBRARY_PATH : ${glib}/lib:${gtk}/lib:${libXtst}/lib

Besides the libraries, I also had to wrap android to use the right JDK version. Apparently, loading these libraries only works with OpenJDK in Nix.

After wrapping the android utility and running it without parameters, it shows me the following:


As you may see in the above image, the GUI works. It also shows us all the plugins that are installed through Nix. Although the GUI seems to work fine, we have to keep in mind that we cannot use it to install additional Android SDK plugins. All plugins must be packaged and installed with Nix instead.

Supporting external system images


The Android distribution provides a collection of system images that can be used with an emulator to test an app during development. However, the default repository only contains ARM-based system images, which are quite slow to emulate.

There also seem to be x86-based system images available (provided by Intel) and MIPS-based images (provided by MIPS Technologies). These system images reside in external repositories. I was able to discover the locations of these repositories, by running the following command:

$ android list sdk
Refresh Sources:
  Fetching https://dl-ssl.google.com/android/repository/addons_list-2.xml
  Validate XML
  Parse XML
  Fetched Add-ons List successfully
  Refresh Sources
  Fetching URL: https://dl-ssl.google.com/android/repository/repository-8.xml
  Validate XML: https://dl-ssl.google.com/android/repository/repository-8.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/repository-8.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/addon.xml
  Validate XML: https://dl-ssl.google.com/android/repository/addon.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/addon.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Validate XML: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml

The latter two URLs referring to a file called sys-img.xml provide x86 and MIPS images. The structure of these XML files are similar to the main repository. Both files contain XML elements that look like this:

<sdk:system-image>
    <sdk:description>Android SDK Platform 4.1.1</sdk:description>
    <sdk:revision>1</sdk:revision>
    <sdk:api-level>16</sdk:api-level>
    <sdk:abi>x86</sdk:abi>
    <sdk:uses-license ref="intel-android-sysimage-license">
    <sdk:archives>
      <sdk:archive arch="any" os="any">
        <sdk:size>131840348</sdk:size>
        <sdk:checksum type="sha1">9d35bcaa4f9b40443941f32b8a50337f413c021a</sdk:checksum>
        <sdk:url>sysimg_x86-16_r01.zip</sdk:url>
      </sdk:archive>
    </sdk:archives>
</sdk:uses-license></sdk:system-image>

These elements can be converted to a Nix expression through XSL in a straight forward manner:

{stdenv, fetchurl, unzip}:

let
  buildSystemImage = args:
    stdenv.mkDerivation (args // {
      buildInputs = [ unzip ];
      buildCommand = ''
        mkdir -p $out
        cd $out
        unzip $src
    '';
  });
in
{
  sysimg_x86_16 = buildSystemImage {
    name = "sysimg-x86-16";
    src = fetchurl {
      url = https://dl-ssl.google.com/android/repository/sys-img/x86/sysimg_x86-16_r01.zip;
      sha1 = "9d35bcaa4f9b40443941f32b8a50337f413c021a";
    };
  };

  ...
}

The generated expression shown above is nearly identical to the one shown in the previous blog post. The only difference is that we also use the ABI identifier in the attribute names. For example: sysimg_x86_16 is used to refer to a x86-based system image for Android API-level 16. Likewise, we can refer to the ARM-based variant with: sysimg_armeabi-v7a_16.

Apart from giving each attribute a proper name, the resulting system image package must be symlinked into the Android SDK basedir taking the ABI identifier into account. We symlink every architecture-dependent system image package into: system-images/android-<api-level>/<abi>.

I have adapted the androidenv.emulateApp {} Nix function to also take ABI versions into account. By default, the emulator function uses an ARM-based system image (armeabi-v7a), since this is the most common hardware architecture used by Android OSes. By setting the abiVersion parameter to x86, we can use an x86-based Android system-image:

{androidenv, myfirstapp}:

androidenv.emulateApp {
  name = "MyFirstApp";
  app = myfirstapp;
  platformVersion = "16";
  abiVersion = "x86";
  package = "com.example.my.first.app";
  activity = "MainActivity";
}

An interesting benefit of using an x86 system image with the emulator (which is based on QEMU) is that KVM can be used, allowing programs inside the emulator to run at nearly native speed with only little emulation.

And of course, setting the abiVersion parameter to mips makes it possible to use MIPS-based system images.

Enabling GPU acceleration


The x86 system images in the emulator may give testing a significant boost. Emulation performance can even be improved a bit further by enabling GPU acceleration.

To make GPU acceleration possible, I had to adapt the emulator's wrapper script to include Mesa in the linker path. Moreover, in the AVD configuration I have to add the following boolean value to $ANDROID_SDK_HOME/.android/avd/device.avd/config.ini:

hw.gpu.enabled = yes

The androidenv.emulateApp {} automatically enables GPU acceleration from API-levels 15 and onwards. It can be disabled by setting the enableGPU parameter to false.

Supporting Google APIs in non-ARM-based system images


Although supporting x86 system images have various benefits, they also have a major drawback -- the Google APIs are not supported preventing some applications from working. One of the solutions I have seen (described in this Stack Overflow post) is to create a custom system image by starting an ARM-based emulator instance with Google APIs, fetch the libraries, start an x86-based emulator instance without Google APIs and manually install the fetched libraries.

I was able to automate the steps in the Stackflow article in a Nix expression:

{ platformVersion ? "16"
, abiVersion ? "x86"
, androidenv
, jdk
}:

let
  androidsdk = androidenv.androidsdk {
    platformVersions = [];
    abiVersions = [];
    useGoogleAPIs = false;
  };
  
  emulateAndroidARMWithGoogleAPIs = androidenv.emulateApp {
    name = "emulate-arm";
    inherit platformVersion;
    abiVersion = "armeabi-v7a";
    useGoogleAPIs = true;
  };
  
  emulateAndroidX86WithoutGoogleAPIs = androidenv.emulateApp {
    name = "emulate-x86";
    inherit platformVersion abiVersion;
    useGoogleAPIs = false;
  };
  
  mkfsYaffs2X86 = fetchurl {
    url = http://android-group-korea.googlecode.com/files/mkfs.yaffs2.x86;
    sha1 = "9362064c10a87ca68de688f09b162b171c55c66f";
  };
in
stdenv.mkDerivation {
  name = "sysimg_x86_${platformVersion}-with-google-apis";
  buildInputs = [ jdk androidsdk ];
  NIX_ANDROID_EMULATOR_FLAGS = "-no-window";
  buildCommand = ''
    source ${emulateAndroidARMWithGoogleAPIs}/bin/run-test-emulator
    portARM=$port
    adb -s emulator-$portARM pull /system/etc/permissions/com.google.android.maps.xml
    adb -s emulator-$portARM pull /system/framework/com.google.android.maps.jar
    adb -s emulator-$portARM emu kill
    
    export NIX_ANDROID_EMULATOR_FLAGS="$NIX_ANDROID_EMULATOR_FLAGS -partition-size 1024 -no-snapshot-save"
    source ${emulateAndroidX86WithoutGoogleAPIs}/bin/run-test-emulator
    portX86=$port
    adb -s emulator-$portX86 remount rw
    adb -s emulator-$portX86 push com.google.android.maps.jar /system/framework
    adb -s emulator-$portX86 push com.google.android.maps.xml /system/etc/permissions
    
    cp ${mkfsYaffs2X86} mkfs.yaffs2.x86
    adb -s emulator-$portX86 push mkfs.yaffs2.x86 /data
    adb -s emulator-$portX86 shell chmod 777 /data/mkfs.yaffs2.x86
    adb -s emulator-$portX86 shell /data/mkfs.yaffs2.x86 /system /data/system.img
    adb -s emulator-$portX86 pull /data/system.img
    adb -s emulator-$portX86 emu kill
    
    mkdir -p $out
    cp system.img $out
  '';
}

The above Nix function creates an ARM emulator instance with Google APIs and an x86 emulator instance without Google APIs. Then it starts the ARM emulator instance, fetches the Google API files from it and then stops it. Then the x86 emulator instance is started and the Google APIs are pushed to it. Finally we generate a system image from the system folder that is stored in the Nix store.

By using the Nix function shown above and setting the extraAVDFiles parameter of the emulateApp {} function, we can use our custom system image in a script that automatically spawns the emulator instance:

{androidenv, jdk}:

let
  platformVersion = "16";
  systemImg = import ./generateGoogleAPISystemImage.nix {
    inherit androidenv jdk platformVersion;
    abiVersion = "x86";
  };
in
androidenv.emulateApp {
  name = "MyFirstApp";
  extraAVDFiles = [ "${systemImg}/system.img" ];
  inherit platformVersion;
  abiVersion = "x86";
}

The above expression invokes the function shown previously and starts an emulator instance with the system image containing Google APIs. Of course, we can also use the same trick for MIPS-based system images, which also don't include Google APIs.

Building Android apps for arbitrary API-level revisions


I have also made a small change to the androidenv.buildApp {} Nix function to take API-levels into account. By default, it will use the version that is inside the Ant configuration. However, by setting the target property through the command-line, we can override this:

$ ant -Dtarget=android-17

The above command-line instruction ensures that we build the app against the API-level 17 platform. I have added an antFlags parameter to the Nix function to make it provide arbitrary flags to Ant:

{androidenv}:

androidenv.buildApp {
  name = "MyFirstApp";
  src = ../../src/myfirstapp;
  platformVersions = [ "17" ];
  useGoogleAPIs = true;
  antFlags = "-Dtarget=android-17";
}

Improving the test suite


In the previous blog post, I have implemented a testcase based on the Android introduction tutorial (MyFirstApp) that I have used throughout this article as well. To test all these new features, I have extended its composition expression to take three parameters: buildPlatformVersions (defaults to 16), emulatePlatformVersions (defaults to 16), and abiVersions (defaults to armeabi-v7a). The expression automatically generates all possible combinations with the values inside these three lists:

For example, to build a debug version of the test app APK for API-level 16, we can run:

$ nix-build -A myfirstapp_debug.build_16

We can also build a release version of the same APK that is signed with a key:

$ nix-build -A myfirstapp_release.build_16

Furthermore, we can automatically generate a script starting an emulator instance running the app. The following instruction builds an App for Android API-level 16, generates a script launching an emulator with a Android API-level 16 system-image that uses the armeabi-v7a ABI:

$ nix-build -A emulate_myfirstapp_debug.build_16.emulate_16.armeabi-v7a
$ ./result/run-test-emulator

To support more API-levels and ABIs, the parameters of the composition function must be altered. For example, by running the following command-line instruction:

$ nix-build --arg buildPlatformVersions '[ "16" "17" ]' \
  --arg emulatePlatformVersions '[ "15" "16 " 17" ]' \
  --arg abiVersions '[ "armeabi-v7a" "x86" ]' \
  -A emulate_myfirstapp_release.build_16.emulate_17.x86

The cartesian product of build and emulator instances is created taking the three dimensions into account. This allows us to (for example) build the App against the API-level 16 platform API, emulate on an API-level 17 system image using the x86 ABI, which emulates much more efficiently on x86 machines, because hardware visualization features (such as KVM) can be used.:

Apart from specifying these dimensions through the command line, the composition expression can also be used together with Hydra, the Nix-based continuous integration server, allowing it to pass these parameters through its web interface. Hydra will build all the combinations generated by the composition expression automatically:



As can be observed from the above screenshot, quite a few jobs are being generated.

Conclusion


In this blog post, I have described some new changes and features that I have implemented in the Nix Android SDK package since the last blog post, such as the fact that x86 and MIPS system-images can be used. All these changes are part of the Nixpkgs project.

Moreover, the Nix Android example testcase can be obtained from my GitHub page.