Sunday, December 30, 2012

Second blog anniversary

Today, it's my blog's second anniversary, so it looks like this is a good time for some reflection.

2012 in a nutshell


Although I have fewer blog posts this year compared to last year (although the difference is not that big), 2012 was in fact an extremely busy/stressful year. First, I had to write my PhD thesis, which was quite a project. Actually, from February up to August nearly all my time was consumed writing it, when I was not sleeping or eating. Besides my thesis, also some publication work had to be done, such as the HotSWUp publication about deploying mutable software components and a journal version of the WASDeTT paper about Disnix.

Although I was very busy, I've tried to keep blogging as much as possible. In the beginning this worked out well, but later this year I had some difficulties posting anything relevant (as may be observed from April 2012 until October 2012).

I also had some depressing moments. One of them was about science in general. One of the things that made me really mad/frustrated were reviewers "complimenting" our papers with: "this is a remarkable piece of engineering" (which implies that it's not science, so the paper will be rejected). I wrote quite a lengthy blog post about this issue, and dived into literature to find an explanation, which I was able to find.

Another thing that bothered me was the fact that so little of the work that we are doing in the research world are picked up by the other fractions of the software engineering community, such as industry. Similarly, I also see that people in industry have their struggles and needs (in particular: practical application of new concepts), but refrain from finding solution elsewhere or collaborating, such as with the research world. Today, all software engineering fractions more or less all live in our separate world, while it is (obviously) better to collaborate. Although it makes sense to do that, only a few are actually doing it.

Although I had some depressing/stressful moments, I also published two blog posts at the beginning of this year, about fun projects involving my good old Amiga, with a bit of overlap to my research. Most of the work for these blog posts was already done last year, but I still had to write them down.

After finishing my thesis draft and a short break, my employment contract with the university expired and I had to switch jobs. The process of finding a new job was quite interesting. A few things that I've noticed is that only a few companies have real interest in who you are and what background you have. A lot of them use a standard application procedures, such as capacity tests, and a lot of "standard" interview questions. Most of them also treated me as an undergrad and offered me salaries way lower than my PhD salary.

Furthermore, before I started talking to these companies I considered myself still as somebody who's more like an engineer as a scientist. But when I spoke to them, I've noticed then I'm more "scientific" than I thought. Although these observations, nearly all the applications I did went fine and I have received many job offers. I was rejected one time, probably because that company was looking for somebody "who just wants to program in Java", instead of a critical person like me.

Finally, I was hired by Conference Compass as a Software Architect. Here, I still use some stuff I was working on during research, such as deployment with Nix and related tools. I've produced two blog posts related to my deployment work there -- deployment of mobile apps on Android and iOS devices.

Blog posts


Another interesting observation is the "popularity" of my blog. This year, I think I have attracted 10 times as many visitors compared to last year. Especially in the last two months, I've attracted a lot of readers, thanks to Reddit. At this moment, the top 10 of the most frequently read blog posts is:

  1. On Nix and GNU Guix. Not so long ago, the GNU project had announced GNU Guix, a package manager based on Nix, implementing an internal DSL using GNU Guile, instead of using the Nix expression language, an external DSL. I wanted to give my opinion about it and the blog post has been picked up by the Reddit community, attracting a huge amount of visitors.
  2. An alternative explanation of the Nix package manager. I wrote this blog post, because I had to explain Nix to my new employer. I've used a different explanation style that is language-oriented and I actually find this version a bit better that the explanation recipe I normally use. I've decided to blog about it and it has been picked up by the Reddit community, attracting a lot of readers.
  3. Second computer. This is the blog post about my good ol' Commodore Amiga. It was also in last year's top 10, and it still seems to be very popular. This shows me that there are still a lot of people out there with nostalgic feelings :-).
  4. An evaluation and comparison of GoboLinux. This blog post was also in last year's top 10 in which I compare the deployment properties of NixOS with GoboLinux. This blog has also been reddited some time ago.
  5. Porting software to AmigaOS (unconventional style). This was a random fun experiment in which I combine some nostalgic Amiga feelings with my current research. I did not expect it to be that interesting, but for some reason it has attracted a lot of readers and even a few people who wanted to try the function I have developed.
  6. NixOS: A purely functional Linux distribution. Another blog post that was in last year's top 10 and still seems to attract readers. This proves that NixOS remains interesting!
  7. Software deployment complexity. One year ago, this was my most popular blog post, that's highly related to my research. It still seems to be relevant, fortunately :-).
  8. Software engineering fractions, collaborations and "The System". This was a blog post I wrote this year in a depressing period, but nonetheless, I still think it's very relevant for anyone involved in the software engineering community to read. Some people say that I'm in a good shape when I'm ranting about something.
  9. The Nix package manager. This was my original explanation of the Nix package manager to use the standard explanation recipe. This blog post was also in last year's top 10.
  10. First blog post. Yet another blog post, that was ranked third in my top 10 last year. People still want to know who I am :-).

Some thoughts


As with my last's year reflection about my blog, I have noticed that my blog attracts many more readers than all my scientific papers combined. Another thing that I observed is that writing for my blog is more convenient than writing papers, because I'm not bound to all kinds of restrictions, such as the page limit, topics that are interesting (and are uninteresting), the amount of "greek symbols" that I should use, whether something is engineering or science etc. It feels like a big relief, it's much more fun, attracts more readers and gives me more feedback.

Actually, it seems that this observation is not unique to me and has been noticed quite some time ago already by Edsger Dijkstra, a very famous computer scientist who published most of his contributions through EWDs, manuscripts which he wrote about anything he wanted. In EWD1000 titled: 'Twenty-eight years', he reflected over his previous 28 research years and one of the interesting things he wrote in that manuscript is:
"If there is one "scientific" discovery I am proud of, it is the discovery of the habit of writing without publication in mind. I experience it as a liberating habit: without it, doing the work becomes one thing and writing it down becomes another one, which is often viewed as an unpleasant burden. When working and writing have merged, that burden has been taken away."
Most manuscripts of him after 1974 were hand-written and he never owned a computer. Because of that people consider him old fashioned. But on the other hand, he was way ahead of his time. In fact, Dijkstra's way of working could be considered an early form of blogging using a pen+paper+photocopier instead of a computer+internet.

I also think that writing more frequently and distributing your findings more often offers much more value. Quite frequently people give the expression that publishing a paper goes like this:
You reach your goal without any obstacles. But in reality it goes more like this:
You will reach many obstacles and you may not even fully reach your goal, or only under special circumstances. These obstacles are also interesting to know about, as well as how to practically implement certain aspects. It gives you better feedback, helps to "structure your mind" and to better reach your audience. Most researchers refrain from doing that.

And another thing is, if you really want any research work to succeed, you much also reach non-researchers. For this practical aspects as well as usable software is important. That's probably a big reason why this blog attract quite some readers.

Concluding remarks


In this blog post I've reflect over 2012 and it seems that it's still worth having this blog. Next year, there is more interesting stuff to come. The final thing I'd like to say is:

HAPPY NEW YEAR!!!!!

Thursday, December 27, 2012

Deploying iOS applications with the Nix package manager

Some time ago, I have explored the mobile deployment space domain a bit, by implementing a function capable of building Android Apps with the Nix package manager. However, Android is definitely not the only platform used in the mobile space. Another major one is Apple's iOS, used by mobile devices, such as the iPhone, iPod, and iPad. In this blog post, I describe how I have implemented a Nix function capable of building Apps for these platforms.

Although it was quite some work to package the Android SDK, it took me less work for iOS. However, all prerequisites for this platform cannot be entirely packaged in Nix, due to the fact that they require Xcode, that must be installed through the App store, and it has all kinds of implicit assumptions on the system (such as location of certain tools and binaries), which prevents all required components to be packaged purely. Therefore, I had to use similar "tricks" that I have used to allow .NET applications to be deployed through the Nix package manager.

Installing Xcode


The first dependency that requires manual installation is Apple's Xcode, providing a set of development tools and an IDE. It must be obtained from the App store through the following URL: https://developer.apple.com/xcode. The App store takes care of the complete installation process.

After installing Xcode, we must also manually install the command-line utilities. This can be done by starting Xcode, picking the preferences option from the 'Xcode' menu bar, selecting the 'Downloads' tab in the window and then by installing the command-line utilities, as shown in the screenshot below:


An Xcode installation includes support for the most recent iOS versions by default. If desired, SDKs for older versions can be installed through the same dialog.

Installing MacPorts


To use the Nix package manager, we can either download a collection of bootstrap binaries or we can compile it from source. I took the latter approach for which it is required to install a number of prerequisites through MacPorts. Installation of MacPorts is straight forward -- it's just downloading the installer from their website (for the right Mac OS X version) and running it.

Installing the Nix package manager


To install Nix from source code, I first had to install a bunch of dependencies through MacPorts. I opened a terminal and I've executed the following commands:
$ sudo port install sqlite3
$ sudo port install pkgconfig
$ sudo port install curl
$ sudo port install xz
I have obtained the source tarball of Nix from the Nix homepage, and I have installed it from source by running the following command-line instructions:
$ tar xfvj nix-1.1.tar.bz2
$ cd nix-1.1
$ ./configure
$ make
$ sudo make install
Furthermore, I was a bit lazy and I only care about using a single-user installation. To use Nix as a non-root user I ran the following command to change the permissions to my user account:
sudo chown -R sander /nix
For development, it's also useful to have a copy of Nixpkgs containing expressions for many common packages. I cloned the git repository, by running:
$ git clone https://github.com/NixOS/nixpkgs.git
To conveniently access stuff from the Nix packages repository, it is useful to add the following line to the ~/.profile file so that they are in Nix's include path:
$ export NIX_PATH=nixpkgs=/path/to/nixpkgs
I also added channels that provides substitutes for builds performed at our build infrastructure, so that I don't have to compile everything from source:
$ nix-channel --add http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable
$ nix-channel --update

"Packaging Xcode" in Nix


In order to package iOS applications, we have to make the Xcode build infrastructure available to Nix package builds. However, as explained earlier, we cannot package Xcode in Nix, because it must be installed through other means (through Apple's App store) and it has a number of assumptions on locations of utilities on the host system.

Instead we use the same trick that we have used earlier for the .NET framework -- we create a proxy component wrapper that contains symlinks to essential binaries on the host system. Our xcodewrapper package expression looks as follows:

{stdenv}:

let
  version = "4.5.2";
in
stdenv.mkDerivation {
  name = "xcode-wrapper-"+version;
  buildCommand = ''
    ensureDir $out/bin
    cd $out/bin
    ln -s /usr/bin/xcode-select
    ln -s /usr/bin/xcodebuild
    ln -s /usr/bin/xcrun
    ln -s /usr/bin/security
    ln -s "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform\
/Developer/Applications/iPhone Simulator.app/Contents/MacOS/iPhone Simulator"

    # Check if we have the xcodebuild version that we want
    if [ -z "$($out/bin/xcodebuild -version | grep ${version})" ]
    then
        echo "We require xcodebuild version: ${version}"
        exit 1
    fi
  '';
}
Basically, the following expression contains a number of symlinks to essential Xcode binaries:

  • xcode-select is a utility to determine filesystem location of Xcode, or to switch between default versions.
  • xcodebuild is a command-line utility to build Xcode projects.
  • xcrun is a utility to execute specific Xcode-specific operations, such as packaging a build result.
  • security is a command-line interface to Mac OS X's keychain facility that manages security certificates and identities.
  • iPhone Simulator. A program to simulate iOS applications on the host system.

Furthermore, the expression contains a check that verifies whether the installed Xcode matches the version that we want to use, ensuring that -- if we upgrade Xcode -- we use the version that we want, and that everything that has Xcode as build-time dependency gets rebuilt, if necessary.

By adding the xcodewrapper to the buildInputs parameter of a package expression, the Xcode build utilities can be found and used.

Building iOS apps for the simulator


Getting Xcode working in Nix is a bit ugly and does not provide full reproducibility as Xcode must be installed manually, but it works. Another important question to answer is, how do we build iOS apps with Nix?

Building iOS apps is pretty straight forward, by running xcodebuild inside an Xcode project directory. The xcodebuild reference served as a good reference for me. The following instruction suffices to build an app for the iPhone simulator:
$ xcodebuild -target HelloWorld -configuration Debug \
  -sdk iphonesimulator6.0 -arch i386 ONLY_ACTIVE_ARCH=NO \
  CONFIGURATION_TEMP_DIR=$TMPDIR CONFIGURATION_BUILD_DIR=$out
The above command-line instruction builds an app for the i386 architecture (Apps in the simulator have the same architecture as the host system) using the iPhone 6.0 simulator SDK. The last two parameters ensure that intermediate files generated by the compiler are stored in our temp directory and that the resulting build is stored in our desired prefix, instead of the Xcode project directory.

Building iOS apps for release


Building actual releases for Apple devices is a bit trickier though. In order to deploy an App to a Apple device, a distribution certificate is required as well as an provisioning profile. To test on an Apple device, an ad-hoc provisioning profile is required. For the App store, you need a distribution profile. These files must be obtained from Apple. More information on this can be found here.

Furthermore, in order to use a certificate and provisioning profile, we need to import these into the keychain of the user that executes a build. I use the following instructions to automatically create and unlock a new keychain:
$ security create-keychain -p "" mykeychain
$ security default-keychain -s mykeychain
$ security unlock-keychain -p "" mykeychain
The above instructions generate a keychain with the name mykeychain in the user's home directory: ~/Library/Keychains.

After creating the keychain, we must import our certificate into it. The following instruction imports a key with the name mykey.p12 into the keychain with name: mykeychain and uses the password: secret for the certificate (the -A parameter grants all applications access to the certificate):
$ security import mykey.p12 -k mykeychain -P "secret" -A 
We must also make the provisioning profile available. This can be done by extracting the UUID from the mobile provisioning file and to copy the file into ~/Library/MobileDevice/Provisioning Profiles/<UUID>.mobileprovision:
PROVISIONING_PROFILE=$(grep UUID -A1 -a myprovisioningprofile.mobileprovision | \
  grep -o "[-A-Za-z0-9]\{36\}")
mkdir -p "$HOME/Library/MobileDevice/Provisioning Profiles"
cp myprovisioningprofile.mobileprovision \
  "$HOME/Library/MobileDevice/Provisioning Profiles/$PROVISIONING_PROFILE.mobileprovision"
After importing the certificate and provisioning files, we can invoke xcodebuild to build a release App using the earlier certificate and provisioning profile (I have based this invocation on a Stack overflow post about setting a provisioning profile):
$ xcodebuild -target HelloWorld -configuration Release \
  -sdk iphoneos6.0 -arch armv7 ONLY_ACTIVE_ARCH=NO \
  CONFIGURATION_TEMP_DIR=$TMPDIR CONFIGURATION_BUILD_DIR=$out \
  CODE_SIGN_IDENTITY="My Cool Company" \
  PROVISIONING_PROFILE=$PROVISIONING_PROFILE \
  OTHER_CODE_SIGN_FLAGS="--keychain $HOME/Library/Keychains/mykeychain"
The above command-line invocation is nearly identical to the one that builds an App for the simulator, except that we have a different SDK parameter: iphoneos6.0, and the armv7 architecture. Moreover, we have to specify the identity name of the key, and the locations of the provisioning profiles and keychains.

To test an app on an Apple device, we must generate an IPA file, a Zip archive containing all files belonging to an App. This can be done by running:
$ xcrun -sdk iphoneos PackageApplication \
  -v HelloWorld.app -o HelloWorld.ipa
By using the Xcode organiser the given App can be uploaded to an Apple device.

In order to release an App through the Apple App store, we must generate an xcarchive, a zip archive that also contains debugging symbols and plist files. An xcarchive can be generated by adding the archive parameter to xcodebuild. The resulting xcarchive is automatically added to the Xcode organiser.

If the build process has been completed, we can eventually remove our keychain:
$ security default-keychain -s login.keychain
$ security delete-keychain mykeychain

NOTE: Before we delete the keychain, we must turn another keychain (in this case the login keychain) the default. Omitting this step might cause corruption to the entire keychain service!

Simulating iOS applications


We can also script the iPhone simulator to automatically start an App produced by xcodebuild, for a particular iOS device:
"iPhone Simulator" -SimulateApplication build/HelloWorld \
  -SimulateDevice 'iPhone'
The above command launches an iPhone instance running our simulator build performed earlier. The former command-line option refers to the executable that we have built. The latter specifies the type of device we want to simulate. Other device options that worked for me are: 'iPad', 'iPad (Retina)', 'iPhone (Retina 3.5-inch)', 'iPhone (Retina 4-inch)'.

Implementing a test case



To demonstrate how the iOS deployment functions work, I have implemented the trivial 'Hello world' example case described in Apple's iOS tutorial. It was also a nice exercise to get myself familiar with Xcode and Objective C development.

Apart from implementing the example case, I have made two extensions:

  • The example only implements the iPhone storyboard. On the iPad, it shows me an empty screen. I also implemented the same stuff in the iPad storyboard to make it work smoothly on the iPad.
  • In order to pass the validation process for releases, we need to provide icons. I added two files: Icon.png (a 59x59 image) and Icon-72.png (a 72x72 image) and I added them to the project's info.plist file.

Deploying iOS applications with Nix


I have encapsulated the earlier described build steps in a Nix function called: xcodeenv.buildApp. To build the Hello World example for the simulator, one could write the following expression:
{xcodeenv}:

xcodeenv.buildApp {
  name = "HelloWorld";
  src = ../../src/HelloWorld;
  release = false;
}
The function is very simple -- it invokes the xcodeenv.buidApp function and takes three parameters: name is the name of the xcode project, src is a path referring to the source code and release indicates whether we want to build for release or not. If release is false, then it will build the app for the simulator. It also assumes that the architecture (arch) is i386 and the SDK is the iphonesimulator, which is usually the case. These parameters can be overridden by adding them to the function parameters of the function invocation.

To make a build for release, the following expression can be written:
{xcodeenv}:

xcodeenv.buildApp {
  name = "HelloWorld";
  src = ../../src/HelloWorld;
  release = true;

  certificateFile = /Users/sander/mycertificate.p12;
  certificatePassword = "secret";
  codeSignIdentity = "iPhone Distribution: My Cool Company";  
  provisioningProfile = /Users/sander/provisioningprofile.mobileprovision;
  generateIPA = false;
  generateXCArchive = false;
}
The above expression sets the release parameter to true. In this case, the architecture is assumed to be armv7 and the SDK iphoneos, which ensures that the App is build for the right architecture for iOS devices. For release builds we also need to specify the certificate, provisioning profile, and their relevant properties. Furthermore, we can also specify whether we want to generate an IPA (an archive to deploy an App to a device) or xcarchive (an archive to deploy an App to Apple's App store).

The function invocation above conveniently takes care of the build, key signing and packaging process, although there were two nasty caveats:

  • To ensure purity, Nix restricts access to the user's home directory by setting the HOME environment variable to /homeless-shelter. For some operations that require a home directory, it suffices to set HOME to a temp directory. This seems not to work for key signing on Mac OS X. Generating a keychain and importing a certificate seems to work fine, but signing does not. I suspect this has something to do with the fact that the keychain is a system service.

    I have circumvented this issue by generating a keystore in the real user's home directory, with exactly the same name as the Nix component name that we are currently building, including the unique hash derived from the build inputs. The hash should ensure that the name never interferes with other components. Furthermore, the keychain is always removed after the build, whether it succeeds or not. This solution is a bit nasty and tricky, but it works fine for me and should not affect an existing build, running simultaneously.
  • Another caveat is the generation of xcarchives. It works fine, but the result is always stored in a global location: ~/Library/Developer/Xcode/Archives that allows the Xcode organiser to find it. It looks like there is no way to change this. Therefore, I don't know how to make the generation of xcarchives pure, although this is a task that only needs to be performed when releasing an App through Apple's App store. This task is typically performed only once in a while.

As with ordinary Nix expressions, we cannot use the previous two expressions to build an App directly, but we also need to compose them by calling them with the right xcodeenv parameter. This is done in the following composition expression:
rec {
  xcodeenv = import ...

  helloworld = import ./pkgs/helloworld {
    inherit xcodeenv;
  };

  simulate_helloworld_iphone = import ./pkgs/simulate-helloworld {
    inherit xcodeenv helloworld;
    device = "iPhone";
  };

  simulate_helloworld_ipad = import ./pkgs/simulate-helloworld {
    inherit xcodeenv helloworld;
    device = "iPad";
  };
  ...
}
By using the expression above and by running nix-build, we can automatically build an iOS app:
$ nix-build default.nix -A helloworld
/nix/store/0nlz31xb1q219qrlmimxssqyallvqdyx-HelloWorld
$ cd result
$ ls
HelloWorld.app  HelloWorld.app.dSYM
Besides build an App, we can also write an expression to simulate an iOS app:
{xcodeenv, helloworld, device}:

xcodeenv.simulateApp {
  name = "HelloWorld";
  app = helloworld;
  inherit device;
}
The above expression calls the xcodeenv.simulateApp function that generates a script launching the iPhone Simulator. The name parameter corresponds to the name of the App, the app parameter refers to a simulator build of an App, an the device parameter specifies the device to simulate, such as an iPhone or iPad.

If we take the composition expression (shown earlier), we also see a simulator attribute: simulate_helloworld_iphone, which we can evaluate to generate a script launching a simulator for the iPhone. By executing the generated script we launch the simulator:
$ nix-build -A simulate_helloworld_iphone
$ ./result/bin/run-test-simulator
Which shows me the following:


In addition to the iPhone, I have also developed a function invocation simulating the iPad (as can be seen in the composition expression, shown earlier):
$ nix-build -A simulate_helloworld_ipad
$ ./result/bin/run-test-simulator
And this is the result:


Keychain issues


I have used this function for a quite a few builds and it should work fine. However, in very rare cases it may happen that a generated keychain for signing is not properly removed due to errors in the build infrastructure.

To remedy the problem, I typically open the 'Keychain Access' program and select the following option in the menu: Keychain Access -> Keychain First Aid. In the resulting dialog, I click on the 'Repair' button. Now all the invalid keychain references are removed and everything should work fine again.

Conclusion


In this blog post, I have described two Nix functions that allow users to automatically build, sign and package iOS applications and to run these in an iPhone simulator instance.

Although the Nix functions seems to work, they are still highly experimental, have some workarounds (such as the key signing) and are not entirely pure, due to the fact that it requires Xcode which must be installed through Apple's App store.

I have conducted these experiments on a Mac OS X 10.7.5 (Lion) machine. Mac OS X 10.8 Mountain Lion, is also reported to work.

I'd like to thank my colleagues Mathieu Gerard (for showing me the keysigning command-line instructions) and Alberto Gonzalez Sanchez (for testing this on Mac OS X Mountain Lion) from Conference Compass.

This function is part of Nixpkgs. The example case can be obtained from my github page.

Presentation


UPDATE: On July 14, 2016 I have given a presentation about this subject at the Nix meetup in Amsterdam. For convenience, I have embedded the sildes into this blog post: