CircleCI + Calabash for Android

Calabash is the greatest mobile acceptance testing tool out there (I know of). We love it at Fovea: not a single mobile project without it. This allows to express our app’s behaviors in plain english, allowing non technical people to read, validate, and even edit them! Those behaviors are then executed using ruby rules that turn plain english into clicks and assertions.

I recently started using CircleCI for our new projects’ continuous integration, coming from an in-house Jenkins, it’s just way easier to setup and saves headache to maintain.

So of course, I wanted calabash-android to run with CircleCI.

How to make Calabash tests run within CircleCI?

A nice thing about CircleCI is that it runs your tests inside a full-featured container, bundled with much more services that you’ll probably ever need: all kinds of databases, SDKs, compilers, interpreters, etc. Among those: the android SDK, ruby and a pre-bundled android virtual device (AVD).

Installing calabash is then no more than running

gem install calabash-android

To start the emulator, first, you can start the pre-bundled AVD named “circleci-android21” or you can build and start your own if needed:

emulator -avd circleci-android21 -no-audio -no-window

Also add the necessary options making the command run in background.

Then you need to wait for the device to be fully booted. CircleCI provides a convenient command that does just that:

circle-android wait-for-boot

Final step is to unlock the device. Unfortunately, the emulator comes in locked, calabash doesn’t unlock the device and fails to run if you don’t do it manually. I solved the issue with adb, by emitting the unlock input command:

adb shell input keyevent 82

I also added a “trick-click” to the “OK” button of the android first-start welcome screen:

adb shell input touchscreen swipe 370 735 371 735

The last one is probably not necessary, but I guess it doesn’t hurt either.

Here’s the full dependencies and pre-test section on your circleci.yml file:

machine:
  environment:
    ANDROID_HOME: /usr/local/android-sdk-linux

dependencies:
  override:
    - gem install calabash-android

test:
  pre:
    - emulator -avd circleci-android21 -no-audio -no-window:
        background: true
        parallel: true
    - circle-android wait-for-boot
    - sleep 5
    - adb shell input keyevent 82
    - sleep 5
    - adb shell input touchscreen swipe 370 735 371 735
    - sleep 5

We’re now ready to run the calabash tests.

Running the tests

To run a calabash test, you first need to build your APK. For the sake of this article let’s say you have a “build.sh” script that generates an “unsigned.apk” file in the project directory.

Then you can sign your APK as usual:

cp unsigned.apk calabash.apk
calabash-android resign "`pwd`/calabash.apk"

Finally run your tests:

calabash-android run "`pwd`/calabash.apk" --format junit --out /tmp/junit

As you noticed, I made sur calabash generates junit formatted output. CircleCI is able to process JUNIT output, so that’s useful.

We now have to copy all the generated artifacts and test results to the appropriate location:

    - cp unsigned.apk $CIRCLE_ARTIFACTS
    - cp *.png $CIRCLE_ARTIFACTS
    - cp /tmp/junit/* $CIRCLE_TEST_REPORTS
    - adb logcat -d > $CIRCLE_ARTIFACTS/logcat.txt

We chose to copy the screenshots taken by calabash (*.png), the logcat output and the APK as artifacts.

Full test override section in circle.yml

  override:
    - cp unsigned.apk calabash.apk
    - calabash-android resign "`pwd`/calabash.apk"
    - calabash-android run "`pwd`/calabash.apk" --format junit --out /tmp/junit
    - cp unsigned.apk $CIRCLE_ARTIFACTS
    - cp *.png $CIRCLE_ARTIFACTS
    - cp /tmp/junit/* $CIRCLE_TEST_REPORTS
    - adb logcat -d > $CIRCLE_ARTIFACTS/logcat.txt

The bad part

The android emulator runs only in ARM mode, no x86 with KVM supported. This is pretty slow: emulators’ startup time takes about 5-7 minutes. Then, you also need to increase your calabash tests timeout duration or you might get wrongly rejected tests.

Apart from that it does the job. It may take 15 minutes to tell you that you latest push is Ok or broken, but you get at least to know it’s broken without having to worry about running the tests on your own. That’s one less thing to think about for you and your team.

Going further, we could now trigger running the same tests on real devices using services like Appthwack, Xamarin, or any other that support Calabash. Those services charge money for time your spend running the tests on their devices, so by saving you from running those tests on builds that failed on CircleCI, you also save money.

I’m a consultant and developer for Mobile, Web, Games and Apps projects. I co-founded Fovea 8 years ago.

Posted in Blog
One comment on “CircleCI + Calabash for Android
  1. kavita says:

    Hi Jean-Christophe Hoelt, Thanks for the nice article,it helped me a lot,i am trying to run the tests on real devices using services Xamarin,can you guide me whats the best way.

    thanks
    kavita