Calabash is the greatest mobile acceptance testing tool out there (I know of). We love it at Fovea: not a single mobile project without it. This allows to express our app’s behaviors in plain english, allowing non technical people to read, validate, and even edit them! Those behaviors are then executed using ruby rules that turn plain english into clicks and assertions.
So of course, I wanted calabash-android to run with CircleCI.
How to make Calabash tests run within CircleCI?
A nice thing about CircleCI is that it runs your tests inside a full-featured container, bundled with much more services that you’ll probably ever need: all kinds of databases, SDKs, compilers, interpreters, etc. Among those: the android SDK, ruby and a pre-bundled android virtual device (AVD).
Installing calabash is then no more than running
gem install calabash-android
To start the emulator, first, you can start the pre-bundled AVD named “circleci-android21” or you can build and start your own if needed:
emulator -avd circleci-android21 -no-audio -no-window
Also add the necessary options making the command run in background.
Then you need to wait for the device to be fully booted. CircleCI provides a convenient command that does just that:
Final step is to unlock the device. Unfortunately, the emulator comes in locked, calabash doesn’t unlock the device and fails to run if you don’t do it manually. I solved the issue with adb, by emitting the unlock input command:
adb shell input keyevent 82
I also added a “trick-click” to the “OK” button of the android first-start welcome screen:
adb shell input touchscreen swipe 370 735 371 735
The last one is probably not necessary, but I guess it doesn’t hurt either.
Here’s the full dependencies and pre-test section on your circleci.yml file:
machine: environment: ANDROID_HOME: /usr/local/android-sdk-linux dependencies: override: - gem install calabash-android test: pre: - emulator -avd circleci-android21 -no-audio -no-window: background: true parallel: true - circle-android wait-for-boot - sleep 5 - adb shell input keyevent 82 - sleep 5 - adb shell input touchscreen swipe 370 735 371 735 - sleep 5
We’re now ready to run the calabash tests.
Running the tests
To run a calabash test, you first need to build your APK. For the sake of this article let’s say you have a “build.sh” script that generates an “unsigned.apk” file in the project directory.
Then you can sign your APK as usual:
cp unsigned.apk calabash.apk calabash-android resign "`pwd`/calabash.apk"
Finally run your tests:
calabash-android run "`pwd`/calabash.apk" --format junit --out /tmp/junit
As you noticed, I made sur calabash generates junit formatted output. CircleCI is able to process JUNIT output, so that’s useful.
We now have to copy all the generated artifacts and test results to the appropriate location:
- cp unsigned.apk $CIRCLE_ARTIFACTS - cp *.png $CIRCLE_ARTIFACTS - cp /tmp/junit/* $CIRCLE_TEST_REPORTS - adb logcat -d > $CIRCLE_ARTIFACTS/logcat.txt
We chose to copy the screenshots taken by calabash (*.png), the logcat output and the APK as artifacts.
Full test override section in circle.yml
override: - cp unsigned.apk calabash.apk - calabash-android resign "`pwd`/calabash.apk" - calabash-android run "`pwd`/calabash.apk" --format junit --out /tmp/junit - cp unsigned.apk $CIRCLE_ARTIFACTS - cp *.png $CIRCLE_ARTIFACTS - cp /tmp/junit/* $CIRCLE_TEST_REPORTS - adb logcat -d > $CIRCLE_ARTIFACTS/logcat.txt
The bad part
The android emulator runs only in ARM mode, no x86 with KVM supported. This is pretty slow: emulators’ startup time takes about 5-7 minutes. Then, you also need to increase your calabash tests timeout duration or you might get wrongly rejected tests.
Apart from that it does the job. It may take 15 minutes to tell you that you latest push is Ok or broken, but you get at least to know it’s broken without having to worry about running the tests on your own. That’s one less thing to think about for you and your team.
Going further, we could now trigger running the same tests on real devices using services like Appthwack, Xamarin, or any other that support Calabash. Those services charge money for time your spend running the tests on their devices, so by saving you from running those tests on builds that failed on CircleCI, you also save money.