Creating Calabash Tests

8/17/2019 Creating Calabash Tests

1/14


2/14

more scenarios . A scenario describes a test that describes when some functionality of an application

has been properly implemented. It exists within the context of the feature. A scenario is in turnsubdivided into steps – a list of sentences that describe the pre-conditions for the scenario, theactions a user would take, and the expected results of the user’s actions. Each step is matched to astep definition - a piece of Ruby code that executes when the test runs. The following diagram illustrates how all these pieces fit together:

Step definitions use the Calabash APIs to interact with the application while it is running on a deviceor in the simulator/emulator. The APIs contain methods to simulate user actions such as touchingthe screen, entering text into text fields, and more. Calabash also provides a rich query language tolocate and interact with objects on the screen.

This guide is broken up into two main sections. The first section covers the basics of creating an

acceptance test in Calabash. It will discuss the basic workflow and the steps involved in buildingacceptance tests. It discusses how to use the predefined steps that ship with Calabash, as well ashow to write custom steps using the Calabash APIs and the Calabash query language. The second part of this guide introduces some of the techniques and best practices for writing testsfor both iOS and Android applications.

Requirements This guide builds on the material covered in several other Xamarin documents. It is crucial that youread and understand the content in the Introduction To Calabash guide. This guide assumes that

Ruby, Cucumber, and Calabash are installed and configured as per the steps defined in theCalabash Installation document. Familiarity with the Ruby programming language and Cucumber is helpful. A text editor such as Sublime Text or TextMate is required when developing the features and thestep definitions. Optionally, you can use RubyMine, a full-featured commercial Ruby IDE with built in

support for Cucumber.

http://localhost/var/www/apps/conversion/tmp/scratch_2/creating_calabash_tests/Images/image1.pnghttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/calabash_ruby_api/http://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/intro_to_calabashhttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/installationhttp://www.sublimetext.com/http://macromates.com/http://www.jetbrains.com/ruby/http://www.jetbrains.com/ruby/http://macromates.com/http://www.sublimetext.com/http://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/installationhttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/intro_to_calabashhttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/calabash_ruby_api/http://localhost/var/www/apps/conversion/tmp/scratch_2/creating_calabash_tests/Images/image1.png


3/14

1.

Creating Tests with Calabash Before we can start writing tests, we need to create the proper directory structure to house the testsand the supporting files for the tests. Calabash provides some helper commands to create the basicdirectory structure. These helpers are run at the command from within the solution directory. If wewant to start writing tests for an iOS project, we would run calabash-ios gen at the command line, as

illustrated by the following snippet: $ calabash-ios gen ----------Question---------- I'm about to create a subdirectory called features.

features will contain all your calabash tests. Please hit return to confirm that's what you want. -

-------------------------- ----------Info---------- Features subdirectory created. Make sure you've

build your -cal scheme in XCode and try executing cucumber ---------------------------

This command creates the directory structure in the following screenshot:

calabash-ios created the features folder, a sample .feature file, and the folders step_definitions and

support. The tests are written using Gherkin, a plain text domain specific language (DSL) that can beread and understood by anybody. The two subdirectories will hold Ruby source code files. The files in the step_definitions hold the

step definitions - the code snippets that make up the steps that make up the feature. The support directory holds the source code that is shared amongst the step definitions. The calabash-android gen command is the corresponding command for Android projects. It will createthe same directory structure for Android projects. Once the Cucumber directory structure is created, we can begin writing the acceptance tests. Writing

tests follows this general workflow:

Write the feature – As described in the overview, the developer and a subject matter expert

work together to write the Calabash feature. They create a .feature file that describes how the

http://localhost/var/www/apps/conversion/tmp/scratch_2/creating_calabash_tests/Images/image2.pnghttp://cukes.info/gherkin.htmlhttp://cukes.info/gherkin.htmlhttp://localhost/var/www/apps/conversion/tmp/scratch_2/creating_calabash_tests/Images/image2.png


4/14

2.

3.

4.

5.

6.

7.

8.

scenarios should work.

Run the feature – Next, the developer runs the feature that was created. It will fail because the

steps have not yet been defined. However, Cucumber will help us out by providing some code

snippets that we can use to create the step definitions.

Create the Step Definitions – Using the snippets that Cucumber provided in step #2, we

create a Ruby source code file, and paste the snippet output into it.

Explore the Application using the Calabash Console – Not strictly necessary, this step

involves starting up an instance of the Calabash console. The Calabash console is a command

line utility that allows us to issue commands to the Calabash test server communicating with

our app. We can use this to discover the Calabash queries necessary to interact with the UI

object in the application.

Update Step Definitions – Once we have figured out the Calabash queries for locating and

manipulating the UI object, we can use these - along with the Calabash API - to implement the

step definitions.

Run the Feature – When the step definitions are finished, we can run the features. If this is a

brownfield application, the functionality has already been coded and the tests should all pass. If

this is a greenfield project, the tests will not pass as there is no application code – we will moveon to step #7 below.

Implement the Feature in the Application – This step is only necessary in a greenfield

project. The developer will shift attention to the application and write the code to implement the

desired functionality, and make the test pass.

Repeat – Once the feature is implemented with a passing test, it is done. Time to move on and

implement the next bit of functionality in the application.

With this understanding of the basics in place, lets examine each of these steps in a bit more detailas part of a simple walkthrough.

A Simple Walkthrough To better appreciate the workflow described in the previous section, let’s put them in action as part ofa walkthrough. In this walkthrough, we will write a feature on a simple mobile application that

performs some basic validation of a credit card number. In the application, the user enters a creditcard number into a text field, and clicks the Validate button. If the credit card number is invalid, theapplication displays an error message on the screen for the user. The following low fidelity mockupshows an example of what this application might look like on either iOS or Android:

Note: calabash-ios gen or calabash-android gen only need to run once. These commands willterminate if they detect an existing features directory.


5/14

There are many rules for credit card validation, but let's keep things simple so that it’s possible tofocus on learning how to write an acceptance test. This sample application will only have one rule: a

credit card must be exactly 16 digits long.

Getting Started To save time, we’ll use a pre-built application for testing. The CreditCardValidation.zip file contains afunctional Xamarin.iOS application and has been already been prepared to support Calabash testing

by ios-calabash gen. This is a working application, but there are no acceptance tests written for it atthis point. Running the application, we should see the following:

http://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/creating_calabash_tests/Resources/CreditCardValidation.ziphttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/creating_calabash_tests/Resources/CreditCardValidation.ziphttp://localhost/var/www/apps/conversion/tmp/scratch_2/creating_calabash_tests/Images/image3.png


6/14

http://localhost/var/www/apps/conversion/tmp/scratch_2/creating_calabash_tests/Images/image4.png


7/14

So, with a working application and the basic file structure for the features in place, lets get started

with the feature.

Creating the Feature Our first acceptance test will consist of one Cucumber feature with two scenarios:

Credit card is to shortCredit card is to long

Following the workflow of the previous section, add a text file to the features folder and name itcredit_card_validation.feature. Edit the file so that it contains the following feature and scenario: Feature: Credit card validation. Credit card numbers must be exactly 16 digits. Scenario: Credit

card number is to short Given I use the native keyboard to enter "123456" into text field number 1

And I touch the "Validate" button Then I see the text "Credit card number is to short."

This first scenario is an example of using the pre-defined steps that come with Calabash iOS. Pre-

defined steps are handy because they don’t require us to write any step definitions. We can run thistest against the sample right now and this first feature should pass. Executing the test is as simple

as running Cucumber from the command line, as illustrated by the following snippet: $ cucumber Feature: Credit card validation. Credit card numbers must be exactly 16 digits.

Sceanario: Credit Card is too short # features/credit_card_validation.feature:4 Given I use the

native keyboard to enter "123456" into text field number 1 # calabash-cucumber-

0.9.162/features/step_definitions/calabash_steps.rb:140 And I touch the "Validate" button #

calabash-cucumber-0.9.162/features/step_definitions/calabash_steps.rb:31 Then I see the text

"Credit card number is to short." # calabash-cucumber-

0.9.162/features/step_definitions/calabash_steps.rb:373 1 scenario (1 passed) 3 steps (3 passed)0m18.035s

This is great! With minimal effort on our part we have our first test up and running. Let’s move on tothe remaining scenarios and see how they would be implemented using custom steps.

Implementing the Second Scenario The predefined steps are a great way to get started, but we can’t rely on them for every possiblescenario for the following reasons:

There are a limited number of predefined steps – it’s impossible for Xamarin or the Calabashcommunity to create steps for every possible scenario.

The predefined steps promote an imperative style of writing steps. This means that scenarios

are saying how to do something, as opposed to what to do. This will make the scenario hard to

read and difficult to maintain.

Many of the predefined steps have step definitions that use an older, deprecated Calabash API.

In this section, we will address these three points by creating custom steps.

We’ll implement the second scenario using custom steps. Open up the filecredit_card_validation.feature, and add the following scenario at the end of the file: Scenario: Credit card number is to long Given I try to validate a credit card number that is 17

characters long Then I should see the error message "Credit card number is to long."


8/14

This is an example of a declarative style of step writing. Note that the scenario itself is not concerned

with what field to enter text into or what view to touch to validate the credit card. The scenarioexplains what should be done and what the application should do. The details of how enter 17characters onto the screen and what view should be touched are implementation details and are theresponsibility of the step definition, and we will get to that in a bit. Before we dive into creating the step definitions, lets run the test again at the command line. Weshould see the following output:

$ cucumber Feature: Credit card validation. Credit card numbers must be exactly 16 digits.

Scenario: Credit card number is to short # features/credit_card_validation.feature:4 Given I use

the native keyboard to enter "123456" into text field number 1 # calabash-cucumber-




0.9.162/features/step_definitions/calabash_steps.rb:373 Scenario: Credit card number is to long #

features/credit_card_validation.feature:9 Given I try to validate a credit card number that is 17

characters long # features/credit_card_validation.feature:10 Then I should see the error message

"Credit card number is to long." # features/credit_card_validation.feature:11 2 scenarios (1

undefined, 1 passed) 5 steps (2 undefined, 3 passed) 0m29.895s You can implement step definitions

for undefined steps with these snippets: Given(/Î try to validate a credit card number that is

(\d+) characters long$/) do |arg1| pending # express the regexp above with the code you wish you

had end Then(/Î should see the error message "(.*?)"$/) do |arg1| pending # express the regexp

above with the code you wish you had end

Notice that Cucumber realizes that we don’t have any step definitions for the custom steps in oursecond scenario, so it tries to help us out. It provides some snippets for the undefined steps. Let’suse these snippets to create the step definitions. Add a new file calledfeatures/step_definitions/credit_card_validation_steps.rb. Edit that file and paste in the two

snippets Cucumber suggested. The step definition file should resemble the following bit of code: Given(/Î try to validate a credit card number that is (\d+) characters long$/) do |arg1| pending #

express the regexp above with the code you wish you had end Then(/Î should see the error message

"(.*?)"$/) do |arg1| pending # express the regexp above with the code you wish you had end

These step definitions are stubbed in with one line and the pending keyword. Cucumber can runthem, but it will stop when it hits the pending keyword, raise a notification, and skip the remainingsteps. The next thing we need to do is to actually implement the step definitions. The first step definition willtake care of entering text into the UITextView for the password and pressing the Validate button. Letsdo that now.

Creating a Step Definition

To create this step definition, we need to figure out the Calabash queries to locate the credit cardtext field, the Validate button, and the error messages text field. We can perform this explorationusing the Calabash console . To start up the Calabash console:

Drop to the command line.

Change the working directory to the solution directory.

Run calabash-ios console at the command line.


9/14

Start the Calabash test server on the device.The following snippet shows an example of this: $ calabash-ios console Running irb... irb(main):001:0> start_test_server_in_background

Calabash::Cucumber::Launcher: Launch Method instruments Log file:

/var/folders/61/bhbtr4_51tx9bl3l5lqzl87m0000gn/T/run_loop20131205-90049-c66i85/run_loop.out

irb(main):002:0>

At this point, the iOS simulator should be running and display the first view in the application. Ourapplication will respond to any Calabash commands that come in through the Calabash console.

The Calabash API provides the query function that will find objects on the screen. It takes a query string and returns an array of hash objects with information about the objects that match the query.The query string format is very similar to the CSS selector. To locate all objects on the screen, wewould invoke query like so: irb(main):002:0> query "*" [ [0] { "class" => "UIView", "id" => nil, "rect" => { "center_x" => 160,

"y" => 0, "width" => 320, "x" => 0, "center_y" => 284, "height" => 568 }, "frame" => { "y" => 0,

"width" => 320, "x" => 0, "height" => 568 }, "label" => nil, "description" => "

The "*" query string is handy for when we don’t know exactly what we're looking for on the screen. It

will dump a list of all objects on the screens. We can examine the properties of those objects anduse the details of each object to craft a query string. It is possible to get very specific with the query string to locate individual objects on the screen. Forexample, the Calabash query string to locate the Validate button would look something like thefollowing: irb(main):003:0> query "button marked:'Validate'" [ [0] { "class" => "UIButton", "id" => nil,

"rect" => { "center_x" => 160, "y" => 242, "width" => 280, "x" => 20, "center_y" => 264, "height"=> 44 }, "frame" => { "y" => 242, "width" => 280, "x" => 20, "height" => 44 }, "label" =>

"Validate", "description" => "

If we were to translate the query string to English, it would read something like “Find all of theUIButton objects on the screen that are marked with the text ‘Validate’”. The marked keyword usesan algorithm that examines the properties of an object and tries to find one that matches the string.For more information about the Calabash query syntax, read the Calabash query languagedocumentation. The following table lists the Calabash query strings to locate the various objects on the screen.

Now that we have the Calabash query strings for the various UI objects on the screen, we canimplement the step definitions. Open up the file credit_card_validation_steps.rb, and change theGiven step definition to match the following code: Given(/^I try to validate a credit card number that is (\d+) characters long$/) do

|number_of_digits| touch("textField marked:'CreditCardNumberField'") wait_for_keyboard

keyboard_enter_text("9" * number_of_digits.to_i) touch("button marked:'Validate'") end

This step definition makes extensive use of the Calabash API to interact with the view objects on the

View Object Calabash Query String

Credit Card Number text field textField marked:'CreditCardNumberField'

Validate button button marked:'Validate'

Error Messages text view textView marked:'ErrorMessagesField'

http://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/calabash-query_syntaxhttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/calabash-query_syntaxhttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/calabash-query_syntaxhttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/calabash-query_syntax


10/14

screen. This step definition will touch the Credit Card text field, giving that view focus, and wait for

the keyboard to appear. Then the step definition uses the keyboard_enter_text function to input acredit card number using the iOS keyboard. Once that is done, the touch method will simulate a usertapping the Validate button. The final step definition is responsible for ensuring that the correct error message is displayed on thescreen. Edit credit_card_validation_steps.rb so that it resembles the following code snippet: Then(/^I should see the error message "(.*?)"$/) do |error_message| text_view = query("textView

marked:'ErrorMessagesField' {text CONTAINS '#{error_message}'}") raise "The error message

'#{error_message}' is not visible in the View." unless text_view.any? end

This step definition will query for a that contains a particular string using a predicate . The predicateis enclosed in the curly brackets can be used to test properties on an object. In this example thepredicate will check to see if the text property of the text view contains a specific string. If the querymethod is not able to find any objects that match the string then it will return an empty array. If thereis an empty array, the step definition will raise an error that will cause the test to fail. At this point, the feature should be complete. If we run the tests, we should see the following outputin command line console: $ cucumber Feature: Credit card validation. Credit card numbers must be exactly 16 digits.

Scenario: Credit card number is to short # features/credit_card_validation.feature:4 Given I use

the native keyboard to enter "123456" into text field number 1 # calabash-cucumber-




0.9.162/features/step_definitions/calabash_steps.rb:373 Scenario: Credit card number is to long #

features/credit_card_validation.feature:9 Given I try to validate a credit card number that is 17

characters long # features/step_definitions/credit_card_validation_steps.rb:1 Then I should see the

error message "Credit card number is to long." #

features/step_definitions/credit_card_validation_steps.rb:7 2 scenarios (2 passed) 5 steps (5

passed) 0m36.442s

If you’ve made it this far, then congratulations! You now have a working application that has someacceptance tests to prove that the application works as desired.

Best Practices

Up to this point in the document, we were only concerned with using Calabash to test a mobileapplication on one platform - iOS in the case of the previous walkthrough. Writing cross-platformtests is also possible, but requires a bit more effort to abstract away the platform specific parts fromthe shared parts. Android and iOS applications are made up of different UI components with zero overlap in their APIs.In order for the step definitions to be cross-platform, they cannot contain any code that is platformspecific. This means that we have to introduce another layer of abstraction. This layer will be used

by the step definitions and is responsible for driving the UI.

A well-established pattern has evolved in test engineering to support this abstraction - the Page- Object Pattern (POP). The Page-Object Pattern maps a single screen (such as an Android Activity oran iOS View) to a class. This class contains methods and properties that can be used to manipulatethe screen (such as pressing a button) or to query the state of the screen (checking to see if some

http://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/calabash-query_syntax/#Predicatehttp://localhost/var/www/apps/conversion/tmp/scratch_2/guides/testcloud/calabash/calabash-query_syntax/#Predicate


11/14

text is present). This pattern changes the focus of the test from “how to drive the screen” to “how to

test a screen”. In other words, our tests will follow a declarative style, while the page objectsthemselves will become more imperative. This abstraction makes it possible to share the features,scenarios, steps, and step definitions, because the platform specific details are encapsulated withinthe page objects. The Page-Object Pattern provides the following benefits:

Reuse – If there are other screens that require a login, the details for doing so are encapsulated

in an easy to use class.

Encapsulation – Different page objects can be created for Android and iOS. This allows the

steps and step definitions to remain the same across all platforms as the platform specific details

are in the page object.

Readability / Maintenance – The code inside the step definitions is reduced. Step definitions

are no longer cluttered with implementation details such as how to scroll a view or touch a

button. This makes them easier to understand and maintain.

A good example of the Page Object Pattern is a login screen. A typical login screen contains ausername, password and a Login button. A page object for the login screen would contain alogin(username, password) method that would abstract the details of entering the login credentials andpressing the Login button. The step definition calls the login method on the page object. The following diagram illustrates how the various components the Page Object relate to each other:


12/14


13/14

The features directory is inside the solution folder, which still contains the step_defintions andsupport directories. There are two new directories, ios and android, which hold the platform-specific

code. The following table describes the content of these folders:

The config directory holds the Cucumber configuration file cucumber.yml. This file contains theCucumber profiles . A Cucumber profile is a shortcut to commonly used Cucumber command lineparameters. The profile will tell Cucumber what files to load before the tests run. See the Cucumberwiki for more information about profiles. Now that we have seen an example of how to create the directory structure for cross platformtesting, let’s make some page objects.

Directory Description

helpersThis directory holds platform specific shared code that would be used by thepage objects.

pages This directory holds the page object implementations.

support This directory holds code used by Cucumber.

https://github.com/cucumber/cucumber/wiki/cucumber.ymlhttps://github.com/cucumber/cucumber/wiki/cucumber.ymlhttps://github.com/cucumber/cucumber/wiki/cucumber.ymlhttps://github.com/cucumber/cucumber/wiki/cucumber.ymlhttp://localhost/var/www/apps/conversion/tmp/scratch_2/creating_calabash_tests/Images/image6.png


14/14

Writing the Page Object in Calabash Calabash provides a Page Object base class for both Android and iOS, Calabash::ABase and

Calabash::IBase. Each screen in the application should have one page object, which is a subclass ofone of these two classes. A page object provides a method called trait. This method returns a Calabash query string that in

turn returns an object unique to this screen. The page object in Calabash calls this method when itneeds to determine if the current screen corresponds to a given page object The following code snippet is an example of a page object called TaskDetailsScreen: require 'calabash-android/abase' class TaskDetailsScreen

Apart from the mandatory trait method, the page object needs additional methods that return querystrings for the important objects on the screen. The page object should also have methods thatcorrespond to the actions a user might make on the screen. For example, the user may touch a text field, enter some text, and then tap a save button. Including

these actions in the step definition will unnecessarily clutter the step definition. It is better to place allthese actions inside a method of the page object class. This allows the step definitions to be briefand focus on what is being tested rather than how to do the test. If you want to see an example of the Page Object pattern, check out the TaskyPro-Calabash

example on GitHub.

Summary This guide began with an overview of what acceptance testing is. It then went on to provide a briefexplanation of what Calabash is, and how it can be used create acceptance tests. We created twosample scenarios to test a sample application. In the first scenario, we used the predefined stepsthat come with Calabash, while in the second scenario we created custom steps. In the process ofwriting these tests, we introduced the Calabash API and Calabash Query language. We closed with

a description of the Page Object Pattern, and how to use it to write cross platform tests.
https://github.com/calabash/calabash-android/blob/master/ruby-gem/lib/calabash-android/abase.rbhttps://github.com/calabash/calabash-ios/blob/master/calabash-cucumber/lib/calabash-cucumber/ibase.rbhttps://github.com/xamarin/mobile-samples/tree/TaskyPro_Calabash/TaskyProhttps://github.com/xamarin/mobile-samples/tree/TaskyPro_Calabash/TaskyProhttps://github.com/xamarin/mobile-samples/tree/TaskyPro_Calabash/TaskyProhttps://github.com/xamarin/mobile-samples/tree/TaskyPro_Calabash/TaskyProhttps://github.com/calabash/calabash-ios/blob/master/calabash-cucumber/lib/calabash-cucumber/ibase.rbhttps://github.com/calabash/calabash-android/blob/master/ruby-gem/lib/calabash-android/abase.rb

Creating Calabash Tests

Documents

Transcript of Creating Calabash Tests