We're accepting new projects!
or by  
We're accepting new projects!
oksanapylyp

(1)

Test coverage for iOS UI automated tests

21/12/2015
Test coverage for iOS UI automated tests

Making mistakes is a peculiar feature of each individual. Perhaps, there is no person in the world to do everything well at first try. This characteristic concerns all the fields of our life: from writing the first words up to choosing a profession and so on. 

Usually, creating the program code is not done without mistakes as well, what results in application failures. This happens because all developers naturally make mistakes from time to time, and are not objective enough to test their own product neatly. That is why the independent team of testers, having completely different approach to the software testing, is the extremely important aspect of a successful project.

Automated testing: what is it?
Pros and cons for mobile testing

Testing automation is the process of writing scripts or using specialized software, which performs testing automatically.

Advantages:

  1. Time saving — if the project is big enough and development of future versions of a certain program is planned, writing the automated tests will considerably save the time for testing, as the existing tests can be only rewritten and some new ones added.
  2. Continuous Integration — usage of such systems to launch the automated tests will allow to get the full report about the current state of the products.
  3. Cross-platform — in most cases test scripts can be launched for all OS types. For instance, if we write automated tests for iOS mobile application, we can launch them on all required models of iPhone and iOS versions.

Disadvantages:

  1. Irrelevance to small projects — writing the automated tests is time consuming enough, thus if application is small and doesn’t contain any complex functionality, more time can be spent to write the automated tests then the application itself. That is why automated tests are reasonable to be applied only to mid-size and big projects.
  2. Familiarity with the programming languages — only those testers who know certain programming languages can be involved in such type of testing. This fact certainly proves that a qualified tester is not a person “too weak” to become a developer.
  3. Cost — naturally, automated testing is much more expensive than the manual one. QA Automation Engineers services are often equal to developers work.

What does the statistics say?

The smartphones market grows every day. New models, versions and companies appear. But, as in any other sphere of business, leaders make the most out of it. This year they are as follows: Samsung, Apple, Huawei, Xiaomi and Lenovo.

ios ui automated tests gr

As we may see, the percentage of Android devices is much higher than of Apple ones:

ios-ui-automated-tests-growHowever, the products of Apple are being sold much better, than smartphones of each separate company. In China, for instance, Apple is the undeniable leader:

graph-ios-ui-automated-tests

Instruments for writing the automated tests for iOS

Appium — open source test automation framework for testing native and hybrid applications and mobile web applications. It works with iOS applications by means of Selenium WebDriver API and language-specific client libraries.

appium-ios-ui-automated-testsAdvantages:

  1. Can be used for both iOS and Android apps automated testing;
  2. No need in SDK connection and having access to project source code;
  3. Tests can be run on both simulator and mobile devices;
  4. Support of many different programming languages (Java, JavaScript, C#, Ruby, Python and others);
  5. Possibility of connection to continuous integration systems;
  6. Has its own inspector, which allows to get information about components of the screens.

Disadvantages:

  1. Problems with the inspector appear from time to time: it doesn’t recognize all the components, displays incorrect xpath;
  2. Possibility to switch between a few apps and sessions is not built-in;
  3. Tests are performed slowly as the server operates indirectly via the additional REST server.

Automation — framework developed by Apple for testing iOS mobile applications.

Advantages:

  1. Testing scenarios are performed outside of the application and modulate user interaction with the help of UI Automation API;
  2. Automated tests may be launched on the simulators and real devices;
  3. The result of tests is the log file containing information about successfully passed tests, and the ones that failed;
  4. Built-in possibility of integration with other tools for testing, e.g. for the measurement of memory leaks and detection of low productivity causes;
  5. Has the built-in recorder, with the help of which automated tests may be quickly recorded and edited;
  6. May be launched in a few ways, for example via Instrumentals or the command prompt (Terminal);
  7. Tests may be launched changing the location.

Disadvantages:

  1. Tests must be written in JavaScript, thus testers may need to learn additional programming language;
    Works only with the apps signed by a development provisioning profile;
  2. Does not work with an app signed by distribution provisioning profile;
  3. Launching tests do not prevent the phone from locking, that is why Auto-Lock setting “Never” should be applied before test is launched;
  4. Possibility of connection to the Continuous Integration system without connecting additional frameworks is not included.

To write the tests both Appium and Automation tools may be used (to launch Automation you need to go to “Product/Profile/” and choose “Automation” in the “Instruments” window). In our case it was better to use Automation as all the tests in it are written with the help of Javascript and are launched much faster than with the help of Appium (which works indirectly via the additional REST server).

How to present the results of automated
UI tests of the iOS application?

Writing the automated tests with the pre-set task to measure the test coverage has been initiated by client’s demand to present him the “nice and clear” result of the completed work, meaning at least 80 percent of the code was verified. Investigation of this issue started exactly from the question of how to combine test coverage with UI tests. At that time such solution for the Android app has been found by our team. Fortunately, Apple has already thought of this issue and developed the framework containing such function. However, specific settings had to be applied.

Of course the code itself is needed to measure it’s test coverage. For this reason, the first step was to get the access to the source code of the project and have the basic skills of operating with XCode. The next step was to apply the following settings:

Turn on “Instrument Program Flow” and “Generate Test Coverage Files” in the general setting of the build (select “Yes”):

ios ui automated tests 1

Open the project folder and in the “Info.plist” file add the following line: “Application does not run in background”, choose “Yes”

ios ui automated tests in bacBoth conditions are OBLIGATORY for the creation of files displaying the test coverage.

Tests launch is performed either via the Automation application UI in Xcode or starting the command in the prompt:
instruments -t ‘/Applications/ Xcode.app/ Contents/ Applications /Instruments.app/ Contents/ PlugIns/ AutomationInstrument.xrplugin/ Contents/ Resources/ Automation.tracetemplate’ -w ‘device name’ ‘/Users/PC name/Library /Developer/ Xcode/ DerivedData/ project name- /Build/Products/Debug-iphonesimulator/application name.app’ -e UIASCRIPT ‘/Users/PC name/Documents/Main.js’

After tests have been finished, the results may be viewed.
One should go to “Window/Projects”, choose the current project and click on the right arrow located in front of “Derived Data”.

ios-ui-automated-tests-snapshots

After that go to “Build/Intermediates” and choose the current version of the build, open “Object-normal” format and system bit (х86 or х64). All the files created after project compilation, namely the test coverage, should be stored in the last folder in “.gcda” extension.

To open these files we used the Cover Story application.

ios-ui-automated-tests-list

Test coverage may be displayed as on the picture above, or by exporting to HTML file (“File/Export to HTML”). After the export we open the “index.html” file in the folder, created after the export. It contains the detailed report on the test coverage.

ios-ui-automated-tests-conclusions

 

Conclusions

While performing the UI tests for iOS, like in case with Android mobile testing, we did not find the ready-made solution for measuring the test coverage. But ready-made frameworks for measuring Unit tests exist, for this reason both iOS development tools and automated testing were involved. By applying our team’s experience from this article, you’ll be easily able to present the clear visual result of your work to the client.