Posts Tagged unit tests

XCTest with Xcode 6

These are my notes from the 2014 WWDC Session 414 Testing with Xcode 6

What is covered

  • Benefits of testing
  • Getting started and adding tests to old projects
  • Asynchronous testing
  • Catch performance regressions


Why Test?

  • Find bugs
  • Hate regressions
  • Capture performance changes
  • Codify requirements – tests help to codify the range of functionality of code.
  • The tests themselves will tell other engineers what expected behaviour is.
Add tests to an existing project
In a new project
  • You can write tests first
  • Write code that passes those tests
  • This is known as Test driven development

Continuous integration workflow

  • Start off in a green state
  • Green state represents known quality
  • Add features, fix bugs, add new tests
    • At some point a test will fail
    • This will be flagged immediately

Xcode testing with Framework XCTest

  • Expectations passes/failures
  • To create tests subclass XCTestCase
  • Implement test methods
  • Return void, method name starts with “test”. Rest of name should describe purpose of test
    • -(void)testThatMyFunctionWorks
    • Use assertion APIs to report failures
    • XCTAssertEqual(value, expectedValue); // Compares two scalar values
    • If a failure, outputs a string and reports a failure to the test harness.
    • Xcode has test targets for managing tests.
  • Test targets build bundles
    • Contains the compiled test code
    • Resources you use in the test
    • These go in the test bundle, not in the application
    • Test targets are automatically included in new projects
    • You can have as many test targets as you want so you can break tests up into groups.
  • Test bundles
  • Test hosting
    • Tests are hosted in an executable process
    • The tests are usually injected into your app.
    • That means the tests have available all the code in the application.
    • Alternatively you can run the tests in an hosting process provided by Xcode
    • When you go to load resources for tests the resources are not in the main bundle
      • Don't do [NSBundle mainBundle]
      • Instead: [NSBundle bundleForClass:[MyTest class]]
  • Running tests
    • Simplest way is Command-U
      • Takes the active Scheme and runs the tests for that scheme
    • In the editor window gutter there is also run test buttons. You can run a single test, all tests in a test class
    • There is a similar set of buttons in the test Navigator
    • You can also run tests using xcodebuild
      • You can create your own automation setup
      • xcodebuild test -project ~/MyApp.xcodeproj -scheme MyApp -destination 'platoform=iOS,name=iPhone'
    • Where are the results displayed
      • The test navigator where you’ll have green/red checkmarks against each test
      • In the issue navigator
        • You’ll get the failure, and the reason for the failure.
      • In the Source editor gutter.
      • In the test reports, which show all tests that are run and associated logs.
    • Demo – Add tests to an existing project
    • One major point. Keep tests simple so it is clear why the test failed.
      • E.g. If your testing parsing data from the internet. You don’t want to test internet access.
      • Download & save the data and add the saved data to the test target.
      • Then load the data from a file.
  • Asynchronous Tests
    • More and more APIs are asynchronous
      • Block invocations
      • Delegate callbacks
      • Make network requests
      • Background processing
    • Unit tests run synchronously so this creates a challenge
    • XCTests adds the “Expectation” objects API to Xcode 6 testing
      • The expectation object describes events that you expect to happen at some point in the future
      • - (XCTestExpectation *)expectationWithDescription:(NSString *)description
    • XCTestsCase waits for expectations to be fulfilled.
      • - (void)waitForExpectationsWithTimeout:(NSTimeInterval)timeout handler:(XCWaitCompletionHandler)handlerOrNil
      • This will wait until the timeout interval is complete or when all events have been fulfilled
      • Testing opening of a document asynchronously:
- (void)testDocumentOpening
  XCTestExpectation *expectation = [self expectationWithDescription:@"open doc"];
  IDocument * doc = ...;
  [doc openWithCompletionHandler:^(BOOL success) {
    [expectation fulfill];
  [self waitForExpectationsWithTimeout:5.0 handler:nil];

See the demo code for writing an asynchronous test. Basically look for earthquake parser.

Performance testing

  • It can be easy to introduce performance regressions in code.
  • Catching these regressions is difficult.
  • Performance testing allows you to automate this
  • New APIs to measure performance
  • New Xcode UI to interpret results
    • Can profile tests with instruments
    • Use the new measure block API `- (void)measureBlock:(void (^)(void))block;`
    • Takes a block of code and runs it 10 times and shows the results in Xcode


  • See the demo code for testing performance. This is the mac version of the eartquake parser.
    • Call -measureBlock: to detect performance regressions
    • View results in Source Editor and Test Report
    • Profile tests with instruments.
    • Setting Baselines – Needs a fix point to compare against
    • Standard deviation – How much variation is allowed before a performance is considered to have failed
    • Measuring precisely
    • Baseline is the Average from a previous run
      • Set Baseline Average to detect regressions
      • Fail if > 10% increase from Baseline Average
      • Regressions less than 0.1 seconds are ignored – to remove false positives
    • Baselines are stored in source
    • Baselines are per-device configuration
    • Include device model, CPU, and OS


Tags: , ,

Unit and system testing

I’ve been way behind the curve with looking into whether I should be going down the unit test approach.

I’ve finally started looking into what approach to use for testing during the process of development for MovingImages that will work best.

My first efforts at finding out about using test driven development and Xcode 5 were frustrating. Xcode 5 comes with XCTest but there is very little documentation in relation to unit testing and Xcode 5. I found some Apple sample code which demonstrates the use of unit testing, unfortunately it uses SenTestingKit which was Apple’s former unit testing framework that has been replaced by XCTest in Xcode 5. There is a migrator tool that you can use to move your pre Xcode 5 testing code to Xcode 5. But still not much help with documentation.

Read the rest of this entry »

Tags: , ,