These are my notes from the 2014 WWDC Session 414 Testing with Xcode 6

What is covered

  • Benefits of testing
  • Getting started and adding tests to old projects
  • Asynchronous testing
  • Catch performance regressions

Overview

Why Test?

  • Find bugs
  • Hate regressions
  • Capture performance changes
  • Codify requirements – tests help to codify the range of functionality of code.
  • The tests themselves will tell other engineers what expected behaviour is.
Add tests to an existing project
In a new project
  • You can write tests first
  • Write code that passes those tests
  • This is known as Test driven development

Continuous integration workflow

  • Start off in a green state
  • Green state represents known quality
  • Add features, fix bugs, add new tests
    • At some point a test will fail
    • This will be flagged immediately

Xcode testing with Framework XCTest

  • Expectations passes/failures
  • To create tests subclass XCTestCase
  • Implement test methods
  • Return void, method name starts with “test”. Rest of name should describe purpose of test
    • -(void)testThatMyFunctionWorks
    • Use assertion APIs to report failures
    • XCTAssertEqual(value, expectedValue); // Compares two scalar values
    • If a failure, outputs a string and reports a failure to the test harness.
    • Xcode has test targets for managing tests.
  • Test targets build bundles
    • Contains the compiled test code
    • Resources you use in the test
    • These go in the test bundle, not in the application
    • Test targets are automatically included in new projects
    • You can have as many test targets as you want so you can break tests up into groups.
  • Test bundles
  • Test hosting
    • Tests are hosted in an executable process
    • The tests are usually injected into your app.
    • That means the tests have available all the code in the application.
    • Alternatively you can run the tests in an hosting process provided by Xcode
    • When you go to load resources for tests the resources are not in the main bundle
      • Don't do [NSBundle mainBundle]
      • Instead: [NSBundle bundleForClass:[MyTest class]]
  • Running tests
    • Simplest way is Command-U
      • Takes the active Scheme and runs the tests for that scheme
    • In the editor window gutter there is also run test buttons. You can run a single test, all tests in a test class
    • There is a similar set of buttons in the test Navigator
    • You can also run tests using xcodebuild
      • You can create your own automation setup
      • xcodebuild test -project ~/MyApp.xcodeproj -scheme MyApp -destination 'platoform=iOS,name=iPhone'
    • Where are the results displayed
      • The test navigator where you’ll have green/red checkmarks against each test
      • In the issue navigator
        • You’ll get the failure, and the reason for the failure.
      • In the Source editor gutter.
      • In the test reports, which show all tests that are run and associated logs.
    • Demo – Add tests to an existing project
    • One major point. Keep tests simple so it is clear why the test failed.
      • E.g. If your testing parsing data from the internet. You don’t want to test internet access.
      • Download & save the data and add the saved data to the test target.
      • Then load the data from a file.
  • Asynchronous Tests
    • More and more APIs are asynchronous
      • Block invocations
      • Delegate callbacks
      • Make network requests
      • Background processing
    • Unit tests run synchronously so this creates a challenge
    • XCTests adds the “Expectation” objects API to Xcode 6 testing
      • The expectation object describes events that you expect to happen at some point in the future
      • - (XCTestExpectation *)expectationWithDescription:(NSString *)description
    • XCTestsCase waits for expectations to be fulfilled.
      • - (void)waitForExpectationsWithTimeout:(NSTimeInterval)timeout handler:(XCWaitCompletionHandler)handlerOrNil
      • This will wait until the timeout interval is complete or when all events have been fulfilled
      • Testing opening of a document asynchronously:
- (void)testDocumentOpening
{
  XCTestExpectation *expectation = [self expectationWithDescription:@"open doc"];
  IDocument * doc = ...;
  [doc openWithCompletionHandler:^(BOOL success) {
    XCTAssert(success);
    [expectation fulfill];
  }];
  [self waitForExpectationsWithTimeout:5.0 handler:nil];
}

See the demo code for writing an asynchronous test. Basically look for earthquake parser.

Performance testing

  • It can be easy to introduce performance regressions in code.
  • Catching these regressions is difficult.
  • Performance testing allows you to automate this
  • New APIs to measure performance
  • New Xcode UI to interpret results
    • Can profile tests with instruments
    • Use the new measure block API `- (void)measureBlock:(void (^)(void))block;`
    • Takes a block of code and runs it 10 times and shows the results in Xcode

 

  • See the demo code for testing performance. This is the mac version of the eartquake parser.
    • Call -measureBlock: to detect performance regressions
    • View results in Source Editor and Test Report
    • Profile tests with instruments.
    • Setting Baselines – Needs a fix point to compare against
    • Standard deviation – How much variation is allowed before a performance is considered to have failed
    • Measuring precisely
    • Baseline is the Average from a previous run
      • Set Baseline Average to detect regressions
      • Fail if > 10% increase from Baseline Average
      • Regressions less than 0.1 seconds are ignored – to remove false positives
    • Baselines are stored in source
    • Baselines are per-device configuration
    • Include device model, CPU, and OS