Archive for category Tools

XCTest with Xcode 6

These are my notes from the 2014 WWDC Session 414 Testing with Xcode 6

What is covered

  • Benefits of testing
  • Getting started and adding tests to old projects
  • Asynchronous testing
  • Catch performance regressions


Why Test?

  • Find bugs
  • Hate regressions
  • Capture performance changes
  • Codify requirements – tests help to codify the range of functionality of code.
  • The tests themselves will tell other engineers what expected behaviour is.
Add tests to an existing project
In a new project
  • You can write tests first
  • Write code that passes those tests
  • This is known as Test driven development

Continuous integration workflow

  • Start off in a green state
  • Green state represents known quality
  • Add features, fix bugs, add new tests
    • At some point a test will fail
    • This will be flagged immediately

Xcode testing with Framework XCTest

  • Expectations passes/failures
  • To create tests subclass XCTestCase
  • Implement test methods
  • Return void, method name starts with “test”. Rest of name should describe purpose of test
    • -(void)testThatMyFunctionWorks
    • Use assertion APIs to report failures
    • XCTAssertEqual(value, expectedValue); // Compares two scalar values
    • If a failure, outputs a string and reports a failure to the test harness.
    • Xcode has test targets for managing tests.
  • Test targets build bundles
    • Contains the compiled test code
    • Resources you use in the test
    • These go in the test bundle, not in the application
    • Test targets are automatically included in new projects
    • You can have as many test targets as you want so you can break tests up into groups.
  • Test bundles
  • Test hosting
    • Tests are hosted in an executable process
    • The tests are usually injected into your app.
    • That means the tests have available all the code in the application.
    • Alternatively you can run the tests in an hosting process provided by Xcode
    • When you go to load resources for tests the resources are not in the main bundle
      • Don't do [NSBundle mainBundle]
      • Instead: [NSBundle bundleForClass:[MyTest class]]
  • Running tests
    • Simplest way is Command-U
      • Takes the active Scheme and runs the tests for that scheme
    • In the editor window gutter there is also run test buttons. You can run a single test, all tests in a test class
    • There is a similar set of buttons in the test Navigator
    • You can also run tests using xcodebuild
      • You can create your own automation setup
      • xcodebuild test -project ~/MyApp.xcodeproj -scheme MyApp -destination 'platoform=iOS,name=iPhone'
    • Where are the results displayed
      • The test navigator where you’ll have green/red checkmarks against each test
      • In the issue navigator
        • You’ll get the failure, and the reason for the failure.
      • In the Source editor gutter.
      • In the test reports, which show all tests that are run and associated logs.
    • Demo – Add tests to an existing project
    • One major point. Keep tests simple so it is clear why the test failed.
      • E.g. If your testing parsing data from the internet. You don’t want to test internet access.
      • Download & save the data and add the saved data to the test target.
      • Then load the data from a file.
  • Asynchronous Tests
    • More and more APIs are asynchronous
      • Block invocations
      • Delegate callbacks
      • Make network requests
      • Background processing
    • Unit tests run synchronously so this creates a challenge
    • XCTests adds the “Expectation” objects API to Xcode 6 testing
      • The expectation object describes events that you expect to happen at some point in the future
      • - (XCTestExpectation *)expectationWithDescription:(NSString *)description
    • XCTestsCase waits for expectations to be fulfilled.
      • - (void)waitForExpectationsWithTimeout:(NSTimeInterval)timeout handler:(XCWaitCompletionHandler)handlerOrNil
      • This will wait until the timeout interval is complete or when all events have been fulfilled
      • Testing opening of a document asynchronously:
- (void)testDocumentOpening
  XCTestExpectation *expectation = [self expectationWithDescription:@"open doc"];
  IDocument * doc = ...;
  [doc openWithCompletionHandler:^(BOOL success) {
    [expectation fulfill];
  [self waitForExpectationsWithTimeout:5.0 handler:nil];

See the demo code for writing an asynchronous test. Basically look for earthquake parser.

Performance testing

  • It can be easy to introduce performance regressions in code.
  • Catching these regressions is difficult.
  • Performance testing allows you to automate this
  • New APIs to measure performance
  • New Xcode UI to interpret results
    • Can profile tests with instruments
    • Use the new measure block API `- (void)measureBlock:(void (^)(void))block;`
    • Takes a block of code and runs it 10 times and shows the results in Xcode


  • See the demo code for testing performance. This is the mac version of the eartquake parser.
    • Call -measureBlock: to detect performance regressions
    • View results in Source Editor and Test Report
    • Profile tests with instruments.
    • Setting Baselines – Needs a fix point to compare against
    • Standard deviation – How much variation is allowed before a performance is considered to have failed
    • Measuring precisely
    • Baseline is the Average from a previous run
      • Set Baseline Average to detect regressions
      • Fail if > 10% increase from Baseline Average
      • Regressions less than 0.1 seconds are ignored – to remove false positives
    • Baselines are stored in source
    • Baselines are per-device configuration
    • Include device model, CPU, and OS


Tags: , ,

Documentation markdown to html

The branding in relation to the documentation should be minimal but clear. Each documentation page should have a header which has the branding and then some navigation options to move around documentation. A static list that is the same for every page.

Documentation is all in markdown, github/gitlab flavoured markdown.

These are the lists of tools I looked at for generating/hosting customised documentation and the issues I had with them.

  1. GitHub pages with/without Jekyll – Removed from consideration. The stench is too great.
    • Takes markdown as input +
    • Aimed at blogging rather than documentation –
    • I believe it could be forced into a usable documentation system but it would be tricky. –
    • The .io domain for ones without custom domain names. Supports Diego Garcia’s theft. Why would you even choose this. –
    • The guy at github who put it together is the problematic github founder
  2. Sphinx Looks promising
    • Takes .rst rather than markdown –
    • Produces static searchable documentation from markdown +
    • Branding seems possible from css and templates +
    • Installation (I did get it to install, but flaky) –
      • Using MacPorts. I’ve wasted hours in the past trying to get home-brew and MacPorts installed. I’m reluctant to try again
      • Alternate suggested is python easy_install
      • I installed easy_install, which wasn’t quite an easy install
      • I ran sudo easy_install sphinx which seemed to work with some warnings like:
        • warning: no previously-included files matching ‘*.pyc’ found under directory ‘jinja2’
      • I ran sudo sphinx-build as in the install instructions
        • I get a reply: “Error: Insufficient arguments.”
        • Does this mean that my sphinx installation is not yet built? Actually I think that is the actual sphinx command for building documentation. Instructing the user to run that on the install page without actually informing the user that running of the command was purely to confirm sphinx is installed was confusing.
        • Running sphinx-build without sudo produces same result
        • Running sphinx-build –version returns: Sphinx (sphinx-build) 1.2.2
    • It is python based. If something goes wrong and I haven’t used python then I’m dependent on others –
  3. Yard
    • It takes markdown files (using .yardopts file), it certainly sucks in a Readme markdown file & process correctly +
    • I use it already for generating ruby reference documentation. +
    • I’ve not been able to get it to suck up additional markdown files. –
      • I’ve tried the command line option –files
      • I’ve tried list files after: ” – ” on the the command line
      • Neither of the above work, I’ve tried both with simple markdown files
    • Adding templates for branding the documentation looks non-trivial. –
  4. Pandoc Serious consideration
    • Provides an opportunity to produce documentation in almost any format imaginable from markdown +
    • Output can be customised using templates plus headers/footers and css +
    • Can generate table of contents (toc) from markdown +
    • Work needed to write the templates and css needed to get correct branding
    • I’d be capable of this but it will require time for me to do, due to lack of experience

I’m going to go with pandoc, installation trivial, reads markdown, outputs in a huge variety of formats, customizable. As stated the only issue is getting my head around how the templates work. I wish I’d spent time just getting pandoc templates/headers/footers and css files written rather than trying to find a more user friendly tool which doesn’t appear to exist.

SSD versus HDD, Movie Frame Grabs and the importance of profiling

There was an e-mail to Apple’s cocoa-dev e-mail list that provoked a bit of discussion. The discussion thread starts with this e-mail to cocoa dev by Trygve.

Basically Trygve was wanting to get better performance from his code for taking frame grabs from movies and drawing those frame grabs as thumbnails to what I call a cover sheet. Trygve was using NSImage to do the drawing and was complaining that based on profiling his code was spending 90% of the time in a method called drawInRect.

Read the rest of this entry »

Tags: , , , , , , , ,

Git submodules

At this point I’ve decided not to go down the Cocoapods route to manage my project dependencies. This has come up now because I’ve added the first external code to my project and the installation instructions was basically use Cocoapods.

I’m no Cocoapods hater, just that it adds another layer of complexity that makes things easier when it works but adds one more step that can wrong with the toolchain. If I decide to use Cocoapods I will make the effort to properly understand how it works so when it does go wrong I’m not scrabbling around in the dark.

So after a recommendation from @abizern to try git submodules I’ve decided to give that a go.

The external code is a logging framework called CocoaLumberjack.

At the top level of my git repository I want to create a folder called CocoaLumberjack which is where the submodule will live.

git submodule add git:// CocoaLumberjack

The above command creates the CocoaLumberjack folder and downloads the current state of the CocoaLumberjack project. It also creates a “.gitmodules” file if you’ve not previously added a git submodule to your project and adds an entry for the CocoaLumberjack submodule. It then adds the CocoaLumberjack folder to your own repository, this will not add the submodule files to your git repository, and then adds or update the “.gitmodules” file.

After pushing everything to your remote, other clients of that remote should do a git pull like normal which will create the folders for the submodules but not check them out. Within the folder for each submodule you should run the commands:

git submodule init
git submodule update

Obviously if the CocoaLumberjack project is updated and you want the latest revision then git submodule update will need to be run at the top level in each repository.

You can update all submodules at once if needed using the following command when you are in the top level folder of the repository containing the submodules.

git submodule update --init --recursive

I can now go and add the necessary files to my project and away I go.

From Abizer’s comment below, to clone a remote repository which contains submodules you can do the following:

 git clone –-recursive

I’d also recommend reading his blog post that he linked to.


Tags: , ,