It has been a little over a year since I started working on the MovingImages project. I wanted to build something that was similar to “iMagine Photo” which I created over 10 years ago, but with more functionality and provide the users writing scripts with a greater choice of scripting languages to use to drive MovingImages.

MovingImages is a scripting tool for processing images and moving frames. The previous tool iMagine Photo was restricted to AppleScript, MovingImages can be scripted from any scripting language that can call a command line tool. Any Macintosh running OS X 10.9 Mavericks comes with the scripting languages ruby, python, AppleScript or any of the shell script languages (sh, bash, zsh tsch etc.). Another requirement is that MovingImages is simple to install.

iMagine Photo was built to take advantage of Quicktime for processing images and movie frames and to apply Quicktime filters and used Quickdraw and CoreGraphics for drawing. MovingImages also uses CoreGraphics (Quartz) for drawing along with CoreText for drawing text and CoreImage to apply image filters and do image processing. It hasn’t been implemented yet but AVFoundation will be used for processing movie frames so that importing and editing movies will be a central part of the tool and will go beyond the functionality provided by iMagine Photo. I’m currently working on a refactor of the programming interface to MovingImages so that multiple tasks can run concurrently. The core part of the code was written from scratch with concurrency in mind so this task is progressing well. From observing that CPU usage remains fairly low when handling MovingImages commands sequentially if I want to maximise throughput when processing images then MovingImages needs to be able to handle concurrent commands.

To get started with this project I needed to learn a number of Apple technologies that I’d been missing out on in my job and I didn’t have the opportunity for catching up on outside of work hours, so as you will see below there was a lot of catching up to do.

My first blog post was a note about github and looking at project management, task and bug tracking software. I tried Trac, but I’m not sure what happened along the way because for MovingImages I’m using gitlab. Gitlab works for me as it provides a small number of free private git repositories plus a simple wiki and simple project management tools that have been easy to learn. I may find that the simple project management tools are too limited but at present they do what is needed.

The first thing I decided to catch up on was ARC (Automatic Reference Counting). I wrote a blog post based on my notes from reading the ARC documentation and watching the ARC WWDC videos. A few weeks later I wrote a simple command tool to explore in what situations autorelease is used when objects are created and passed around. I now feel more confident as to where autorelease pools are needed. The ARC code is here, and the blog post about ARC and autorelease pools is here. I never took to garbage collection, I just didn’t trust it, but with ARC, it feels like a perfectly workable approach to memory management.

At this time I hadn’t got beyond using Xcode 3.2.6 and Apple had already deprecated the Xcode 3 to Xcode 4 transition guide. So after tracking it down I kept a copy of the pdf for myself and created a tiny blog post linking to the pdf. If the hits I’m getting for this document is anything to go by there is clearly still plenty of people in need of it and like me they are finding it difficult to track down on Apple’s website. The first Xcode 5 beta came out a couple of months later in June but the differences between Xcode 4 and 5 were not as significant as the big changes between Xcode 3 and Xcode 4. Because MovingImages development hadn’t started by the time the Xcode 5 beta was released I felt it was safe to move to Xcode 5 as my development tool from the first beta.

So, after ARC, and Xcode 3-4-5 transition, the next technology I felt I needed to pick up was blocks.

I then decided to have my first quick look at AVFoundation. Ten days later I followed up that intro to AVFoundation with a couple of command line tools which are mostly reworking of Apple sample code and a Commander tool that makes it simple to configure the use of the command line tools.

And then onto the new NSArray, NSDictionary and NSNumber literals.

One thing not captured in the blog post linked to above and the links within that post is that there can be issues with Objective-C++ and static file objects. C++ constructors that call objective-c code for the creation of file static objects is problematic. The implementation of NSDictionary, NSNumber and NSArray literals in some way hides that they are syntactic sugar for actually calling objective-c methods.

Then onto learning about the changes made to class extensions in Xcode 4.3 that make it a lot easier to keep the public interface of a class nice and small. Also note the extra little refinements added in the Xcode 5 update. This is really useful, though I’ve decided on a different approach for what I call base objects in MovingImages. The base object classes also conform to a couple of protocols and you can choose to interact with the objects via the protocols only which keeps the API very small.

And after a very brief detour onto deployment targets and specified SDKs, I found the introduction of explicitly typed enums to be yet another good move. One of the things I have noticed with reading up about all these language changes is just how much it has improved programming with Objective-C.

After the earlier introduction to blocks, I next blog about using blocks and grand central dispatch (gcd) to offload work from the main thread and a typical pattern to use to update the UI on the main queue. The comment about don’t allow too many background threads to become blocked is an important one. 7 months after writing the linked blog post I was playing with a small command line tool where I was trying to maximise the rate at which we could get frame grabs from a video. When there were too many queues, and too many concurrent tasks the program stalled.

And more on blocks. This time on retain cycles, an object that has a strong reference to a block and a block with a strong reference to the object. I like the first solution where the block captures a weak reference to the object, and within the block assigns a strong local variable to the weak one and tests for nil. This may not be the desired behaviour in all cases so it is sensible to keep the second solution in mind.

Another compiler change that I like that has been made available with Xcode LLVM 5 and targeting in 10.9 is modules and then there is return type correctness and the instancetype keyword.

Then a brief introduction to tagged pointers, implicit bridging and the @encode compiler directive. All nice little additions.

I next return to blocks and grand central dispatch in my write up of the video of Apple’s 2012 WWDC Session 712 on Asynchronous Design Patterns with Blocks, GCD and XPC. This session was quite important to me at the time as I was coming to the conclusion that MovingImages was going to be a LaunchAgent and that I was going to communicate with the LaunchAgent using XPC which turned out to be exactly what I did. XPC is basically a way to find services to communicate with and a way to send messages. The section on XPC was focussed on including an xpc service within your application bundle that you would use so some further details were necessary to get things to work with communicating with a LaunchAgent. Getting that extra information in a coherent form proved to be quite difficult as the documentation assumed you were working to build an xpc service and the extra bits for the LaunchAgents were just asides.

The next blog post is my notes from watching the 2011 WWDC Session 308 video called Blocks and Grand Central Dispatch in Practice. This is a fairly practical session demonstrating the use of blocks and grand central dispatch design patterns. I did notice some flaws in the sample code that came with the session and using dispatch_apply.

And yet another blog post based on notes from a WWDC session. This time the 2012 WWDC Session 308 Cocoa Interprocess Communication with XPC. This session goes into more detail about what XPC is and how to get it working.

The blog post linked above looks like it helped me along the way because 12 days later my next blog post is not notes from a WWDC session video, but about the issues I had with getting a LaunchAgent up and running with links to an example project. There does seem to be quite a lot of grumbling about the state of the documentation. I do believe I have a couple of documentation bugs reported to Apple from this time which have not been closed.

It now looks like I’m starting the work on writing MovingImages but I seem to be anxious about the coding conventions, because my next blog post is from reading Apple’s Cocoa coding convention guidelines and trying to find a way to make me remember the conventions.

Then there is a bit of a hiatus with blog posts and I think this is when I am properly getting moving with writing MovingImages. I seem to have been diverted with issues related to privilege helpers but then I do need privilege escalation when I come to install the command line tool that communicates with MovingImages. So I have a short blog post which is basically a few links to blog posts by Steve Streeting on privilege escalation.

The next blog post is on problems with code signing. Well you know from above about the compiler improvements and Objective-C language improvements. Well code signing is where Apple takes all that away. I must be thinking about distribution.

And more grumbling about documentation. So clearly along with code signing, the documentation is the next place where Apple takes away. There is information that is missing in relation to ImageIO which is the library used for importing and exporting image files. In particular the limitations with the delay time property when exporting gif animations is not documented and this blog post is partly about documenting what those limitations are from trial and error.

I now have a tool to drive, because I can see I have already got sick of writing unix shell scripts and I have a collection of blog posts related to the scripting language ruby. The first of which is about which scripting language to choose, the next is my introduction to the features of ruby that are important to my needs. I then move onto an issue I found disconcerting in ruby, that for simple ruby hashes (Objc dictionaries) then members of the hash are stored by value or a copy is made, but as soon as a hash becomes complex then members are stored by reference. If you change the reference in one place you might see that change somewhere else.

I’m then onto writing the installer and to have the LaunchAgent available to use immediately after installation, and to remain available after the computer restarts understanding how the launchctl command was essential. If the length of notes about the “-w” switch is anything to go by I must have wasted a lot of time thinking it was going to help before experience demonstrated otherwise.

The next blog post is about deciding on a timer class and adding the MSWeakTimer class to the project. This is the first bit of code I’ve added that isn’t Apple code or code I’ve written. Since this code is used in the installer and not the MovingImages LaunchAgent I feel less anxious about using it. Even so it was still quite a step for me, using external code that won’t have anything like the breadth of user base that Apple’s code does to give me the confidence that it will behave as promised.

And then the first Alpha release of MovingImages 0.1a.

You’ll have probably have noticed that there is nothing about tests so far, and that is because well, hang my head in shame I haven’t done any tests up to this point of the project. So at this stage I started looking at the unit tests that come integrated with Xcode, and at system tests. I decided to use system tests using ruby for MovingImages. That was as of December 2013, I will soon be writing modules for using MovingImages in ruby and I think taking a test driven development approach could be the right approach.

With the first alpha release done, which was then followed up with writing tests and the reference documentation I then move onto reading up about CoreImage. So here are my notes from watching the WWDC 2011 Session 422 Using CoreImage on iOS and Mac OS X.

The first lot of tests that I wrote had no way of comparing the generated images against reference images so the first tests consisted of testing everything else except image rendering. For some reason I had problems with getting ImageMagick to install so I couldn’t use it’s image comparison tool. Whilst reading up about CoreImage it became clear that I could write my own command line tool for comparing images. This tool doesn’t worry about what file formats an image has been saved as or the metadata. All it does is compare the rendered image pixel values and if the pixel with the biggest colour difference is below a specified value then two images are considered to be equal. The threshold value is a command line option. Any image rendering using CoreGraphics and CoreText produces reproducible images, CoreImage filters though is different, depending on which graphic card was used to apply the filter, or whether a software render was used. Even more confusing was that a graphics card which was already under load could produce different output from the same card which wasn’t. The compare images command line tool threshold value option was helpful.

I then wrote a Chroma Key CoreImage filter and then wrote a small command line tool to demonstrate using the Chroma Key filter.

Then onto some brief ruby notes and a couple of links to ruby blog posts.

I needed better logging functionality than that provided by using NSLog. The CocoaLumberjack framework was the framework I decided on. Looking back I believe this was absolutely the right decision. CocoaLumberjack just assumes that you are going to add it to your project using CocoaPods. Perhaps I’m foolish but I am not comfortable with tools like dependency managers. They are great while they work and Cocoapods certainly looks to be great and can be time savers, but I’m never going to have the time to find out how it works which leaves the option open that if something does go wrong and then I’m going to take an age trying to fix whatever the problem is. So the helpful @abizern suggested using git submodules and for now that is a decision I can be comfortable with.

I then got distracted by a question on Apple’s cocoa-dev e-mail list and I really shouldn’t have as it is not on the critical path. I however did learn something important from this. Resist making assumptions. I had created a command line tool to grab movie frames and create cover sheets in response to the question and had e-mailed a link to the code I’d uploaded to github on cocoa-dev. Another developer took pity on me and suggested that I should make multiple AVAssetImageGenerator objects for a single AVAsset movie file and generate frames to draw to the cover sheets from the AVAssetImageGenerator objects. I resisted this, I mean reading data from a single file the file pointer is going to be sent all over the place and especially so if the file is on a hard disk. This should have real consequences slowing things down or so I thought. The second time it was suggested I decided to let my assumptions go. There is definitely an issue of not having too many AVAssetImageGenerator working at once but I did find that for a hard drive I could get maximum performance with three AVAssetImageGenerator objects generating images at once when the movie file was on a hard drive and five AVAssetImageGenerator objects when the movie file was on a SSD drive. I ended up using 4 working at once for both hard drives and SSD drives because that represented only a small reduction in performance in both cases with significant gains over just using 1. By playing with this stuff I got more hands on experience working with grand central dispatch queues and tried out NSOperationQueues for the first time. I think this exercise was worthwhile in the end but I needed to get back focus on the main task.

Back on track and I hit a brick wall. I can’t get the CIPageCurlTransition filter to work as I expect and I wasted considerable time trying to work out why it didn’t and in the end found out that the problem was the shading input image to the filter needs to be partially transparent and I was passing in an opaque image.

By this point I’m busy writing ruby scripts to demonstrate some of the functionality provided by MovingImages. Of course this gives me an opportunity demonstrate just some of the cool functionality that MovingImages provides.

The Apple documentation for CoreImage states that if you can accept using a linear Generic RGB colorspace for rendering then you’ll get faster frame rates out of CoreImage filters. I started suspecting that this wasn’t the case and actually tested various configurations. The results were completely at odds with what I was expecting. I’m wondering if the difference is I’m saving images to disk, not rendering to the screen where OSX can keep the whole render chain in the graphics card memory and that might make the difference.

And finally on April 24th the second alpha release is out. My god why did it take so long. Here is the Alpha 0.2 release announcement oh and along the way I created an ebook of the Using MovingImages documentation.

Where to for now. I’m working on the refactoring as discussed when I’ve finished that I’ll need to write tests to be used for extending the ruby module and write at least a minimal AppleScript library for using MovingImages. Then I’ll need to write the documentation for using both the ruby and AppleScript libraries. Once that is done I’ll watch Apple’s developer videos on using AVFoundation in preparation for writing the moving importer, and movie editor classes which I’ll be hooking into MovingImages. Then of course it will be extending the ruby and applescript libraries again for the first final release. I’ve got plenty of ideas for adding in extra functionality after that first full release, both to the core MovingImages framework and launch agent, but also for making the interface with scripting languages easier.

This blog post is a long read. I think I need another one which is more of a thoughtful post on experiences rather than this time linear treatment. But I think that is for another day.