I need to update a protocol that advertises that an object can create a CGImage.

I want to have my movie asset wrapping object conform to this protocol but it means I need a way to pass in more information as arguments than either of the two methods of the protocol allows. The protocol methods currently take an image index which objects conforming to the protocol that only have one possible image just ignore.

When requesting images from a movie we need to be able to specify a time in the movie, and also whether we want the image rendered from all video tracks or just some/one. So I think it is relevant to get rid of the image index argument and replace it with a dictionary which contains properties relevant to the object providing the image.

This means I’ve been revisiting looking at the behaviour of the ImageIO framework and the CoreFoundation object CGImageSource. Like last time I find no way to crop whilst creating a CGImage from a CGImageSource. I still find this a big weakness, if you have a very large image and you want just part of the image at original resolution then you first have to create a CGImage containing the full image and then crop it from there. The Quicktime image importer component allowed you to crop when rendering so this feels like a loss of useful functionality.

CGImageSource provides 2 methods for creating a CGImage. These are:

  • CGImageSourceCreateThumbnailAtIndex
  • CGImageSourceCreateImageAtIndex

The CGImageSourceCreateThumbnailAtIndex function provides a way to generate a scaled image up to the dimensions of the original while CGImageSourceCreateImageAtIndex doesn’t. Both of these functions take an options dictionary as well the image index. The scaling functionality of CGImageSourceCreateThumbnailAtIndex is limited. To generate a scaled image you need to specify two properties of the options dictionary passed into CGImageSourceCreateThumbnailAtIndex.

Swift

let imageSource = CGImageSourceCreateWithURL(...)
let thumbnailOptions = [
    String(kCGImageSourceCreateThumbnailFromImageAlways): true,
    String(kCGImageSourceThumbnailMaxPixelSize): 2272.0
]
let tnCGImage = CGImageSourceCreateThumbnailAtIndex(imageSource, 0, thumbnailOptions)

The property with key kCGImageSourceCreateThumbnailFromImageAlways with a value of true means that any thumbnail image embedded in the image file is ignored and the image generated is from the full size image. If you don’t do this and the image file contains a thumbnail image then the max size you specify is ignored and you get the embedded thumbnail image, this is rarely what you want.

The property with key kCGImageSourceThumbnailMaxPixelSize value is the maximum size in pixels of either the width or height of the generated image. There is a limitation, in that you can’t scale up an image only scale down using this option. If you specify a size greater than both the width and height of the image then the generated CGImage will have the same dimensions as the original.

For very large images (I played with images of 21600 x 10800 pixels) on my iPad Mini 2 creating a CGImage at full size using CGImageSourceCreateThumbnailAtIndex failed, whereas CGImageSourceCreateImageAtIndex succeeded, CGImageSourceCreateThumbnailAtIndex successfully created the image on the simulator and OS X. If the options dictionary does not contain the kCGImageSourceThumbnailMaxPixelSize property then the function CGImageSourceCreateThumbnailAtIndex will create an image at full size up to a maximum size of 5000 pixels on OS X and iOS.

CGImageSourceCreateImageAtIndex takes different dictionary options. Peter Steinberger recommends against setting the kCGImageSourceShouldCacheImmediately to true but that was in 2013, the situation may have changed. I’ll add an addendum to this when I know for sure. There is little documentation for the kCGImageSourceShouldCache property so I’d only be guessing as to what it does exactly. On a 64bit system this value is true by default and false on 32bit systems. Leaving this with its default value is probably best.

Creating CGImages on OS X from an image with dimensions: 21600 x 10800 using CGImageSourceCreateThumbnailAtIndex

  • 1.87 seconds to create a thumbnail image with a width of 21600 and height 10800 pixels.
  • 0.81 seconds to create a thumbnail image with a width of 10800 and height 5400 pixels.
  • 0.54 seconds to create a thumbnail image with a width of 2800 and height 1400 pixels.

Drawing the scaled down CGImage in OS X

Creating the 2800 wide CGImage and drawing the image to a bitmap context took 0.57 seconds.

Drawing the thumbnail image to the context took 0.010506 seconds the first time and 0.00232 seconds for subsequent draws.

Creating CGImage using CGImageSourceCreateImageAtIndex and drawing the image to a bitmap context on OS X

Creating a image using CGImageSourceCreateImageAtIndex doesn’t allow scaling of the image so we will always get the full size image, in this case 21600 x 10800.

  • Creating the image for the first time took 0.000342 seconds
  • Subsequently creating the image took 0.00006 seconds
  • Creating and drawing the image for the first time took: 1.67 seconds
  • Subsequent image drawing took 0.15 seconds.

Creating the CGImage using CGImageSourceCreateImageAtIndex doesn’t decompress the data, this doesn’t appear to happen until attempting to draw the image.

Creating CGImages and drawing in iOS on an iPad Mini 2

The 21600 x 10800 image was too large to render at full scale on iPad Mini 2. Couldn’t create a bitmap context big enough. Same goes for the 12000 x 12000 image.

Creating and drawing a 2800 x 1400 thumbnail CGImage on the iPad Mini 2 to a bitmap context

  • Creating the thumbnail took 0.7 seconds
  • Drawing the thumbnail image took 0.030532 seconds for the first draw
  • Subsequent draws took 0.0052 seconds
  • Creating and drawing the thumbnail image took 0.71 seconds

The time taken to generate the 2272 x 1704 thumbnail CGImage from the 2272 x 1704 image file is 0.001169 seconds.

Drawing the non thumbnail 2272 x 1704 CGImage to the bitmap context on the iPad Mini took 0.163412 seconds to draw the first time and 0.11 seconds for subsequent draws. If the CGImage has to be generated each time then the time taken is: 0.117 seconds.

Conclusions in relation to using CGImageSource

  • Use CGImageSourceCreateThumbnailAtIndex where possible but beware of its limitations.
    • You need to know the max size you want (width or height)
    • For very large images CGImageSourceCreateThumbnailAtIndex can fail when creating large CGImages
    • To generate a CGImages at the size of a very large original image use CGImageSourceCreateImageAtIndex instead
    • You can’t scale up
    • The CGImage generated by CGImageSourceCreateThumbnailAtIndex is not cached so requesting it a second time isn’t any faster
    • Cropping is better done using a CGImage created using CGImageSourceCreateImageAtIndex and then calling CGImageCreateWithImageInRect
    • If you are generating a CGImage that has any dimension greater than 5000 pixels it is better to use CGImageSourceCreateImageAtIndex especially on iOS
  • Advantages of using CGImageSourceCreateThumbnailAtIndex over CGImageSourceCreateImageAtIndex are:
    • You can scale the generated CGImage to any size up to the size of the original
    • Drawing of the image is faster as the image data is uncompressed and decoded

CGImageSourceCreateImageAtIndex appears to do very little other than creating the CGImage wrapper. The time taken to create the CGImage is very short but drawing from the CGImage is very slow. Creating an CGImage using CGImageSourceCreateThumbnailAtIndex is slower but drawing afterwards is faster.

As well as the objcio article by @steipete mentioned above this article by @mattt at @nshipster is also helpful. iOS image resizing techniques