Advanced AV Foundation (CocoaConf, Aug '11)

Advanced AV Foundation
Chris Adamson — @invalidname — http://www.subfurther.com/blog

CocoaConf ’11 — Columbus, OH — August 12, 2011

Tuesday, August 23, 11

The Deal

✤ Slides will be posted to conference site and http://
www.slideshare.com/invalidname

✤ Code will be posted to blog at http://www.subfurther.com/blog

✤ Don’t try to transcribe the code examples


No, really

✤ Seriously, don’t try to transcribe the code examples

✤ You will never keep up

✤ AV Foundation has the longest class and method names you have
ever seen:

✤ AVMutableVideoCompositionLayerInstruction

✤ AVAssetWriterInputPixelBufferAdaptor

✤ etc.


Really, really, seriously… don’t

AVMutableVideoCompositionLayerInstruction *aInstruction =
[AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack: trackA];
[aInstruction setOpacityRampFromStartOpacity:0.0
toEndOpacity:1.0
timeRange:CMTimeRangeMake(CMTimeMakeWithSeconds(2.9, VIDEO_TIME_SCALE),
CMTimeMakeWithSeconds(6.0, VIDEO_TIME_SCALE))];


The Road Map

✤ Capture callbacks

✤ Writing samples with AVAssetWriter

✤ Reading samples with AVAssetReader

✤ Editing with effects

✤ The Future!


Capture Callbacks


Recap: Capture basics

✤ Create an AVCaptureSession to coordinate the capture

✤ Investigate available AVCaptureDevices

✤ Create AVCaptureDeviceInput and connect it to the session

✤ Optional: set up an AVCaptureVideoPreviewLayer

✤ Optional: connect AVCaptureOutputs

✤ Tell the session to start recording


Processing Capture Data… Why?

✤ Audio:

✤ Pitch detection (Glee Karaoke), vocal effects (I Am T-Pain)

✤ Video

✤ Augmented Reality (Word Lens), Face Detection (Face Pong),
Barcode Scanners…


Example: ZXing Project

✤ Open-source (Apache License 2.0) Java library for reading 1-D, 2-D
barcodes

✤ iOS Obj-C++ port of QR Code decoder.

✤ ZXingWidget library

✤ Barcodes and ScanTest sample apps

http://code.google.com/p/zxing/


Demo
Zxing Barcodes


AVCaptureVideoDataOutput

✤ Just another AVCaptureOutput

✤ Delivers frames to an in-app delegate

✤ Delegate gets callback
captureSession:didOutputSampleBuffer:fromConnection:

✤ Sample buffer is a CMSampleBufferRef


AVCaptureVideoDataOutput

AVCaptureDeviceInput *captureInput =
[AVCaptureDeviceInput deviceInputWithDevice:
[AVCaptureDevice
defaultDeviceWithMediaType:AVMediaTypeVideo]
error:nil];
AVCaptureVideoDataOutput *captureOutput =
[[AVCaptureVideoDataOutput alloc] init];
captureOutput.alwaysDiscardsLateVideoFrames = YES;
[captureOutput setSampleBufferDelegate:self
queue:dispatch_get_main_queue()];
// some configuration stuff omitted...
[self.captureSession addOutput:captureOutput];


AVCaptureVideoDataOutputSam
pleBufferDelegate (yes, really)
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
CVImageBufferRef imageBuffer =
CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer,0);
// Core Video pixel counting math omitted...
CGContextRef newContext =
CGBitmapContextCreate(baseAddress, width, height, 8,
bytesPerRow, colorSpace,
kCGBitmapByteOrder32Little |
kCGImageAlphaNoneSkipFirst);
CGImageRef capture = CGBitmapContextCreateImage(newContext);
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
// Zxing then does secondary crop, converts to UIImage, and
// calls its own -[Decoder decodeImage:cropRect:]
}


Writing Samples


AVAssetWriter

✤ Introduced in iOS 4.1

✤ Allows you to create samples programmatically and write them to an
asset

✤ Used for synthesized media ﬁles: screen recording, CGI, synthesized
audio (text to speech, “Hatsune Miku”), etc.


Using AVAssetWriter

✤ Create an AVAssetWriter

✤ Create and conﬁgure an AVAssetWriterInput and connect it to the
writer

✤ -[AVAssetWriter startWriting]

✤ Repeatedly call -[AVAssetWriterInput appendSampleBuffer:] with
CMSampleBufferRef’s

✤ Set expectsDataInRealTime appropriately, honor
readyForMoreMediaData property.


Example: iOS Screen Recorder

✤ Set up an AVAssetWriter to write to a QuickTime movie ﬁle, and an
AVAssetWriterInput with codec and other video track metadata

✤ Set up an AVAssetWriterPixelBufferAdaptor to simplify converting
CGImageRefs into CMSampleBufferRefs

✤ Use an NSTimer to periodically grab the screen image and use the
AVAssetWriterPixelBufferAdapter to write to the AVAssetWriterInput


Demo
CCFScreenRecorder


Create writer, writer input, and
pixel buffer adaptor
assetWriter = [[AVAssetWriter alloc] initWithURL:movieURL
fileType:AVFileTypeQuickTimeMovie
error:&movieError];
NSDictionary *assetWriterInputSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInt:FRAME_WIDTH], AVVideoWidthKey,
[NSNumber numberWithInt:FRAME_HEIGHT], AVVideoHeightKey,
nil];
assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType: AVMediaTypeVideo
outputSettings:assetWriterInputSettings];
assetWriterInput.expectsMediaDataInRealTime = YES;
[assetWriter addInput:assetWriterInput];

assetWriterPixelBufferAdaptor = [[AVAssetWriterInputPixelBufferAdaptor alloc]
! ! ! ! ! ! ! ! initWithAssetWriterInput:assetWriterInput
! ! ! ! ! ! ! ! sourcePixelBufferAttributes:nil];
[assetWriter startWriting];

Settings keys and values are deﬁned in AVAudioSettings.h
and AVVideoSettings.h, or AV Foundation Constants Reference


Getting a screenshot


Create a pixel buffer
// get screenshot image!
CGImageRef image = (CGImageRef) [[self screenshot] CGImage];
NSLog (@"made screenshot");

// prepare the pixel buffer
CVPixelBufferRef pixelBuffer = NULL;
CFDataRef imageData= CGDataProviderCopyData(CGImageGetDataProvider(image));
NSLog (@"copied image data");
cvErr = CVPixelBufferCreateWithBytes(kCFAllocatorDefault,
FRAME_WIDTH,
FRAME_HEIGHT,
kCVPixelFormatType_32BGRA,
(void*)CFDataGetBytePtr(imageData),
CGImageGetBytesPerRow(image),
NULL,
NULL,
NULL,
&pixelBuffer);
NSLog (@"CVPixelBufferCreateWithBytes returned %d", cvErr);


Calculate time and write sample

// calculate the time
CFAbsoluteTime thisFrameWallClockTime = CFAbsoluteTimeGetCurrent();
CFTimeInterval elapsedTime = thisFrameWallClockTime - firstFrameWallClockTime;
NSLog (@"elapsedTime: %f", elapsedTime);
CMTime presentationTime = CMTimeMake (elapsedTime * TIME_SCALE, TIME_SCALE);

// write the sample
BOOL appended = [assetWriterPixelBufferAdaptor appendPixelBuffer:pixelBuffer
withPresentationTime:presentationTime];


Reading Samples


AVAssetReader

✤ Introduced in iOS 4.1

✤ Possible uses:

✤ Audio: Showing an audio wave form in a timeline

✤ Video: Generating frame-accurate thumbnails

✤ Both: Perform effects or DSP in non-realtime


Using AVAssetReader

✤ Create an AVAssetReader

✤ Create and conﬁgure an AVAssetReaderOutput

✤ Three concrete subclasses: AVAssetReaderTrackOutput,
AVAssetReaderAudioMixOutput, and
AVAssetReaderVideoCompositionOutput.

✤ Get data with -[AVAssetReader copyNextSampleBuffer]


Example: Convert iPod song to
PCM

✤ In iOS 4, Media Player framework exposes a new metadata property,
MPMediaItemPropertyAssetURL, that allows AV Foundation to open
the library item as an AVAsset

✤ Obvious use is to let you use your iTunes music as BGM in your
iMovies

✤ Create an AVAssetReader to read sample buffers from the song

✤ Create an AVAssetWriter to convert and write PCM samples


Demo
VTM_AViPodReader


Coordinated reading/writing

✤ You can provide a block to -[AVAssetWriter
requestMediaDataWhenReady:onQueue:]

✤ Only perform your asset reads / writes when the writer is ready.

✤ In this example, AVAssetWriterInput.expectsMediaInRealTime is NO


Set up reader, reader output,
writer
NSURL *assetURL = [song valueForProperty:MPMediaItemPropertyAssetURL];
AVURLAsset *songAsset =
[AVURLAsset URLAssetWithURL:assetURL options:nil];

NSError *assetError = nil;
AVAssetReader *assetReader =
[[AVAssetReader assetReaderWithAsset:songAsset
error:&assetError]
retain];

AVAssetReaderOutput *assetReaderOutput =
[[AVAssetReaderAudioMixOutput
assetReaderAudioMixOutputWithAudioTracks:songAsset.tracks
audioSettings: nil]
retain];
[assetReader addOutput: assetReaderOutput];
AVAssetWriter *assetWriter =
[[AVAssetWriter assetWriterWithURL:exportURL
fileType:AVFileTypeCoreAudioFormat
error:&assetError]
retain];


Set up writer input
AudioChannelLayout channelLayout;
memset(&channelLayout, 0, sizeof(AudioChannelLayout));
channelLayout.mChannelLayoutTag = kAudioChannelLayoutTag_Stereo;
NSDictionary *outputSettings =
[NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInt:kAudioFormatLinearPCM], AVFormatIDKey,
[NSNumber numberWithFloat:44100.0], AVSampleRateKey,
[NSNumber numberWithInt:2], AVNumberOfChannelsKey,
[NSData dataWithBytes:&channelLayout length:sizeof(AudioChannelLayout)],
AVChannelLayoutKey,
[NSNumber numberWithInt:16], AVLinearPCMBitDepthKey,
[NSNumber numberWithBool:NO], AVLinearPCMIsNonInterleaved,
[NSNumber numberWithBool:NO],AVLinearPCMIsFloatKey,
[NSNumber numberWithBool:NO], AVLinearPCMIsBigEndianKey,
nil];
AVAssetWriterInput *assetWriterInput =
[[AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeAudio
outputSettings:outputSettings]
retain];

Note 1: Many of these settings are required, but you won’t know which until you get a runtime error.
Note 2: AudioChannelLayout is from Core Audio

Start reading and writing

[assetWriter startWriting];
[assetReader startReading];
AVAssetTrack *soundTrack = [songAsset.tracks objectAtIndex:0];
CMTime startTime = CMTimeMake (0, soundTrack.naturalTimeScale);
[assetWriter startSessionAtSourceTime: startTime];


Read only when writer is ready

__block UInt64 convertedByteCount = 0;
dispatch_queue_t mediaInputQueue = dispatch_queue_create("mediaInputQueue", NULL);
[assetWriterInput requestMediaDataWhenReadyOnQueue:mediaInputQueue
usingBlock: ^
{
while (assetWriterInput.readyForMoreMediaData) {
CMSampleBufferRef nextBuffer = [assetReaderOutput copyNextSampleBuffer];
if (nextBuffer) {
// append buffer
[assetWriterInput appendSampleBuffer: nextBuffer];
convertedByteCount += CMSampleBufferGetTotalSampleSize (nextBuffer);
// update UI on main thread only
NSNumber *convertedByteCountNumber = [NSNumber
numberWithLong:convertedByteCount];
[self performSelectorOnMainThread:@selector(updateSizeLabel:)
withObject:convertedByteCountNumber
waitUntilDone:NO];
}


Close file when done
else {
// done!
[assetWriterInput markAsFinished];
[assetWriter finishWriting];
[assetReader cancelReading];
NSDictionary *outputFileAttributes = [[NSFileManager defaultManager]
attributesOfItemAtPath:exportPath
error:nil];
NSNumber *doneFileSize = [NSNumber numberWithLong:[outputFileAttributes
fileSize]];
[self performSelectorOnMainThread:@selector(updateCompletedSizeLabel:)
withObject:doneFileSize
waitUntilDone:NO];
// release a lot of stuff
[assetReader release];
[assetReaderOutput release];
[assetWriter release];
[assetWriterInput release];
[exportPath release];
break;
}
}
}];

Media Editing


Video Editing? On iPhone?
Really?

Comparison specs from everymac.com

Really?
1999:
Power Mac G4 500 AGP


Really?
1999:
Power Mac G4 500 AGP

CPU: 500 MHz G4
RAM: 256 MB
Storage: 20 GB HDD

Really?
1999: 2010:
Power Mac G4 500 AGP iPhone 4

CPU: 500 MHz G4
RAM: 256 MB
Storage: 20 GB HDD

Really?
1999: 2010:
Power Mac G4 500 AGP iPhone 4

CPU: 500 MHz G4 CPU: 800 MHz Apple A4
RAM: 256 MB RAM: 512 MB
Storage: 20 GB HDD Storage: 16 GB Flash

AVComposition

✤ An AVAsset that gets its tracks from multiple ﬁle-based sources

✤ To create a movie, you typically use an AVMutableComposition

composition = [[AVMutableComposition alloc] init];


Copying from another asset

✤ -[AVMutableComposition insertTimeRange:ofAsset:atTime:error:]

CMTime inTime = CMTimeMakeWithSeconds(inSeconds, 600);
CMTime outTime = CMTimeMakeWithSeconds(outSeconds, 600);
CMTime duration = CMTimeSubtract(outTime, inTime);
CMTimeRange editRange = CMTimeRangeMake(inTime, duration);
NSError *editError = nil;

[targetController.composition insertTimeRange:editRange
! ! ! ! ! ! ! ofAsset:sourceAsset
atTime:targetController.composition.duration
! ! ! ! ! ! ! error:&editError];


Demo
VTM_AVEditor


Editing With Effects


Multiple video tracks

✤ To combine multiple video sources into one movie, create an
AVMutableComposition, then create AVMutableCompositionTracks

// create composition
self.composition = [[AVMutableComposition alloc] init];

// create video tracks a and b
// note: mediatypes are defined in AVMediaFormat.h
[trackA! release];
trackA = [self.composition addMutableTrackWithMediaType:AVMediaTypeVideo
preferredTrackID:kCMPersistentTrackID_Invalid];
[trackB release];
trackB = [self.composition addMutableTrackWithMediaType:AVMediaTypeVideo

// locate source video track
AVAssetTrack *sourceVideoTrack = [[sourceVideoAsset tracksWithMediaType:
AVMediaTypeVideo]
objectAtIndex: 0];


A/B Roll Editing

✤ Apple recommends alternating between two tracks, rather than using
arbitrarily many (e.g., one track per shot)


Sound tracks

✤ Treat your audio as separate tracks too.

// create music track
trackMusic = [self.composition addMutableTrackWithMediaType:AVMediaTypeAudio

CMTimeRange musicTrackTimeRange = CMTimeRangeMake(kCMTimeZero,
musicTrackAudioAsset.duration);
NSError *trackMusicError = nil;
[trackMusic insertTimeRange:musicTrackTimeRange
ofTrack:[musicTrackAudioAsset.tracks objectAtIndex:0]
atTime:kCMTimeZero
error:&trackMusicError];


Empty ranges

✤ Use -[AVMutableCompositionTrack insertEmptyTimeRange:] to
account for any part of any track where you won’t be inserting media
segments.

CMTime videoTracksTime = CMTimeMake(0, VIDEO_TIME_SCALE);
CMTime postEditTime = CMTimeAdd (videoTracksTime,
CMTimeMakeWithSeconds(FIRST_CUT_TRACK_A_IN_TIME,
VIDEO_TIME_SCALE));
[trackA insertEmptyTimeRange:CMTimeRangeMake(kCMTimeZero, postEditTime)];
videoTracksTime = postEditTime;


Track-level inserts

✤ Insert media segments with -[AVMutableCompositionTrack
insertTimeRange:ofTrack:atTime:error]

postEditTime = CMTimeAdd (videoTracksTime, CMTimeMakeWithSeconds(FIRST_CUT_DURATION,
VIDEO_TIME_SCALE));
CMTimeRange firstShotRange = CMTimeRangeMake(kCMTimeZero,
CMTimeMakeWithSeconds(FIRST_CUT_DURATION,
VIDEO_TIME_SCALE));
[trackA insertTimeRange:firstShotRange
ofTrack:sourceVideoTrack
atTime:videoTracksTime
error:&performError];
videoTracksTime = postEditTime;


AVVideoComposition

✤ Describes how multiple video tracks are to be composited together.
Mutable version is AVMutableVideoComposition

✤ Not a subclass of AVComposition!

✤ Contains an array AVVideoCompositionInstructions

✤ Time ranges of these instructions need to not overlap, have gaps, or
fail to match the duration of the AVComposition


AVVideoCompositionInstruction

✤ Represents video compositor instructions for all tracks in one time
range

✤ These instructions are a layerInstructions property

✤ Of course, you’ll be creating an
AVMutableVideoCompositionInstruction


AVVideoCompositionLayerInstru
ction (yes, really)
✤ Identiﬁes the instructions for one track within an
AVVideoCompositionInstruction.

✤ AVMutableVideoCompositionLayerInstruction. I warned you about
this back on slide 3.

✤ Currently supports two properties: opacity and afﬁne transform.
Animating (“ramping”) these creates fades/cross-dissolves and
pushes.

✤ e.g., -[AVMutableVideoCompositionLayerInstruction
setOpacityRampFromStartOpacity:toEndOpacity:timeRange]


An
AVVideoCompositionInstruction

AVMutableVideoCompositionInstruction *transitionInstruction =
[AVMutableVideoCompositionInstruction videoCompositionInstruction];
transitionInstruction.timeRange = CMTimeRangeMake(kCMTimeZero, composition.duration);
AVMutableVideoCompositionLayerInstruction *aInstruction =
[AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:
trackA];
[aInstruction setOpacityRampFromStartOpacity:0.0 toEndOpacity:1.0
timeRange:CMTimeRangeMake(CMTimeMakeWithSeconds(2.9, VIDEO_TIME_SCALE),
CMTimeMakeWithSeconds(6.0, VIDEO_TIME_SCALE))];
AVMutableVideoCompositionLayerInstruction *bInstruction =
[AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:
trackB];
[bInstruction setOpacity:0 atTime:kCMTimeZero];
transitionInstruction.layerInstructions = [NSArray arrayWithObjects:aInstruction, bInstruction, nil];
[videoInstructions addObject: transitionInstruction];


Attaching the instructions

AVMutableVideoComposition *videoComposition =
[AVMutableVideoComposition videoComposition];
videoComposition.instructions = videoInstructions;
videoComposition.renderSize = videoSize;
videoComposition.frameDuration = CMTimeMake(1, 30); // 30 fps
compositionPlayer.currentItem.videoComposition =
videoComposition;


Titles and Effects

✤ AVSynchronizedLayer gives you a CALayer that gets its timing from
an AVPlayerItem, rather than a wall clock

✤ Run the movie slowly or backwards, the animation runs slowly or
backwards

✤ Can add other CALayers as sublayers and they’ll all get their timing
from the AVPlayerItem


Creating a main title layer

// synchronized layer to own all the title layers
AVSynchronizedLayer *synchronizedLayer =
[AVSynchronizedLayer
synchronizedLayerWithPlayerItem:compositionPlayer.currentItem];
synchronizedLayer.frame = [compositionView frame];
[self.view.layer addSublayer:synchronizedLayer];

// main titles
CATextLayer *mainTitleLayer = [CATextLayer layer];
mainTitleLayer.string = NSLocalizedString(@"Running Start", nil);
mainTitleLayer.font = @"Verdana-Bold";
mainTitleLayer.fontSize = videoSize.height / 8;
mainTitleLayer.foregroundColor = [[UIColor yellowColor] CGColor];
mainTitleLayer.alignmentMode = kCAAlignmentCenter;
mainTitleLayer.frame = CGRectMake(0.0, 0.0, videoSize.width,
videoSize.height);
mainTitleLayer.opacity = 0.0; // initially invisible
[synchronizedLayer addSublayer:mainTitleLayer];


Adding an animation

// main title opacity animation
[CATransaction begin];
[CATransaction setDisableActions:YES];
CABasicAnimation *mainTitleInAnimation =
[CABasicAnimation animationWithKeyPath:@"opacity"];
mainTitleInAnimation.fromValue = [NSNumber numberWithFloat: 0.0];
mainTitleInAnimation.toValue = [NSNumber numberWithFloat: 1.0];
mainTitleInAnimation.removedOnCompletion = NO;
mainTitleInAnimation.beginTime = AVCoreAnimationBeginTimeAtZero;
mainTitleInAnimation.duration = 5.0;
[mainTitleLayer addAnimation:mainTitleInAnimation forKey:@"in-animation"];

Nasty gotcha: AVCoreAnimationBeginTimeAtZero is a special value that is used for AVF
animations, since 0 would otherwise be interpreted as CACurrentMediaTime()


Multi-track audio

✤ AVPlayerItem.audioMix property

✤ AVAudioMix class describes how multiple audio tracks are to be
mixed together

✤ Analogous to videoComposition property (AVVideoComposition)


Basic Export

✤ Create an AVAssetExportSession

✤ Must set outputURL and outputFileType properties

✤ Inspect possible types with supportedFileTypes property (list of
AVFileType… strings in docs)

✤ Begin export with exportAsynchronouslyWithCompletionHandler:

✤ This takes a block, which will be called on completion, failure,
cancellation, etc.


Advanced Export

✤ AVAssetExportSession takes videoComposition and audioMix
parameters, just like AVPlayerItem

✤ To include AVSynchronizedLayer-based animations in an export, use
a AVVideoCompositionCoreAnimationTool and set it as the
animationTool property of the AVMutableVideoComposition (but
only for export)


Hazards and Hassles


Only effects are dissolve and
push?

How would we do this checkerboard wipe in AV Foundation?
It’s pretty easy in QuickTime!

How do you…

✤ Save a composition to work on later?

✤ Even if AVMutableComposition supports NSCopying, what if
you’ve got titles in an AVSynchronizedLayer?

✤ Support undo / redo of edits?

✤ Add import/export support for other formats and codecs?


AV Foundation Sucks!

✤ Too hard to understand!

✤ Too many classes and methods!

✤ Verbose and obtuse method naming

✤ AVComposition and AVVideoComposition are completely
unrelated? WTF, Apple?


Complex things usually aren’t easy

Simple Complex

Hard

Easy


AV Foundation Rocks!

✤ Addresses a huge range of media functionality

✤ The other guys don’t even try

✤ Same framework used by Apple for iMovie for iPhone/iPad and Final
Cut Pro X for Mac OS X.

✤ You can create functionality equivalent to iMovie / Final Cut in a few
hundred lines of code

✤ Added to Mac OS X in 10.7 (Lion)


Q&A
Chris Adamson — @invalidname — http://www.subfurther.com/blog

CocoaConf ’11 — Columbus, OH — August 12, 2011


Also available!

✤ “Core Audio is serious black
arts shit.” — Mike Lee (@bmf)

✤ It’s tangentially related to AV
Foundation, so you should
totally buy it when it comes
out.


Advanced AV Foundation (CocoaConf, Aug '11)

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Advanced AV Foundation (CocoaConf, Aug '11)

Ähnlich wie Advanced AV Foundation (CocoaConf, Aug '11) (20)

Mehr von Chris Adamson

Mehr von Chris Adamson (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Advanced AV Foundation (CocoaConf, Aug '11)