iOS: H264 encoder crash on disabled stream
The H264 encoder defaults to using the kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
pixel format. If the frames coming into the encoder is RTCCVPixelBuffer frames,
we check the pixel format in the contained CVPixelBuffer and send the same format
to the encoder when possible, by switching the encoder's pixel format. When we
receive frames with buffers conforming to the RTCI420Buffer protocol, we copy
the frame contents to the target pixel buffer, hardcoded to be the default NV12.
This works except when switching incoming frames from RTCCVPixelBuffer frames to
I420 frames during runtime. If we received RTCCVPixelBuffers wrapping e.g. an
RGB CVPixelBuffer, the encoder's pixel format have been changed to RGB. If we
now get incoming frames in I420, we must convert these to RGB instead of NV12
to match the encoder's format.
This bug can be triggered by calling `[_localVideoTrack setIsEnabled:NO]` in
`ARDAppClient.m`. This will make the stream start sending black i420 frames to
the encoder.
This CL fixes this by resetting the compression session with the default NV12
format if the input frame type changes from native to I420.
Bug: webrtc:8638
Change-Id: I5d784d204b7b1d09313a0f4cea6302ea72e9ed94
Reviewed-on: https://webrtc-review.googlesource.com/33260
Reviewed-by: Peter Hanspers <peterhanspers@webrtc.org>
Reviewed-by: Magnus Jedvert <magjed@webrtc.org>
Commit-Queue: Anders Carlsson <andersc@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#21382}diff --git a/sdk/objc/Framework/Classes/VideoToolbox/RTCVideoEncoderH264.mm b/sdk/objc/Framework/Classes/VideoToolbox/RTCVideoEncoderH264.mm
index f84998e..86855a2 100644
--- a/sdk/objc/Framework/Classes/VideoToolbox/RTCVideoEncoderH264.mm
+++ b/sdk/objc/Framework/Classes/VideoToolbox/RTCVideoEncoderH264.mm
@@ -502,11 +502,10 @@
// If we're capturing native frames in another pixel format than the compression session is
// configured with, make sure the compression session is reset using the correct pixel format.
+ // If we're capturing non-native frames and the compression session is configured with a non-NV12
+ // format, reset it to NV12.
OSType framePixelFormat = kNV12PixelFormat;
- if (pixelBufferPool && [frame.buffer isKindOfClass:[RTCCVPixelBuffer class]]) {
- RTCCVPixelBuffer *rtcPixelBuffer = (RTCCVPixelBuffer *)frame.buffer;
- framePixelFormat = CVPixelBufferGetPixelFormatType(rtcPixelBuffer.pixelBuffer);
-
+ if (pixelBufferPool) {
// The pool attribute `kCVPixelBufferPixelFormatTypeKey` can contain either an array of pixel
// formats or a single pixel format.
NSDictionary *poolAttributes =
@@ -520,6 +519,11 @@
compressionSessionPixelFormats = @[ (NSNumber *)pixelFormats ];
}
+ if ([frame.buffer isKindOfClass:[RTCCVPixelBuffer class]]) {
+ RTCCVPixelBuffer *rtcPixelBuffer = (RTCCVPixelBuffer *)frame.buffer;
+ framePixelFormat = CVPixelBufferGetPixelFormatType(rtcPixelBuffer.pixelBuffer);
+ }
+
if (![compressionSessionPixelFormats
containsObject:[NSNumber numberWithLong:framePixelFormat]]) {
resetCompressionSession = YES;