Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Native Development

Reply
BlackBerry Development Advisor
Posts: 668
Registered: ‎11-29-2011
My Device: developer

Re: Camera API NV12 frame to AVFrame (FFmpeg)

Got it, thanks.  NV12 buffer looks ok, though you're missing the final 3k bytes for some reason.. not importatnt.  4096 * 1920 * 1.5 = 11796480.  Can you confirm that uv_offset = 7864320, stride = 4096, height = 1920 ?

Your u.bin and v.bin files look good @ 540x960, but your y.bin file is all zeros - I'm assuming because you commented out the code to set frame->data[0] = buf->framebuf ?

 

Cheers,

Sean

 

Developer
Posts: 1,041
Registered: ‎07-16-2008
My Device: ಠ_ಠ

Re: Camera API NV12 frame to AVFrame (FFmpeg)

Yea I forgot to uncomment the y plane. Here is the fixed version though.

 

uv_offset: 7864320

stride: 4096

width: 1080

height: 1920

 

 

printf("uv_offset: %d\n", (int)buf->framedesc.nv12.uv_offset);
printf("stride: %d\n", buf->framedesc.nv12.stride);
printf("width: %d\n", pVideoCodec->width);
printf("height: %d\n", pVideoCodec->height);

 

BlackBerry Development Advisor
Posts: 668
Registered: ‎11-29-2011
My Device: developer

Re: Camera API NV12 frame to AVFrame (FFmpeg)

Ok.  All 4 files look good, so there's no problem in your conversion.

The problem is likely a misunderstanding in how ffmpeg is parameterizing the frame.  Are all of your AVFrame fields initialized to appropriate values?  You might want to try fishing around the ffmpeg forums and describe your image as:

 

Y plane: 1080 (strided to 4096) x 1920, 8 bit

U plane: 540 x 960, 8 bit

V plane: 540 x 960, 8 bit

 

And see if anyone can spot a problem with your AVFrame fields (which you should dump).

 

Cheers,

Sean

Developer
Posts: 1,041
Registered: ‎07-16-2008
My Device: ಠ_ಠ

Re: Camera API NV12 frame to AVFrame (FFmpeg)

Developer
Posts: 1,041
Registered: ‎07-16-2008
My Device: ಠ_ಠ

Re: Camera API NV12 frame to AVFrame (FFmpeg)

Developer
Posts: 1,041
Registered: ‎07-16-2008
My Device: ಠ_ಠ

Re: Camera API NV12 frame to AVFrame (FFmpeg)

[ Edited ]

Apparently it is a QuickTime issue. It plays in VLC. Doesn't seem to like the data in the Y plane. It displays with the correct size if I memset the buffer to 0, but VLC is fine with it either way.

 

EDIT: Yea, its an encoding issue. I switched to MPEG2 codec instead of MPEG1 and it plays fine* in QuickTime now.

*fine besides the frame rate is too high.

Developer
Posts: 1,041
Registered: ‎07-16-2008
My Device: ಠ_ಠ

Re: Camera API NV12 frame to AVFrame (FFmpeg)

[ Edited ]

Attached is the output video. Each "write frame" was printing at about 1/second while I was recording, but playback is much faster. Even though writing to the file stops when I press the "Stop Record" button, the video still shows the last frame writes that were displayed right before I told it to stop. I thought this callback was async... but since it is writing a frame a second and still shows all the frames between when I hit start and stop... are frames getting dropped and not reported to the callback? (Writing is slower then the camera is recording, so I would expect to fall behind and not write out the last frames displayed right before I hit stop.)

 

Console ouput: it gets tons of frames when not encoding, then drops significantly. The "got video buffer" is outside the mutex, so those should still be coming in.

got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
onStartStopRecording
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
### BAD_FRAME pid=13443105 at 462296 ms -> t: 33 ms p1: 3 ms, r: 2 ms, p2: 0 ms, u: 28 ms
### BAD_FRAME pid=13443105 at 462361 ms -> t: 65 ms p1: 0 ms, r: 65 ms, p2: 0 ms, u: 0 ms
### BAD_FRAME pid=13443105 at 462446 ms -> t: 37 ms p1: 0 ms, r: 8 ms, p2: 0 ms, u: 29 ms
### BAD_FRAME pid=13443105 at 462476 ms -> t: 30 ms p1: 0 ms, r: 1 ms, p2: 0 ms, u: 29 ms
### BAD_FRAME pid=13443105 at 462510 ms -> t: 34 ms p1: 0 ms, r: 2 ms, p2: 0 ms, u: 32 ms
skipped write
got video buffer of size 1080 x 1920, bytes: 11796480
### BAD_FRAME pid=13443105 at 462542 ms -> t: 32 ms p1: 0 ms, r: 2 ms, p2: 0 ms, u: 30 ms
### BAD_FRAME pid=13443105 at 462576 ms -> t: 33 ms p1: 0 ms, r: 1 ms, p2: 0 ms, u: 32 ms
### BAD_FRAME pid=13443105 at 462612 ms -> t: 36 ms p1: 0 ms, r: 1 ms, p2: 0 ms, u: 35 ms
### BAD_FRAME pid=13443105 at 462643 ms -> t: 31 ms p1: 0 ms, r: 0 ms, p2: 0 ms, u: 31 ms
write frame 0 (size=103128)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 1 (size=265998)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 2 (size=353544)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 3 (size=271032)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 4 (size=309829)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 5 (size=322940)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 6 (size=319980)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 7 (size=300666)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 8 (size=268354)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 9 (size=337469)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 10 (size=286112)
got video buffer of size 1080 x 1920, bytes: 11796480
write frame 11 (size=271387)
onStartStopRecording 
got video buffer of size 1080 x 1920, bytes: 11796480
### BAD_FRAME pid=13443105 at 468002 ms -> t: 22 ms p1: 0 ms, r: 15 ms, p2: 0 ms, u: 7 ms
### BAD_FRAME pid=13443105 at 468037 ms -> t: 19 ms p1: 0 ms, r: 6 ms, p2: 0 ms, u: 13 ms
### BAD_FRAME pid=13443105 at 468073 ms -> t: 36 ms p1: 0 ms, r: 3 ms, p2: 0 ms, u: 33 ms
### BAD_FRAME pid=13443105 at 468104 ms -> t: 31 ms p1: 0 ms, r: 25 ms, p2: 0 ms, u: 6 ms
### BAD_FRAME pid=13443105 at 468138 ms -> t: 34 ms p1: 0 ms, r: 3 ms, p2: 0 ms, u: 31 ms
### BAD_FRAME pid=13443105 at 468171 ms -> t: 33 ms p1: 0 ms, r: 3 ms, p2: 0 ms, u: 30 ms
### BAD_FRAME pid=13443105 at 468204 ms -> t: 33 ms p1: 0 ms, r: 1 ms, p2: 0 ms, u: 32 ms
### BAD_FRAME pid=13443105 at 468238 ms -> t: 34 ms p1: 1 ms, r: 0 ms, p2: 0 ms, u: 33 ms
### BAD_FRAME pid=13443105 at 468271 ms -> t: 33 ms p1: 0 ms, r: 1 ms, p2: 0 ms, u: 32 ms
### BAD_FRAME pid=13443105 at 468306 ms -> t: 35 ms p1: 0 ms, r: 1 ms, p2: 0 ms, u: 34 ms
### BAD_FRAME pid=13443105 at 468338 ms -> t: 32 ms p1: 0 ms, r: 1 ms, p2: 0 ms, u: 31 ms
write frame 12 (size=340790)
write (delayed) frame 13 (size=282541)
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480
got video buffer of size 1080 x 1920, bytes: 11796480

 

BlackBerry Development Advisor
Posts: 668
Registered: ‎11-29-2011
My Device: developer

Re: Camera API NV12 frame to AVFrame (FFmpeg)

Cool.  Glad to see some progress.

Curious about quicktime not being able to recognize the file, but I can't speak for quicktime's supported codecs and formats.  I know windows media can have trouble with videos taller than 1080 sometimes - wonder if it's similar.

 

Frames will be dropped, correct.  If you cannot process a video frame callback within 33ms (@ 30fps), then you will be dropping frames.  You might consider a much lower resolution for use with a software encoder - try 288x512 as a start maybe.

 

Cheers,

Sean

 

Developer
Posts: 1,041
Registered: ‎07-16-2008
My Device: ಠ_ಠ

Re: Camera API NV12 frame to AVFrame (FFmpeg)

Okay, good to know. Right now the only bottleneck is making a copy of each camera_buffer from the callback to put into a queue. Everything else was moved to other threads (so the encoding/writing part is fast enough for now). Yesterday I tried changing the dimensions using IMGPROP_WIDTH/HEIGHT but the framedesc kept reporting 1080/1920 for the nv12 frame. I'll try again in a bit.

BlackBerry Development Advisor
Posts: 668
Registered: ‎11-29-2011
My Device: developer

Re: Camera API NV12 frame to AVFrame (FFmpeg)

Make sure to change width/height using both:

camera_set_videovf_property() and camera_set_video_property().

This is required because the CAMERA_FEATURE_PREVIEWISVIDEO feature is asserted on this platform.

 

You mentioned you are doing a full buffer copy per frame - are you copying the entire buffer and then doing the de-interleave copy of U and V, or are you doing the U/V deinterleave and a straight Y copy?  I would avoid copying U and V more than once if you can.  Also, you may as well de-stride the Y plane while you're at it and construct an AVFrame with linesize[0] = width.

 

Cheers,

Sean