Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Native Development

Reply
Developer
Posts: 1,041
Registered: ‎07-16-2008
My Device: ಠ_ಠ

Re: Options for displaying video

Yea, I would go ahread and move all the ffmpeg stuff into a new thread using pthread.

Trusted Contributor
Posts: 160
Registered: ‎09-13-2012
My Device: 9900
My Carrier: vodafone

Re: Options for displaying video

I am now pretty much certain no blocking is going on with the ui thread. A new thread is created and the 2 threads seem to be doing exactly what they should be doing - except the frames are not displayed!

 

I will post my 2 new functions to improve on the last code but it's pretty much the same issue with 1 or 2 threads.

 

Regards.

 

void* decoding_thread(void* arg)
{
	           printf("start thread decode \n");
	           fflush( stdout );
		       av_register_all();
		       avformat_network_init();
	           AVFormatContext* pFormatCtx = avformat_alloc_context();
	           avformat_open_input(&pFormatCtx, filename, NULL, NULL);

	             screen_create_window_type(&screen_window, screen_cxt, SCREEN_CHILD_WINDOW);
	            // screen_join_window_group(screen_window, groupArr.constData());
	             //screen_set_window_property_cv(screen_window, SCREEN_PROPERTY_ID_STRING, idArr.length(), idArr.constData());

	             int usage = SCREEN_USAGE_NATIVE;
	             screen_set_window_property_iv(screen_window, SCREEN_PROPERTY_USAGE, &usage);

	             int video_size[] = { 900,500 };
	             screen_set_window_property_iv(screen_window, SCREEN_PROPERTY_BUFFER_SIZE, video_size);
	             screen_set_window_property_iv(screen_window, SCREEN_PROPERTY_SOURCE_SIZE, video_size);

	             int z = -1;
	             screen_set_window_property_iv(screen_window, SCREEN_PROPERTY_ZORDER, &z);

	             int pos[] = { 0, 0 };
	             screen_set_window_property_iv(screen_window, SCREEN_PROPERTY_POSITION, pos);

	             screen_create_window_buffers(screen_window, 1);

	             screen_pixmap_t screen_pix;
	             screen_create_pixmap(&screen_pix, screen_cxt);

	             usage = SCREEN_USAGE_WRITE | SCREEN_USAGE_NATIVE;
	             screen_set_pixmap_property_iv(screen_pix, SCREEN_PROPERTY_USAGE, &usage);

	             int format = SCREEN_FORMAT_YUV420;
	             screen_set_pixmap_property_iv(screen_pix, SCREEN_PROPERTY_FORMAT, &format);

	             screen_set_pixmap_property_iv(screen_pix, SCREEN_PROPERTY_BUFFER_SIZE, video_size);

	             screen_create_pixmap_buffer(screen_pix);

	             screen_buffer_t screen_pixel_buffer;
	             screen_get_pixmap_property_pv(screen_pix, SCREEN_PROPERTY_RENDER_BUFFERS, (void**) &screen_pixel_buffer);

	             int stride;
	             screen_get_buffer_property_iv(screen_pixel_buffer, SCREEN_PROPERTY_STRIDE, &stride);
	             AVFrame *frame = avcodec_alloc_frame();
	             AVFrame *pFrameRGB = avcodec_alloc_frame();
	             AVPacket packet;
	             AVCodec *codec;
	             uint8_t *buffer;
	             int numBytes = avpicture_get_size(PIX_FMT_YUV420P, 900, 500);
	             buffer = (uint8_t *) av_malloc(numBytes*sizeof(uint8_t));


	               //assign appropriate parts of buffer to image panes in pFrameRGB
	             avpicture_fill((AVPicture*) pFrameRGB, buffer, PIX_FMT_YUV420P, 900,500);

	             av_init_packet(&packet);
	             int got_frame = 0;

	             codec = avcodec_find_decoder(CODEC_ID_FLV1);
	             if (!codec) {
	                 fprintf(stderr, "codec not found\n");
	                 exit(1);
	                }
	             AVCodecContext *c = avcodec_alloc_context3(codec);


	             if(codec->capabilities&CODEC_CAP_TRUNCATED)
	                 c->flags|= CODEC_FLAG_TRUNCATED; /* we do not send complete frames */


	             if (avcodec_open(c, codec) < 0) {
	               fprintf(stderr, "could not open codec\n");
	                exit(1);
	                }

	             while(av_read_frame(pFormatCtx, &packet) >= 0 && !exit_application)
	               {
	            	 printf("before decode \n");
	            	 fflush( stdout );
	            	 int got_frame = 0;
	                 int decode_result = avcodec_decode_video2(c, frame, &got_frame, &packet);
	                 printf("after decode \n");
	                 fflush( stdout );
	                 if (decode_result < 0)
	                 {
	                  fprintf(stderr, "Error while decoding video\n");


	                 }

	                        if(got_frame)
	                        {
	                        	printf("got frame \n");
	                        	fflush( stdout );
	                            uint8_t *srcy = frame->data[0];
	                            uint8_t *srcu = frame->data[1];
	                            uint8_t *srcv = frame->data[2];
	                            unsigned char *ptr = NULL;
	                               screen_get_buffer_property_pv(screen_pixel_buffer, SCREEN_PROPERTY_POINTER, (void**) &ptr);
	                            unsigned char *y = ptr;
	                            unsigned char *u = y + (500 * stride);
	                            unsigned char *v = u + (500 * stride) / 4;
	                            int i = 0;
	                            for ( i = 0; i < 500; i++)
	                            {
	                                int doff = i * stride;
	                                int soff = i * frame->linesize[0];
	                                memcpy(&y[doff], &srcy[soff], 900);
	                            }

	                            for ( i = 0; i < 500 / 2; i++)
	                            {
	                                int doff = i * stride / 2;
	                                int soff = i * frame->linesize[1];
	                                memcpy(&u[doff], &srcu[soff], 900 / 2);
	                            }

	                            for ( i = 0; i < 500 / 2; i++)
	                            {
	                                int doff = i * stride / 2;
	                                int soff = i * frame->linesize[2];
	                                memcpy(&v[doff], &srcv[soff], 900 / 2);
	                            }
	                            printf("before posttoscreen \n");
	                            fflush( stdout );
	                            screen_buffer_t screen_buffer;
	                            screen_get_window_property_pv(screen_window, SCREEN_PROPERTY_RENDER_BUFFERS, (void**) &screen_buffer);

	                            int attribs[] = { SCREEN_BLIT_SOURCE_WIDTH, 900, SCREEN_BLIT_SOURCE_HEIGHT, 500, SCREEN_BLIT_END };
	                            screen_blit(screen_cxt, screen_buffer, screen_pixel_buffer, attribs);

	                            int dirty_rects[] = { 0, 0, 900, 500 };
	                            screen_post_window(screen_window, screen_buffer, 1, dirty_rects, 0);
	                            printf("after posttoscreen  \n");
	                            fflush( stdout );
	                            av_free_packet(&packet);
	                            av_init_packet(&packet);
	                            }

	                           printf("after freepacket  \n");
	                           fflush( stdout );
	                           //av_init_packet(&packet);
	               }
}
int main(int argc, char *argv[]) {
    int rc;
    screen_create_context(&screen_cxt, 0);
    printf("after av registerall\n");
    //Initialize BPS library
    bps_initialize();

    //Use utility code to initialize EGL for rendering with GL ES 1.1
    if (EXIT_SUCCESS != bbutil_init_egl(screen_cxt)) {
        fprintf(stderr, "bbutil_init_egl failed\n");
        bbutil_terminate();
        screen_destroy_context(screen_cxt);
        return 0;
    }
  if (EXIT_SUCCESS != initialize()) {
        fprintf(stderr, "initialize failed\n");
        bbutil_terminate();
        screen_destroy_context(screen_cxt);
        return 0;
    }

    //Signal BPS library that navigator and screen events will be requested
    if (BPS_SUCCESS != screen_request_events(screen_cxt)) {
        fprintf(stderr, "screen_request_events failed\n");
        bbutil_terminate();
        screen_destroy_context(screen_cxt);
        return 0;
    }

    if (BPS_SUCCESS != navigator_request_events(0)) {
        fprintf(stderr, "navigator_request_events failed\n");
        bbutil_terminate();
        screen_destroy_context(screen_cxt);
        return 0;
    }

    //Signal BPS library that navigator orientation is not to be locked
    if (BPS_SUCCESS != navigator_rotation_lock(false)) {
        fprintf(stderr, "navigator_rotation_lock failed\n");
        bbutil_terminate();
        screen_destroy_context(screen_cxt);
        return 0;
    }
    render();
    if(pthread == NULL)
           pthread_create(&pthread, 0, &decoding_thread, NULL);

    while (!exit_application) {
        //Request and process all available BPS events
        bps_event_t *event = NULL;

        for(;;) {
            rc = bps_get_event(&event, 0);
            assert(rc == BPS_SUCCESS);

            if (event) {
                int domain = bps_event_get_domain(event);

                if (domain == screen_get_domain()) {
                    handleScreenEvent(event);
                } else if ((domain == navigator_get_domain())
                        && (NAVIGATOR_EXIT == bps_event_get_code(event))) {
                    exit_application = 1;
                }
            } else {
                break;
            }
        }


             //avformat_free_context(pFormatCtx);//Initialize application logic

    }

    //Stop requesting events from libscreen
    screen_stop_events(screen_cxt);

    //Shut down BPS library for this process
    bps_shutdown();

    //Use utility code to terminate EGL setup
    bbutil_terminate();

    //Destroy libscreen context
    screen_destroy_context(screen_cxt);
    return 0;
}

 

BlackBerry Development Advisor
Posts: 683
Registered: ‎11-29-2011
My Device: PRIV
My Carrier: Rogers

Re: Options for displaying video

Your ZORDER is -1.  That puts the window behind the application window.  Unless you are doing some complex transparency or hole-punching on your application window, you will never see this window.

Move it in front by setting a positive ZORDER.

 

I haven't looked at the rest in detail, but I'll throw some other comments out there while skimming it:

-you are not selecting a format for your window (ARGB8888, NV12, etc.).  it is probably defaulting to ARGB8888.

-you are creating a YUV420 pixmap... are you sure this is even supported?  I know for a fact the hardware platform doesn't natively support fully planar yuv, but maybe there is a software conversion that takes place.

-you are memcpy()ing to a pimap and then blitting to the screen - while this is okay, it will be slow.  there are 3 copies happening here (ffmpeg->buffer, buffer->pixmap, pixmap->screen).  You should in theory be able to do (ffmpeg->screen), but that is an optimization for later

 

Anyway, you first need to get your window to show up.  I would suggest looking at the screen C template code.  it just starts up, creates a window, fills it with a colour and then posts it to the screen.  You should take the colour-filling bit and stick it in your code before you hit the while() loop, just to make sure that your window is even visible.

 

Cheers,

Sean

 

 

 

 

Trusted Contributor
Posts: 160
Registered: ‎09-13-2012
My Device: 9900
My Carrier: vodafone

Re: Options for displaying video

Thanks Sean, you were right, I removed all reference to egl stuff and just created a normal screen with group associated with it and there it is - my video stream almost real time - less than a second delay! (still with the tcp lag/stall now and again but it catches up).

 

So now I need to go further...

 

Your suggestion to make the code even quicker is obviously important to me - don't want to add work to process but I don't fully understand that part of the code (i just grabbed it from libffbb and made it work (finally!)). Would you mind elaborating a little more so i can do some searching or testing?

 

I earlier had to remove audio from this video stream because I was getting bad header errors and something to do with image start when decoding. So i assumed (looking at examples out there) that it was because i wasn't getting references to audio and video codec streams earlier in code and checking if(packet.index == videostream) but when i tried to use av_find_stream_info(&pFormatCtx) it never returned - any suggestions on this or how to get audio going?

 

Now, I'm planning on going back to my udp socket client/server I got working last week to overcome the stall, I seen this morning that ffmpeg can just receive and play media using udp://0.0.0.0Smiley TongueORT - would it be possible to change/hack my code a little to receive udp data?

 

I'm probably pushing my luck by now with asking for help so anything anyone can offer is greatly appreciated. I can't believe how far I have come with the help of this community. Thank you.

BlackBerry Development Advisor
Posts: 683
Registered: ‎11-29-2011
My Device: PRIV
My Carrier: Rogers

Re: Options for displaying video

I can't likely help you with the performance stuff until after next week.. I am bogged down with work for BBJam at the moment.  I think I left enough info in my previous post to get you started, or at least to enable someone else to help you.

 

Cheers,

Sean

Trusted Contributor
Posts: 160
Registered: ‎09-13-2012
My Device: 9900
My Carrier: vodafone

Re: Options for displaying video

I completely understand Sean. Thanks again.

 

If anybody else can help me in what really is the final hurdle for me playing video natively i would greatly appreciate it. 

 

I now have the audio packet data just as i have the video packet data, I just need to find a way to output the audio data to the speakers of the device?

 

There is absolutly nothing to show me this process on the web so without a little shove in the right direction im stumped.

 

Thanks in advance.

BlackBerry Development Advisor
Posts: 683
Registered: ‎11-29-2011
My Device: PRIV
My Carrier: Rogers

Re: Options for displaying video

if it's PCM audio, look at the PlayWav sample.

If it's compressed audio, then you will need to decompress first.