05-28-2011 11:31 PM - edited 05-28-2011 11:31 PM
I'm struggling with a sound timing issue related to garbage collection.
The background: I am writing a timing-sensitive app that plays short sound clips rapidly. These sound clips -- wav or mp3 -- are small and loaded into memory. The usage pattern is to call Player.prefetch, then repeatedly call Player.start(). The problem is that every time I play a sound clip a significant chunk of memory gets allocated and deallocated. Because of this heavy memory allocation, it appears the VM frequently runs its stop-the-world garbage collection cycle, and the app pauses.
On an OS6 phone like the Torch, you can use the applications list in the Options app to monitor memory usage. I can observe the memory of the app climb before the app halts for the GC. The app then halts for a bit, the memory usage drops significantly, and the pattern repeats itself.
This is not a memory leak. It's a memory usage pattern that triggers full GC excessively. Unfortunately, it's happening in an area of code that I have no control over. This must be a frequently encountered issue among games developers. How can I prevent or reduce this pausing problem?
Thanks in advance,
06-02-2011 11:04 PM
I've done some memory analysis using the BlackBerry Objects View tool in the BlackBerry Java Plug-in for Eclipse. What I found is both illuminating and dismaying. First off, my code to play the sound looks like:
Player p = Manager.createPlayer(new ByteArrayInputStreamDataSource(...));
The content is a byte array (only 580 bytes) loaded from a WAV file. After playing this short clip a number of times, I compared the memory snapshot. I found that my app had allocated numerous instances of net.rim.device.internal.media.MediaPlayer, approximately 1 instance for every time the app played the sound clip (p.start()). Each instance of MediaPlayer has an instance of net.rim.device.internal.util.RingBuffer with a byte array of size 59392. This byte array is filled with the content of the 580 byte WAV.
In other words, every time I played that 580 byte WAV file, the phone allocates over 60K worth of media-related storage, then drops it on the floor. So it should be no surprise that playing this short sound clip rapidly would make the garbage collector go bonkers.
Is there a way to play a sound clip repeatedly -- but not as a dumb loop -- without causing the OS to create a heavy new instance of MediaPlayer every time? All I'm doing here is calling Player.start()!
Thanks in advance for any insight.
06-03-2011 05:09 AM - edited 06-03-2011 06:29 AM
OK I see you have done a great analysis.
Did you call player.deallocate() and player.stop() after each file is done playing?
Also calling player.setLoopCount(int n) should loop the same file n times.
If you want to load and player different files I suggest that you subclass InputStream and provide custom implementation and load all your audio as a continuous stream. Thus you won't need to start the player multiple times.
06-03-2011 02:17 PM
I did try stop(). I did not bother with deallocate(), because my problem was too much deallocation, not too little!
That is a good idea about fooling the player by manipulating an InputStream to make the individual clips look like one continuous stream. This will be quite a bit of work to meet my usage pattern, but it might work. Thanks.
06-12-2011 07:57 PM
Unfortunately, I don't think the problem is with the player reuse at all. The behavior I'm seeing pretty much matches what is being described in this thread. It references a bug report (JAVAAPI-1757), but access to that bug report seems to be restricted. In a nutshell, the media player allocates excessive memory due to strings created by debug-level logging. Preventing the player from reaching the end of media won't help, as I discovered: the memory consumption still climbs until the GC kicks in, pausing the app.
In other words, there is probably no way to solve this problem. Without a fix for the underlying OS, playing sound will always cause excessive memory allocation and GC activity. I suppose any app will have to live with occasional pauses if it uses sound in this way.
06-13-2011 03:01 AM
This is really sad....and probably explains some issues I had with my applications working with audio.
I think this player API needs to be fixed because there have been some issues with it. Personally it bothers me that:
1)There is this buffer which needs to be filled before the player actually starts playing which totally cripples and makes it unusable for real time active processes such as VOIP. Not to mention that this buffer has totally inconsistent behavior through different OS versions and devices.
2)There is no listener/event dispatched when the GC is activated so basically you do not know when the little loading screen shows and the device freezes. I don't need to mention how bad this is.
Anyway thanks for the info. Do you have info whether this has been fixed with some of the updates?
06-13-2011 11:30 AM - edited 06-16-2011 12:21 PM
The issue described in the thread you linked only affects recording and not playback, so it is not related to this problem.
When a player is prefetched a RingBuffer is created to read from the data stream into the media playback hardware, it is a 64kB buffer by default for audio but can vary. The MMAPI is geared towards music playback rather than 'real-time' sound playback, and in that context this is actually a small buffer. When I can, I will spend some time investigating how we can improve this experience, so thanks for bringing it to my attention!
The best way to get around this was suggested by dx22, use an InputStream and feed constant data to the player. If your wav files are PCM data (especially if they are all the same sample rate and sample size) this is actually trivial to do... mixing audio in real time rather than just appending it is only slightly trickier but is also relatively simple to do - you can either interleave the samples or add the 2 wavs and normalize the volume. The Dev04 session available in the DevCon 2010 On-Demand Portal (if you have access) will show you code examples of how to do this.
@dx22 There is a new setBufferTime method on the StreamingBufferControl in BlackBerry OS 7.0 specifically designed for VoIP applications which hopefully addresses your concerns in this regard. On previous OS versions you can use a trick to get past this buffering limitation: The general idea is that you can pre-fill the buffer with data that is valid for the codec but doesn’t result in valid sound output. This essentially ‘pads’ the buffer so the minimum size before playback is reached quickly to make playback start, but when the player starts to play it’ll skip over that ‘faked’ data and start playing where it can, thus avoiding any delays.
Specifically if you are using the AMR codec for a VoIPapplication, you can send 3900 NOPs (frame type 15) followed by valid data. If you fill the buffer with those, the player will count them for their duration, but they actually don’t produce output. Then you can append your sound data and get playback to start almost imemdiatelUpdated Not in fact true for AMR unfortunately.
Hope this helps
06-13-2011 04:04 PM
@BVP - this is post of the month for me personally!
Its really great news that there will be solution in OS7 and I am looking forward to see it Currently SreamingBufferControl isn't of much use for me since it only flushes the data.
The trick which you mentioned is really interesting. Currently I'm working on a similar approach. I am using PCM as a format and feed 'silence' to the player but it still produces actual output. I transfer the data over the network in PCMU and decode it to PCM. So is there a similar way to make the player skip some parts of the stream so the delay is minimized for PCM/PCMU audio?
One last question - are these 'type 15 AMR frames' always skipped by the player. I mean can I use them if I go into buffering mode again for a second time?(due to network delay for instance)
Thanks again !
PS @cwong15 sorry for hijacking the thread
06-13-2011 04:35 PM
Glad I can help! No need to wait to see the new API, it's in beta now, which maybe I could have mentioned earlier
I am not aware of how to make the trick work for PCM, as I mentioned the trick requires being able to create valid data for the codec but invalid data when parsed, and I am not aware of any way to do that with PCM. If the data is just silence that's valid so the player will play-out silence as you've seen.
As for the AMR frames question, yes you can use that trick at any point... I believe the player will stop if you give too many consecutive bytes of this data (since it will think there is a problem) so be sure not to do it too long. For some reason 32 KB sticks out in my head as the magic number but I can't seem to find where I put that information at the moment... just feed some actual silence once in a while to avoid that
PS @cwong15 Ditto!
06-13-2011 11:00 PM
Thanks for responding! For the record, the thread I linked to does in fact cover both recording and playback. I quote:
Situation with audio player is even worse - when served with "silence amr" data, audio player seems to "waste" around 20 Kb/sec (or 1.2 Mb/min).
This memory is later reclaimed by GC (which means that whole system is blocked for a couple of seconds)
The worst offenders are two debug messages which were left in production code (see below),
but it seems that there is also some other unneeded object allocation.
Unfortunately, I think this means I cannot solve my problem even if I stream NOOP AMR, since the debug statements that are being executed will continue to generate garbage for the GC to collect.