Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Native Development

Reply
Developer
Posts: 76
Registered: ‎02-08-2013
My Device: z10
My Carrier: rogers
Accepted Solution

Persistent Phantom Files - Storage Leak.

Hi folks,

 

I've been working through a somewhat worrying issue lately: if you QFile.remove() a file that MediaPlayer is playing, or is MediaPlayer.pause() or even recently MediaPlayer.stop()'d the file gets removed but the disk storage taken up by the file doesn't become available - ever.

 

The first time I came across this I was mystified: the SD card total files storage used plus the free storage didn't add up to the card's capacity and there was no way to recover the lost space other than formatting the disk.

 

I  also confirmed that this isn't a filesystem thing but can also happen on the QNX6 filesystem on the Phone's storage. What happens when the phone storage is filled with phantom files? You need to do a security wipe!

 

There seems to be two issues here:

1. There's an easy way to delete the place holder for the file in the filesystem that doesn't relinquish the space used. (which can lead to the need to reformat your phone/media card) I guess this can be considered a storage leak.

2. There's no obvious way delete a file that was recently buffered in the MediaPlayer.

 

Here's the command line output I captured when the file _wasn't_ playing:

$ ls -l
total 105107
-rw-rw-rw-   1 devuser   1000_shared  53814712 Feb 08 19:23 sn0441.mp3
$ df -kP .
Filesystem           1024-blocks      Used Available Capacity  Mounted on      
/dev/emmc/user0         13826032   9625376   4200656      70%  /               
$ df -kP .
Filesystem           1024-blocks      Used Available Capacity  Mounted on      
/dev/emmc/user0         13826032   9572792   4253240      70%  /           

 

You can see that 9625376  - 9572792  = 52584k, roughtly the size of the file.

 

If I do the same thing with a file that's playing:

$ df -kP .
Filesystem           1024-blocks      Used Available Capacity  Mounted on      
/dev/emmc/user0         13826032   9625408   4200624      70%  /     

$ ls -l
total 222732
-rw-rw-rw-   1 devuser   1000_shared  60223698 Feb 08 19:17 sn0438.mp3
-rw-rw-rw-   1 devuser   1000_shared  53814712 Feb 08 19:23 sn0441.mp3       

>>>>>> File gets deleted from within the App here.

$ ls -l    
total 105107
-rw-rw-rw-   1 devuser   1000_shared  53814712 Feb 08 19:23 sn0441.mp3
$ df -kP .
Filesystem           1024-blocks      Used Available Capacity  Mounted on      
/dev/emmc/user0         13826032   9625376   4200656      70%  /               


9625408  - 9625376  = 32k.... not anywhere near 96MB.

 

I can also confirm this issue has been observed on 10.0, 10.1 and 10.2.1.

 

Has anyone else seen anything like this? I'd certainly appreciate advice on how remove a file that was recently being used by the MediaPlayer.

 

Cheers,

Eric

 

 

 

Developer
Posts: 6,152
Registered: ‎07-05-2012
My Device: Playbook, Dev Alpha C, Z10 LE, Z30
My Carrier: Orange

Re: Persistent Phantom Files - Storage Leak.

The way file systems work and especially for flash is often more complicated than that.

If you try and write back a larger file than the space available does it work?


If you've been helped click on Like Button, if you've been saved buy the app. Smiley Happy

Developer of stokLocker, Sympatico and Super Sentences.
Developer
Posts: 76
Registered: ‎02-08-2013
My Device: z10
My Carrier: rogers

Re: Persistent Phantom Files - Storage Leak.

You're right, there are clearly some innerworkings I'm unfamiliar with. As I downloaded more and more to the SD card the space used was not proportional to the size of the downloaded files. Then as the free space approached zero writing to files started to generate errors - as expected.

 

I then went on to start deleting files that were currently buffered by the media player, regardless of the space reported by 'df' as long as I freed up enough space to accomodate the new files everything worked. I was happy to see this.

 

Is there any files system documentation that decribes this behaviour?

 

I'll chalk up the two incidents where free space + used space didn't add up to the capacity of the storage device to bugs that have since been fixed - until it happens again Smiley Happy

 

One last question, why was it that if the file wasn't being used by the media player the space would become immediatley available?

 

Thanks,

Eric

Developer
Posts: 6,152
Registered: ‎07-05-2012
My Device: Playbook, Dev Alpha C, Z10 LE, Z30
My Carrier: Orange

Re: Persistent Phantom Files - Storage Leak.

[ Edited ]
Developer
Posts: 76
Registered: ‎02-08-2013
My Device: z10
My Carrier: rogers

Re: Persistent Phantom Files - Storage Leak.

The SD Card doesn't use the QNX6 file system, it uses fs-dos.so which isn't power safe. Furthermore, the QNX6 filesystem wouln't explain the behaviour I'm reporting.

 

Eric

 

 

Developer
Posts: 6,152
Registered: ‎07-05-2012
My Device: Playbook, Dev Alpha C, Z10 LE, Z30
My Carrier: Orange

Re: Persistent Phantom Files - Storage Leak.

[ Edited ]

Actually that link was just meant as a general pointer to the documentation that exists.

Like most UNIX systems any number of filesystems can be used.

 

I looked but couldn't find anything specific to your OP.

 

[Edit] Link changed to the one I meant to link to.


If you've been helped click on Like Button, if you've been saved buy the app. Smiley Happy

Developer of stokLocker, Sympatico and Super Sentences.
BlackBerry Development Advisor
Posts: 683
Registered: ‎11-29-2011
My Device: PRIV
My Carrier: Rogers

Re: Persistent Phantom Files - Storage Leak.

I pinged the filesystem guys for some insight.

It's possible that your stopped media player still has a file handle open.  you could check with "pidin fds" perhaps.  Though I think mm-renderer may own the file descriptors. 

Deleting a file just removes an entry from a directory.  Space is probably not properly reclaimed until all references to the file are closed.  There may either be a background service which periodically cleans things up, or the filesystem won't bother trying until some other operation triggers it to go off and check.

BlackBerry Development Advisor
Posts: 683
Registered: ‎11-29-2011
My Device: PRIV
My Carrier: Rogers

Re: Persistent Phantom Files - Storage Leak.

Okay, here's the reply from the filesystem folks.  Hope this helps clear things up!

 

 

Per POSIX behaviour, file contents are not to be freed until all open handles to the file are freed.  If we cannot free  the space (shutdown for instance), we will recover the space during startup for the QNX6 file system.
 
In FAT formats, it is different because the file system does not have POSIX concepts such as names and inodes.  We do emulate the behaviour in our implementation, but the lack of power safe features means it is possible to lose that space if the media is removed or system powered down with open file handles.  You would have to use mass storage mode or place the card into a host OS to complete a check and fix of the system.

Developer
Posts: 76
Registered: ‎02-08-2013
My Device: z10
My Carrier: rogers

Re: Persistent Phantom Files - Storage Leak.

Excellent, thanks for the great response.

 

What seems to have happened on two occasions (to my self and reported by a user of my app) is that, on the SD Card (FAT fsys), for whatever reason, the disk appeared full but the space taken up by the did not add up to the capacity of the card.

 

I ran a gambit of tools on the disk and I couldn't find or release the space taken up by what I can only call phantom files. There were no errors detected either.

 

I was hoping the behavior I noticed while deleting a file being played could explain it but my experiments over the weekend failed to reproduce the problem - space was eventually made available as requests for it were made even though there wasn't the appearance of available storage.

 

Wouldn't it make more sense, especially for the FAT filesystem to report an error to QFile->remove() if the file is in use? QNX, Linux and Windows does (EBUSY). An app or user shouldn't be able to remove a file that's unremovable or in use at the time.

 

Regardless, I'll keep my eye on things and come back to this thread if I find a way to reproduce the issue or the problem is reported again.

 

Thanks for your time,

Eric

Developer
Posts: 1,265
Registered: ‎12-29-2010
My Device: PlayBook, Z10 LE, Dev Alpha C

Re: Persistent Phantom Files - Storage Leak.

I found this old thread while looking up considerations for app ideas that use the media player & SD card.   Just curious whether the OP ever found a reliable way to make sure all file handles are closed before deleting a file from the SD card.  For example, did you attempt to set the MediaPlayer's sourceURL property to a null string before deleting the file?  

 

It hardly seems like a "solution" for the space on the SD card to only be *eventually* reclaimed - if the user's SD card is full, and they delete old files to make space for new ones, but an app comes back and says "sorry but you still don't have any free space on your SD card" they are not going to be happy w/the app developer.