Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Java Development

Reply
New Developer
Posts: 45
Registered: ‎03-04-2009
My Device: Not Specified

Re: Persistent storage on 872 MB Device Memory?


webmasterpdx wrote:

A better design would be to store it in a file on the SD card and read it in real time. Note that that is plenty fast to be visible as an animation or whatever you are doing.

 

Note that I just went through hell and back figuring out the memory internals and have it finally solved. This may be of use to you.

 

There is 1GB of fast flash storage internally. 128MB is used for apps and 872MB is used as device memory (I think persistent storage gets put in there). Remember rewriting flash thousands of time is not wise as this flash has 100K write cycles before it dies (thats reduced through wear levelling algorithms), but count on the number of times you can do that being limited. There is also 128MB of internal RAM. This is where data goes when you allocate memory (non-persistant storage). So, if you want to do this, I'd recommend you use non-persistant storage in conjunction with a file on the SD card.

 

Good luck.


 

@webmasterpdx 

 

You mentioned below that 872 MB is used  as device memory. Can this be used to store persistent objects ? What i m seeing in different threads on this forum is that this device memory is being used in the same was as the SD Card , that is for pictures,videos,songs,etc. The folder structure present in the device memoryalso seems to confirm the same.

 

If a lot of data has to be persisted, then it is not a good idea to use the 128 MB flash as it may take up lot of space and in case , memory is less, some data might be deleted by low memory manager.

 

Is there anyway we can store persistent objects on the 872 MB device memory ? If not , atleast can we write data to a file and store it on the device memory?

 

Appreciate your help.

 

Developer
Posts: 558
Registered: ‎11-25-2008
My Device: Not Specified

Re: Persistent storage

Sorry for not getting back to you sooner, I had to verify a few things. The one thing I'm not certain of is the size of OneNand memory.

It's unclear if it's 1Gbit or 1GByte. A teardown view that I looked at said 1GBit, which means only 872Mbit of storage.

 

Now that space is broken up into Persistent store AND file space (I'm unsure as to how it's broken up that way....I'll try to find out for you).

Next, the entire Flash Card is 8GBits or 1Gbyte of space (SDCard) that is all used as a file space.

 

I've seen titles of articles saying that it's possible to expand persistent storage into the filespace on the OneNand flash (not sure if that's possible into the SDCard space). Again, I'll try to find out. You can be searching too, and share what we find on this thread. I'd like to get this resolved.

 

So, if you have a large amout of storage (in the category of 1Gbits), I would NOT use the persistent space on the internal flash, but I'd use a file database of some sort. In reality, if I personally were doing that, I'd store it on the web someplace and access it using the app locally.....like Google Maps works.

That makes far more sense.

 

Good Luck and I'll post whatever else I can find.

 

Thx

-Donald

 

====================

Donald Murray

CTO IPPUB

 

"If this helps you, please click the kudos on this post"

 

 

1G of DDR (it makes more sense based on manufacturing sizes).

 

Now, that 872Mbyes of

New Developer
Posts: 45
Registered: ‎03-04-2009
My Device: Not Specified

Re: Persistent storage

Thanks webmasterpdx for the info.

 

I have also been searching the threads in this forum and through the documentation, but i have not found any thing that mentions this setup clearly.

 

So , if the 872Mbit of Storage is broken into persistent and file space, who handles the data storage  on 872Mbit ? Is it taken care of by the OS itself ( when flash memory is not available, does the OS automatically start using the persistent space on the 872Mbit)  or do we have the flexibility to make our application create persistent objects in the 872Mbit of memory witout using the flash memory? This is another thing that we need to find out.

 

If we dont have teh flexibility to use the persistent space  on the 872Mbit , then the only option is to use the filespace on the 872Mbit memory ( I m hoping that we have access to filespace atleast )

 

Regarding my requirement, The data that needs to be downloaded to the device is critical to the device users in the event of an emergency - meaning... In an emergency/disaster situation where the BES Server is down or not available , this data needs to be available to the users..Also this data which is available on the server is updated regularly.  So i m trying to find a way wherein, the data is sent to the users on a periodic basis ( whenever the data on the server is updated or at regular  intervals)  and is then stored persistently on their devices. In the event of an emergency, the users can use my app to pull up the data from persistent storage and can use it.  Of course, the amount of data being stored will have to be within the available memory on the device .. that we will make sure. So storing it on the web will not make sense here.

 

If i use a database , then as it has to be installed on all the devices ( there's  a large number of BB users)  and it adds to the cost.

 

I am looking around for answers and will update this thread when i find some.

 

Thanks,

 

Subash

Developer
Posts: 558
Registered: ‎11-25-2008
My Device: Not Specified

Re: Persistent storage

I have found more info.

I put a request on the java forum for more info and I found more myself.

 

I now believe that memory is arranged like this.

 

1Gbits of RAM

 

1Gbits of app memory (Fast execute in place flash).

 

1GBytes of Storage memory (Slower NAND) internally.

 

So, instead of 872Mbits/bytes, I now believe you have 1Gbytes.

 

OK, this is what I would do. I've been thinking about your application. I don't know what your app is, but I've written many apps with similar situations to this.

 

First of all, I most certainly would have the main database be online.

Then, I'd arrange things that as far as the user is concerned, his program accesses a database interface. This interface then uses some breakdown to address into the database. It would cache the most recently used parts of the database. The database should be broken down into different level's of chunks.

 

e.g. Lets say you are creating a database of stores, restaurants and services for different towns in the USA. Now, this database may be addressable based on a town, zip code within a town, stores, restaurants and utility services within the zip codes.

 

So, when you look for a store within a zip code, it caches just the stores within that zip code. Likewise if you search restaurants and stores within a zip code, it stores everything stored for that zip-code locally. Finally, you could just search the town for stores, so it could store all the stores for all the xip codes in that town. Finally, it could store all the info for that town in the cache.

 

 

So, certain patterns will be noticed. e.g. whenever I pass through Chicago, I may have to wait for another plane, so I might want restaurants around the airport.....but I'd never need anything else in Chicago. So, my cache would have that zip code with it's restaurants cached in my local space. Likewise, my home town of Portland Oregon would have everything cached for the whole city.

 

Now, whenever there is an update, it can be filed based on zip codes or towns.....this info is sent to my device so at it's earliest opportunity it clears the cache for the info based on zip code and towns that have been updated. (In reality, nothing gets cleared, but that flash memory space gets invalidated

 ).

 

So, when I access these areas again, they'll get cached again, with the latest data....and so on.

 

So, as long as the data is stored as an addressable cache system with content addressable memory (store the city, zip code and then the info). you only need to cache little bits of the full database, and it's not full of a giant hole (i.e. you don't have to allocate a space for the full database).

 

 That arrangement would work best.

 

I don't know what your app is, but you should be able to do something similar. In my example, if I don't have access to the internet, my app can give me whatever is cached, so it'll have everything in my home town and it'll have everything around the airport in Chicago. If I'm somewhere else that I've never been berfore, then it won't be there, but the odds are that if the network goes down, that I will be most likely within one of the areas already cached.

 

Note, don't count on your phone being broken. Count on it working. My phone (Storm) is supposed to be connected to the internet all the time.

 

Now, what you'll need to service such an app is a homepage with unlimited access and to put the database on there with a little web front end (like  a web form) to access it. I'm sure there are loads of apps out there to assist you in that part.

 

I hope that is of some help.

 

Thx

-Donald

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

New Developer
Posts: 45
Registered: ‎03-04-2009
My Device: Not Specified

Re: Persistent storage

Thanks Donald for the superb illustration . I was able to visualize a similiar one for my application.

 

I have a few questions, though :

 


I now believe that memory is arranged like this.

 

1Gbits of RAM  

 

1Gbits of app memory (Fast execute in place flash).

 

1GBytes of Storage memory (Slower NAND) internally.

 

So, instead of 872Mbits/bytes, I now believe you have 1Gbytes.


 

          Are we able to see all the above on the Bold ? Because what I see on My BB Bold is this

 

         Device Memory Total Space - 859.2 MB

 

         Application Memory Free Space - 36.8 ( Is this part of Device Memory or RAM or APP Memory or Storage Memory ? )

 

         A folder called /system/ : Total Memory - 99.7 MB . Used Memory - 92.1 MB .  The memory values for this one are almost the

         same everytime i check them.. there s no huge change - Under which category does this come under ?

 

         I m trying to map what is shown on my Bold to your structure .. But most probably,  the complete memory structure will not be

         visible  to the user , right ?

 

         When i persist data, i find my The application Free space memory reducing and there is no change in the Device Memory or

         the /system/ memory.

 

 


First of all, I most certainly would have the main database be online.

Then, I'd arrange things that as far as the user is concerned, his program accesses a database interface. This interface then uses some breakdown to address into the database. It would cache the most recently used parts of the database. The database should be broken down into different level's of chunks.


  

My app is an emergency contact list . My app will consume a web service to receive the contact information. The web service pulls the data from the database and other sources. The purpose for the app is to provide contact info to the user mainly when the server is down. So the complete data has to be available on the device at any given time.Whenever there is an update the app must be able to receive data from the web service and update its contents.

 

Now the data that can be sent from a BES Server per device is a maximum of 1 MB. And the data that needs to be retreived for my app is obviously ,way greater than 1 MB. So i have to get the data chunk by chunk - this can be done by specifying a criteria like the first 500 or 1000 contacts ( making sure that they all come under 1 MB) or by "Location"  as you suggested in your example. .

 

Now comes the storage/persistence part.

 

The data that is received must stored or persisted on the device for future retreival or access. This is where we need to know where exactly the data is stored , when we use the persistence model . Is it on the Flash Memory or the 1 GB/872 MB dev Memory or somewhere else ?

 

From what i have read in other threads in this forum, the general idea is that , if we use the persistence model, then the data is persisted in the flash memory which is shared by BB apps, 3rd part apps, user data and the RAM .

 Question Here : Flash memory used here for storing persistent data - Is it the 128 Mb flash or 872 MB Dev Memory or both ? When i persisted the data , my application memory free space reduced - this looks  like RAM.

 

This is fine if the data is small. But if the data is huge, then what would be the impact ? If it takes up the complete flash memory and RAM  which is shared by other apps , then its a problem. 

 

So is there a way we can store this data on the 1GB/ 872MB device memory through persistence model or file connection model?

 

 


 Now, whenever there is an update, it can be filed based on zip codes or towns.....this info is sent to my device so at it's earliest opportunity it clears the cache for the info based on zip code and towns that have been updated. (In reality, nothing gets cleared, but that flash memory space gets invalidated

 ).


 

 

When you say "caches" what do you mean ? Does that mean persisting the data onto flash memory or just storing it in RAM? I also didnt understand the part about flash memory getting invalidated

 

 


 So, as long as the data is stored as an addressable cache system with content addressable memory (store the city, zip code and then the info). you only need to cache little bits of the full database, and it's not full of a giant hole (i.e. you don't have to allocate a space for the full database).


 

 

In my case, i would need the complete contact information to be available at any time regardless how how its is stored ( by loaction or team or reporting structure..etc)

 

It would be really be good, if the data can be persisted on the 1GB memory, provided, it is not deleted by memory manager if the the number of persistent object handles is all used up. If persistence is not the right way, then i guess the only option is to use the file connecion apis to write data to the 1 Gb memory.

 

 I really appreciate your help on this. It would be great if you can share your thoughts on my observations above.

 

Thanks,

 

Subash

Developer
Posts: 558
Registered: ‎11-25-2008
My Device: Not Specified

Re: Persistent storage

Unfortunately, I don't have a lot of time today, but here is what I have been able to come up with.

 

First of all, the memory looks identical to the Storm based on the Bill of Materials Here. This is a site worth bookmarking if you have a bold.

http://www.phonewreck.com/wiki/index.php?title=BlackBerry_Bold

 

So, it's 1Gbits or 128MBytes of DDR RAM.

1Gbits of fast OneNand flash or 128MBytes which is the app memory.

1GBytes of MoviNand for your persistent storage and internal file system.

 

so, it's not 872MBytes any more, but 1GBytes total space for persistent storage and internal file system.

 

You cannot see the memories and sizes from the settings on the Blackberries. Something that annoyed me very much initially. I had to go out there and figure this all out. Nobody had a clue until those guys did the tear downs of the phones.

 

I have not worked with persistent storage myself yet, but I have seen plenty of references and questions on this forum. If you select in the search area at the top of this page and type in persistence and hit the little magnifying glass icon, it should search the entire forum for this and you'll see there are about a zillion posts on this subject, so the info you need must be in there someplace.

 

I am guessing but some of the apps use their own persisten storage that may be tied in with the app in app memory, but for the general user to access persistent memory, it's done mostly in the 1GBytes space (lets call that persistent storage space). Anyways, what I do not know is how much of this is persistent storage and how much is file space. If you have a file manager program (there are several free ones out there), you should be able to see /sdcard (thats your external card), and anything else is internal to the phone, so /system is probably the file system in the 1GByte storage space.

 

It could be that this space is dynamically allocated so if you allocate more persistence, it'll increase the size and make the file system correspondingly smaller.

 

I don't really have the time to help you with your specific app, but I can answer from what I remember your last post said....

 

OK, first, what I mean by a cache is an area local to your specific device that you store parts of the database as they are accessed by the user. I'm assuming it's a gigantic database. If it's a small database, then you could store it locally without any impunity, but if It's in the order of 1Gbytes, I wouldn't do that. I'd just do something like, lets say your app is a contacts database. Then I would create a local database of the size of (just randomly picking) say 8Mbytes in size. Now, lets say each record (that holds all the info on your client is stored in 1Kbytes of space). It makes things much easier if records are all the same size. You could try compression also to reduce the size used. Then, every time the user accesses a persons name, that record gets loaded in. You keep a track of all the entries in your local database using a linked list between the records. You also keep a track of the least recently used record. Then, when you want to access a record, you look for a free record. If there are none free, you delete the least recently used, and read in (from the internet) the new record and store it in that place. So, you are basically keeping a linked list of lets say n records, and 8K - n free records (where you can store new ones read in)...also a linked list. When the free list gets full, then you can delete the oldest or least recently used record and replace it with a new one read in from the internet.

 

You could also do it 10 records at a time or something like that too. Another compromise you could also do is store all the names locally, but not the 1K info related to each of them. That way, the user interface will seem smoother and only when he needs to see someone's data will it occasionally tell him to wait for a few seconds while it gets his info from the internet.

 

I hope that helps and makes sense???

 

Good Luck

Donald

===========

Donald Murray

CTO IPPUB LLC

 

"If this post has been helpful to you please click the Kudos button on this post"

Highlighted
BlackBerry Development Advisor
Posts: 15,753
Registered: ‎07-09-2008
My Device: BlackBerry PRIV
My Carrier: Bell

Re: Persistent storage

Memory size questions have been answered in this thread:

 

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&thread.id=30355

Mark Sohm
BlackBerry Development Advisor

Please refrain from posting new questions in solved threads.
Problem solved? Click the Accept As Solution button.
Found a bug? Report it using Issue Tracker