Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Java Development

Reply
New Contributor
Posts: 8
Registered: ‎11-04-2009
My Device: Not Specified

Need Expert View\Comments on Process very large size data on BB(Multi Theading)

Hi Guru


I am writing a Mobile Application for BlackBerry on JDE 4.6, currently I am stuck on one issue.


Basically the logic of application is something like..


At the startup from Splash Screen  a Web Service be called which download JSON Data from Server(WS will send 2000 records in compressed format, and app will unzip it and Parse JSON and after that will store data in a Vactor, this process will be continues for 2,00,000 Records) approximate size of these records is  10 MB.) I am displaying download progress, its take approximate 70 Minutes time to complete download all Hotels Records.


So, Above logic is working fine, but the time which its take is not something which mobile user will be accept, so I have to go with any alternate solution(Multithreading kind of logic), so App can complete downloading  quickly. I tried to use 2 Theads one for Downloading, and second for  Parse Data and insert into Vactor, but unfortunetly its not giving me desired result, maybe I have used wrong approach, so I would be happy to know what would be your view\approch if you will be writing code for this application…


Note: As per App logic first time data must be download from Web Server, so I can’t go with way to embed all data with application, it must be download from Web Server.


Any Solution\comment  is Welcome…



Thanks

Suresh Nariya.

Developer
Posts: 19,636
Registered: ‎07-14-2008
My Device: Not Specified

Re: Need Expert View\Comments on Process very large size data on BB(Multi Threading)

[ Edited ]

I agree with the thought of splitting the Threads up as you have done, however it does not surprise me that the time taken was still significant.  It seems likely to me that the network time is the most significant contributor to the delay.  To prove that, you could just throw away the downloaded data and time that. 

 

You could be making the download process more complicated than it needs to be by encoding the encrypted data to Base64 or attempting to process the data using some sort of stream processing.  So have a good look at that processing. 

 

In addition, I'm not sure if you really are using ZIP compression, but that is probably not the most efficient form for a Blackberry device, when processing it in this way.  GZIP would be much better.

 

I've a load more other thoughts, but am interested to hear what your results are from the download processing before suggesting anything else to look at. 

 

Edit: Having read RexDoug's Post, I think the point he makes is a very good one.  Also if you have 2,000,000 rows with 10Mb data, that is 5 bytes per row, which doesn't sound reasonable. 

Developer
Posts: 4,764
Registered: ‎07-21-2008
My Device: Not Specified

Re: Need Expert View\Comments on Process very large size data on BB(Multi Theading)

I think your approach is flawed.

 

Why download 2M records to the blackberry? The information will be obsolete by the time you finish downloading it.  This is 1980's approach,  not 2010, and it is not a "wireless" approach.

 

Most folks will not wait. They'll think your app is broken.

 

And to make matters worse, many Blackberries will have insufficient storage to perist this much data.

 

Retrieve the data as needed from the web service. One approach would be to return only the hotel info for the current locale of the device, or possibly the destination city, or whatever.

 

For example, I wrote an application that provides restaurant information, The data is requested from the server for the immediate vicinity of the device.

 

Otherwise, I would have to store 10M records on the device.

 

Smiley Happy

 

Developer
Developer
Posts: 324
Registered: ‎05-26-2009
My Device: Not Specified

Re: Need Expert View\Comments on Process very large size data on BB(Multi Theading)

Have you also explored the option of creating a blackberry specific mobile website for this specific problem?

Its an alternative to  creating custom java  apps that render regular news/content/lists etc.

Developer
Posts: 141
Registered: ‎02-10-2010
My Device: 9500
My Carrier: Airtel

Re: Need Expert View\Comments on Process very large size data on BB(Multi Theading)

My two cents:

- Download only limited records based on your criteria. Example, city, hotel category, tags, etc.

- Download first hand very limited information for the records downloaded. The details should be queried further on user selection.

Developer
Posts: 638
Registered: ‎07-02-2009
My Device: Not Specified

Re: Need Expert View\Comments on Process very large size data on BB(Multi Theading)

I agree with Pooja,

 

we have developed a similar app, in which we have different XML files as webservices and depending on the user selection we connect to particular WS and parse the xml content for display.

Thanks and Regards,
PraveenGoparaju.
New Contributor
Posts: 8
Registered: ‎11-04-2009
My Device: Not Specified

Re: Need Expert View\Comments on Process very large size data on BB(Multi Theading)

Thanks Peter for sharing your view, I used Gzip, and regarding record size actually its more then 10 MB , as there is one field which having HTML Content.

Thanks RexDoug for your comments, Actually I not described application correctly, the logic of Application is something like, user must be download all data into local for a Region(Assume regions like USA, UK, Asia ), and then he must be able to view all data without internet connection, in future whenever application will get Internet connection it will download latest data from server, so I must allow user to download entire database into local, and whole functionality should be working within local data. And Application don't have any location aware system, so there will be not any meaning like to download data for that particular location only.

Developer
Posts: 17,012
Registered: ‎07-29-2008
My Device: Z10 LE, Z30, Passport
My Carrier: O2 Germany

Re: Need Expert View\Comments on Process very large size data on BB(Multi Theading)

i would suggest to put the file on sd card then. you can download it on your PC and copy it to your BBs sd card. downloading 10MB over a mobile is a feat on itself.

----------------------------------------------------------
feel free to press the like button on the right side to thank the user that helped you.
please mark posts as solved if you found a solution.
@SimonHain on twitter
New Contributor
Posts: 8
Registered: ‎11-04-2009
My Device: Not Specified

Re: Need Expert View\Comments on Process very large size data on BB(Multi Theading)

Thanks Simon, But it will be not possible to have manually download that file, actually data is comming from various feeds, on server there is logic for process some feeds\third party Web services, are preparing data, and that data will be reflect users on their deviec for respective regions. As App will be avaiable on BB Store also, and user will purchase Regions, which they want.

 

Thanks,

Suresh Nariya.

Developer
Posts: 17,012
Registered: ‎07-29-2008
My Device: Z10 LE, Z30, Passport
My Carrier: O2 Germany

Re: Need Expert View\Comments on Process very large size data on BB(Multi Theading)

well, best of luck selling your 70 minutes download app Smiley Happy

----------------------------------------------------------
feel free to press the like button on the right side to thank the user that helped you.
please mark posts as solved if you found a solution.
@SimonHain on twitter