Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Java Development

Reply
New Developer
Posts: 17
Registered: ‎11-17-2009
My Device: Not Specified
Accepted Solution

InputStreams for HTTP[S] Response data: confirm minimal buffering/read-ahead?

[ Edited ]

I need to handle response data from a web service. The format of the response data is under my control and is returned as a stream (net.rim.device.api.crypto.tls.TLSInputStream) to my BlackBerry application. The contents of the stream is an XML stream that contains some simple header information and then one or more "chunks" of data. The data is compressed (gzip) and encoded (Base64). The BB app needs to unencode, decompress, and then process the data in the stream. For the purposes of my application, I never need the entirety of the data at once; true stream processing is what I am looking for. I have implemented a pipeline that, in pseudo-code, looks like this:

SecureConnection httpsConn; // already established

InputStream httpsStream = httpsConn.openInputStream();

InputStream compressedStream = new(Base64InputStream.decode(httpsStream));

InputStream is = new GZIPInputStream( compressedStream ) ;

int aByte = is.read();

The goal is to buffer as little data as possible so that operation on the BB side is not memory intensive as the data grows. The actual implementation of this pseudo-code works fine.

 

The question I have is: where can I confirm that the httpsStream I create from the httpsConn is not itself read completely into the BB by the RIM-specific code? In other words, if there there are 20MB of data in the stream, I don't want to find out that the stream has read the HTTPS data completely - all 20MB - and then made available. Instead, I want to know that only as much data is consumed as I have done is.read()'s, plus a little buffering perhaps, for network efficiency. A third way to put the question: I figure this is supposed to be the definition of a well-implemented InputStream, but I am having a hard time finding a definitive "yes, J2ME (or BB) InputStreams promise to read HTTPS data on demand, and not all at once".

 

I expect that tonnes of streaming audio and video apps are partial evidence that real on-the-fly data from the Net works.  Again, I've left out details like XML processing by SAX - this is a about the HTTP[S] InputStream's behavior.  But that brings up the fourth way to phrase my question: If I use SAX instead of a DOM tool to process my HTML because I want to keep watch over memory pressures from large data streams, will I get undone by buffering I cannot control in the low-level HTTP[S] InputStream?

 

Before you say, "HTTP[S] isn't where you should do streaming", this isn't streaming per se; it is instead a single - possibly large - response to a POST.  Very much a "typical" web interaction. 

 

If the answer changes based on OS version, then we can assume 4.6 or better is the target platform.

 

Thanks!

-Del

Developer
Posts: 1,474
Registered: ‎04-14-2009
My Device: Not Specified

Re: InputStreams for HTTP[S] Response data: confirm minimal buffering/read-ahead?

I believe some in some cases the HTTP connection implementation on BlackBerry does indeed read the whole response and only then lets you read the response as stream. You can avoid this by opening a TLS/SSL connection, forming and sending an HTTP request yourself, then parsing the response and reading the response body as it arrives from the network.

New Developer
Posts: 17
Registered: ‎11-17-2009
My Device: Not Specified

Re: InputStreams for HTTP[S] Response data: confirm minimal buffering/read-ahead?

Thanks, klyubin.  I'm still interested in hearing a little more detail.  I am happy to get a response stream, but I don't want it to be the result of reading the entirety of the response data before the stream can be "tapped".

 

So to be more clear: I am opening a SecureConnection (via javax.microedition.io.SecureConnection) and getting an InputStream from that.  E.g.:

public InputStream myFunc(SecureConnection theConnection) {

  InputStream theStream = theConnection.openInputStream();

}

I then read from this stream.  I want to be as sure as possible that when I read, say, 10 bytes, that I won't have, say, 64KB read under the covers.  I could easily see the implementation pre-reading a little bit for me, but I don't want to see the whole stream read ahead.

 

A pointer to explicit documentation would be really helpful - I've had a hard time finding such in the documentation I have read thus far.

 

Thanks again!

Developer
Posts: 1,474
Registered: ‎04-14-2009
My Device: Not Specified

Re: InputStreams for HTTP[S] Response data: confirm minimal buffering/read-ahead?

I don't recall any such documentation. All I recall is that I suggested this workaround to somebody on this forum and they later confirmed that it resolved the buffering issue (they were streaming audio as large HTTP responses -- the streaming started to work just fine as a result, without much latency).

Developer
Posts: 62
Registered: ‎05-29-2009
My Device: Not Specified

Re: InputStreams for HTTP[S] Response data: confirm minimal buffering/read-ahead?

I think you can call available() on the input stream to see how much data is buffered for immediate read. 

 

Of course to a degree the buffering is controlled by the windows at the TCP layer - if you want to confirm that the phone doesn't buffer loads and loads beneath you  as you pour data from the server you can look at the TCP trace and confirm that the windows get shut as the data flows.  There will be a reasonably sized buffer in the sockets layer - I doubt you can shrink it, but it equally shouldn't grow.

Developer
Posts: 62
Registered: ‎05-29-2009
My Device: Not Specified

Re: InputStreams for HTTP[S] Response data: confirm minimal buffering/read-ahead?

It also depends what kind of connection you've made.  If it is via MDS (BIS or BES) and the SSL isn't hosted on the device then the gateway may buffer your entire request (and probably change it to be "helpful").  But it won't buffer on the phone.