Java Development

[shendz]

Hi experts,

My first post here.

I am building a Java app for Blackberry that needs to send relatively large data (in hundreds of kbs) to a web server.

Since mobile internet connection is not fast enough transferring that much data will take a long time.

Question is, how can I compress that data using gzip and handle it on my web server.

Currently I have

String data = "a=b&c=d";
OutputStream outputStream = httpConnection.openOutputStream();
outputStream.write(data.getBytes());
outputStream.flush();

I am guessing blindly that I need to do something like

GZIPOutputStream gzipOutputStream = (GZIPOutputStream) httpConnection.openOutputStream();
gzipOutputStream.write(data.getBytes());
gzipOutputStream.flush();

Is that correct? Also, how should I handle it on my webserver?

Would my webserver unzip that data automatically, so I can just get the values of parameters a and c normally?

Should I send different Content-Type?

Please advise. Thanks in advance.

[rcmaniac25]

I can tell you that:

GZIPOutputStream gzipOutputStream = (GZIPOutputStream) httpConnection.openOutputStream();
gzipOutputStream.write(data.getBytes());
gzipOutputStream.flush();

won't work.

You need to change the first line to:

GZIPOutputStream gzipOutputStream = new GZIPOutputStream(httpConnection.openOutputStream());

 As for the rest of it there are others with a better understanding of what you would need to do.

[Ted_Hopp · Accepted solution]

LIke rcmaniac25 posted, you need to wrap the output stream that you obtain from the connection object inside a new GZIPOutputStream. To get any compression, though, you need to also specify the compression level.

As to whether the data will be unzipped on the server side, it depends on the server. At a minimum, you would need to set the content-encoding property for the connection before posting the data. I strongly recommend setting the content-length; to do that, write the compressed data into a byte buffer. I also suggest setting a user-agent header because some servers mishandle requests that come without one. Finally, I suggest adding a x-rim-transcode-content header with value "none" since the data isn't meant for a browser. (The server should also set that header in the response.) This is particularly important if going through MDS or BIS.

Putting it all together, it might look something like this:

String data = "a=b&c=d";
NoCopyByteArrayOutputStream bos = new NoCopyByteArrayOutputStream();
GZIPOutputStream gos = new GZIPOutputStream(bos, GZIPOutputStream.COMPRESSION_BEST);
gos.write(data.getBytes());
gos.close();
int len = bos.size();
// unqualified constants below are available if your class
// implements net.rim.device.api.io.http.HttpProtocolConstants
httpConnection.setRequestMethod(HTTP_METHOD_POST);
httpConnection.setRequestProperty(HEADER_USER_AGENT, "MyUserAgent");
httpConnection.setRequestProperty("x-rim-transcode-content", "none");
httpConnection.setRequestProperty(HEADER_CONTENT_TYPE,
    CONTENT_TYPE_APPLICATION_X_WWW_FORM_URLENCODED);
httpConnection.setRequestProperty(HEADER_CONTENT_ENCODING, "gzip");
httpConnection.setRequestProperty(HEADER_CONTENT_LENGTH,
    Integer.toString(len));
OutputStream os = httpConnection.openOutputStream();
os.write(bos.getByteArray(), 0, len);
os.close();

If you are POSTing something besides url-encoded name-value pairs, of course, you should use the appropriate content-type header value.

I believe (although I've never tried it) that Apache will honor the content-encoding header on POSTs, provided mod_gzip is turned on (which it usually is). Most other web servers (Microsoft IIS, Sun Java Web Server, Nginx, etc.) also support HTTP compression.

I suggest that you test all this with a simple script that just echoes your POST data. If it echoes as expected, you're good to go (but see below).

If the server doesn't automatically decompress the request, then you'll have to handle it in whatever script is processing the POST. Php, Perl, Ruby, etc. all have great tools for dealing with gzip'ed data. You'll just have to deal with parsing the url-encoded name-value pairs explicitly. But there are great tools in all those languages for that, too.

Be aware that there may be limits to the amount of data you can POST in one request. You might have to consider breaking down your posts into manageable chunks of data.

[shendz]

Thank you for the replies.

They are very helpful.

However, I am still unable to read it from my server.

So, I use PHP and printing $_REQUEST gives me unreadable strings, which makes me think that mod_gzip is not on (I use Godaddy by the way).

So, I write gzdecode() from http://www.php.net/manual/en/function.gzdecode.php#84174

Still, the request parameters are not readable.

Any other great idea?

[Ted_Hopp]

I'd suggest doing a hex dump of the data you are sending (the contents of the byte buffer) and also print a hex dump of the data received at the server. If they aren't identical, then the problem is in the communication link; if they are, the problem may be in the compression/decompression pairing. After gzdecode, what does the string look like? Also, it can't hurt to test your gzdecode: does it correctly recover a string that was gzip-compressed by, say, a gzip command-line utility?

[marchywka]

if you control both ends you are fortunate and could consider our own compresson

scheme. That is, most compression systems either rely on some apriori belief about file contents ( say image compression) or none  ( zip for example). If you know certain things about your file you may be able to do better yourself jutst by sharing dictionaries or somethng ( send "1" instead of " user first name for user who was in idaho at time of data entry").

[shendz]

Thank you for replying =)

I think I know how to take it from here.

[jaywhy13]

I had to do this very same thing, so just adding to this thread in case someone Googles and finds it like I did. One or two quick pointers, 

• Remember nothing under 200B is gzipped. So check the encoding when you come back...
• Make sure you use httpConnection.setRequestProperty(HttpProtocolConstants.HEADER_ACCEPT_ENCODING, "gzip") as opposed to HEADER_CONTENT_ENCODING. From what I understand I think the latter is a response header, the former tells you what content types you (mobile) can accept. I presume httpConnection.getEncoding() is the same as httpConnection.getHeaderField(HttpProtocolConstants.HEADER_ACCEPT_ENCODING).

In my scenario, I was trying to unzip the content I received. I was using Apache and it turns out that Apache seems to use gzip by default now because I didn't have to do anything special. Once I set the header to tell Apache I can receive gzip it went ahead and sent it to me  

Here's my snippet...

HttpConnection httpConnection = (HttpConnection) connectionDescriptor
.getConnection();

// Set headers if GZIP is enabled...
if(useCompression){
// httpConnection.setRequestMethod(HttpProtocolConstants.HTTP_METHOD_POST);
httpConnection.setRequestProperty(HttpProtocolConstants.HEADER_USER_AGENT, "Blackberry Device");
httpConnection.setRequestProperty("x-rim-transcode-content", "none");
httpConnection.setRequestProperty(HttpProtocolConstants.HEADER_CONTENT_TYPE,
HttpProtocolConstants.CONTENT_TYPE_APPLICATION_X_WWW_FORM_URLENCODED);
httpConnection.setRequestProperty(HttpProtocolConstants.HEADER_ACCEPT_ENCODING, "gzip");
}

// Go ahead and make the request!!!!
int status = httpConnection.getResponseCode();

String contentEncoding = httpConnection.getEncoding();

InputStream is = httpConnection.openInputStream();
String response;
byte[] rawData;
// Now check on the encoding...
if(contentEncoding != null && contentEncoding.equals("gzip")){
// The data is compressed
int length = 0;
StringBuffer rawResponse = new StringBuffer();
GZIPInputStream gzipInputStream = new GZIPInputStream(is);
byte [] responseData = new byte[256];
while (-1 != (length = gzipInputStream.read(responseData))) {
rawResponse.append(new String(responseData, 0, length));
}
response = new String(rawResponse.toString());
rawData = response.getBytes();
} else {
rawData = net.rim.device.api.io.IOUtilities
.streamToBytes(is);
response = new String(rawData);
}

// Response is now in "response" as a String!

Hope this helps someone!!!

[peter_strange]

Thanks for this.

Juat a minor point, I am not sure about this code:

// The data is compressed
int length = 0;
StringBuffer rawResponse = new StringBuffer();
GZIPInputStream gzipInputStream = new GZIPInputStream(is);
byte [] responseData = new byte[256];
while (-1 != (length = gzipInputStream.read(responseData))) {
rawResponse.append(new String(responseData, 0, length));
}
response = new String(rawResponse.toString());
rawData = response.getBytes();

Converting things to String and then back to bytes seems like it could cause encoding problems and it seems like a bit of an overhead too.

Can you replace al the above with jus the following?:

GZIPInputStream gzipInputStream = new GZIPInputStream(is);

rawData = net.rim.device.api.io.IOUtilities.streamToBytes(gzipInputStream);