Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Java Development

Reply
Contributor
yoshiegg
Posts: 22
Registered: ‎06-02-2010
My Device: 8330
My Carrier: Rogers & Bell
Accepted Solution

Limit on the input size for Base64InputStream.decode?

Hello everyone, I am working with V4.5 target system and wrote an app that downloads Base64 encoded data from within a XML file. However on large files I throw an IOException when attempting to decode the byte data. I verified I am indeed pointing to the correct Base64 data. Since the data is too large for a String I am using the decode method that accepts the byte[], offset , length parameters. the byte[] array is 301304 bytes (294K) in size. If it's too large is there any other method to convert this?
Please use plain text.
Developer
Ted_Hopp
Posts: 1,305
Registered: ‎01-21-2009
My Device: Not Specified

Re: Limit on the input size for Base64InputStream.decode?

Where is the byte data coming from? It would help if you posted some code and an indication of where the IOException is being thrown. Does the exception have a message?

 

By the way, Strings in Java can be much longer than you seem to imagine. In theory, it's limited to Integer.MAX_VALUE 16-bit characters. That's about 4 GB worth of data. In practice, the length is usually limited by available memory.




Solved? click "Accept as solution". Helpful? give kudos by clicking on the star.
Please use plain text.
Contributor
yoshiegg
Posts: 22
Registered: ‎06-02-2010
My Device: 8330
My Carrier: Rogers & Bell

Re: Limit on the input size for Base64InputStream.decode?

Here is some more code, However it appears size related because this code works sometimes, only failes when the file size is large, My only option is brute force trials to find the cut-off point where the method starts to fail. The e.getMessage() returns a null and the return data (bDecodedFile) is null.

 

 

thanks for the help this is a real head scratcher.

 

 

m_RawData =  The entire XML file in byte[] array.

bDecodedFile = the decoded file container declared as byte[]

 

Begin Code ----------------------------------------

            // Get the pointers to the start and end location of the Base64 encoded data within the XML file by doing a string search.
            int iFileStart = getIndexOf("<filecontent>", 0);
            int iFileEnd = -1;
            if(iFileStart > 0) {
                iFileEnd = getIndexOf("</filecontent>", iFileStart);
            }
            // Make sure the iFileStart and iFileEnd pointers make sense.
            if((iFileStart < iFileEnd) && (iFileStart > 0) && (iFileEnd > 0)) {
                // Adjust the file end to point to the location at the beginning of the XML termination tag.
                iFileEnd -= 14;
                // Calculate the length of the file from the two offsets.
                int iLen = iFileEnd - iFileStart;
                // Validate the length.
                if(iLen > 0) {
                    // Copy the data from the original(m_RawData) array to the new array to make sure we are pointing to the correct area of the XML.
                    byte bf[] = new byte[iLen];
                    for(int iPtr=0; iPtr < iLen; iPtr++) {
                        bf[iPtr] = m_RawData[iPtr + iFileStart];
                    }
                    /* IF I USE THIS AS A TEST CASE NO EXCEPTION IS THROWN, The Base64 is created using the SAME method as the large file.
                    String szTestString = "dGhpcyBpcyBhIHRlc3Qgb2YgYSBzbWFsbCBmaWxlLg==";
                    bf = szTestString.getBytes();
                    */
                    // finally attempt to decode this.
                    try {
                        bDecodedFile = Base64InputStream.decode(bf, 0 , bf.length);
                    } catch (IOException e) {
                        m_ResponseErrorText = "ZW B64. " + e.getMessage();
                        m_XmlErrorOccured = true;
                    }
                }
            }

Please use plain text.
Developer
Ted_Hopp
Posts: 1,305
Registered: ‎01-21-2009
My Device: Not Specified

Re: Limit on the input size for Base64InputStream.decode?

I take it that getIndexOf returns the index of the first character past the target string? Just checking, because otherwise your indexes are not going to be correct.

 

I'd suggest not copying the bytes out of m_RawData. Just use

bDecodedFile = Base64InputStream.decode(m_RawData, iFileStart, len);

I'd also suggest testing your code by setting

m_RawData = "<filecontent>dGhpcyBpcyBhIHRlc3Qgb2YgYSBzbWFsbCBmaWxlLg==</filecontent>";

and exercising all parts of your code.




Solved? click "Accept as solution". Helpful? give kudos by clicking on the star.
Please use plain text.
Contributor
yoshiegg
Posts: 22
Registered: ‎06-02-2010
My Device: 8330
My Carrier: Rogers & Bell

Re: Limit on the input size for Base64InputStream.decode?

The indexes are fine, I did a compare using the debugger and a hex editor. I took your suggestion and my code performs fine, I made a small adjustment to handle the being at the zero position (never happens in real life) but that was not my issue. Here is the new code, Again my 2 lines of test code work, however if a large base64 is fed in I fail, I am working on finding the breaking point now bye reducing the size by factors of 2 and re-trying it.

 

-------------------------- new code ----------------------------------------

 

            // IF I USE THIS AS A TEST CASE NO EXCEPTION IS THROWN, The Base64 is created using the SAME method as the large file.
            String szTestString = "<heade>crapcrapcrap</header><filecontent>dGhpcyBpcyBhIHRlc3Qgb2YgYSBzbWFsbCBmaWxlLg==</filecontent></body>";
            m_RawData = szTestString.getBytes();
            
            // Get the pointers to the start and end location of the Base64 encoded data within the XML file by doing a string search.
            int iFileStart = getIndexOf("<filecontent>", 0);
            int iFileEnd = -1;
            if(iFileStart >= 0) {
                iFileEnd = getIndexOf("</filecontent>", iFileStart);
            }
            // Make sure the iFileStart and iFileEnd pointers make sense.
            if((iFileStart < iFileEnd) && (iFileStart > 0) && (iFileEnd > 0)) {
                // Adjust the file end to point to the location at the beginning of the XML termination tag.
                iFileEnd -= 14;
                // Calculate the length of the file from the two offsets.
                int iLen = iFileEnd - iFileStart;
                // Validate the length.
                if(iLen > 0) {
                    // Copy the data from the original(m_RawData) array to the new array to make sure we are pointing to the correct area of the XML.
                    byte bf[] = new byte[iLen];
                    for(int iPtr=0; iPtr < iLen; iPtr++) {
                        bf[iPtr] = m_RawData[iPtr + iFileStart];
                    }
                    // finally attempt to decode this.
                    try {
                        bDecodedFile = Base64InputStream.decode(bf, 0 , bf.length);
                    } catch (IOException e) {
                        m_ResponseErrorText = "ZW B64. " + e.getMessage();
                        m_XmlErrorOccured = true;
                    }
                }
            }

Please use plain text.
Contributor
yoshiegg
Posts: 22
Registered: ‎06-02-2010
My Device: 8330
My Carrier: Rogers & Bell

Re: Limit on the input size for Base64InputStream.decode?

I found the breaking point to be 64KB. Nor can you define a string > 64Kb it will cause the compiler to raise an IO error stating the COD file data section is too large. This brings back the days of programming the C-64 or the Vic-20...too bad I cannot use 6502 on the BB. Anyway....anyone have a solution the Base64 decoding a byte[] or stream that is > 64Kb in size?
Please use plain text.
Developer
Ted_Hopp
Posts: 1,305
Registered: ‎01-21-2009
My Device: Not Specified

Re: Limit on the input size for Base64InputStream.decode?

I just tested the following code and it ran with no problems (it printed "0123456789" as expected):

 

byte[] foo = new byte[100000];
for (int i = 0; i < foo.length; ++i) {
foo[i] = (byte)('0' + (i % 10));
}
String fooS = new String(foo);
System.out.println("foo[80000-80009]=\"" + fooS.substring(80000, 80010) + "\"");

 

So the 64K limit on string lengths is just not there. There is a 64K limit on cod file sizes, so any data structures written to a cod file (like compile-time string constants) need to fit in that. But for data on the heap, there is no such limit.

 

Some further points:

1) Why are you copying the base64 data into temporary byte array bf? (BTW, System.arraycopy would be a cleaner way to do this.) Use this code instead:

 

if(iLen > 0) {
// attempt to decode this.
try {
bDecodedFile = Base64InputStream.decode(m_RawData, iFileStart, iLen);
} catch (IOException e) {
m_ResponseErrorText = "ZW B64. " + e.getMessage();
m_XmlErrorOccured = true;
}
}

2) Where is the raw data coming from? Perhaps you can decode from that source directly.

3) If you need to store massive amounts of data as resources, gzip it with maximum compression before including it in the project. Then when you open an input stream to read the resource, just wrap it in a GZIPInputStream before reading it.




Solved? click "Accept as solution". Helpful? give kudos by clicking on the star.
Please use plain text.
Contributor
yoshiegg
Posts: 22
Registered: ‎06-02-2010
My Device: 8330
My Carrier: Rogers & Bell

Re: Limit on the input size for Base64InputStream.decode?

1) The only reason I was copying the array was to make sure my offsets where grabbing the correct data. I can remove that code and point directly to the source data like this.

 

bDecodedFile = Base64InputStream.decode(m_RawData, iFileStart , iLen);

 

 

2) The raw data is coming from a download, so it's received from a socket and assigned  into m_RawData.

 

3) The only reason I was including the string as a resource was for testing purposes. I was declaring a string initialized with the XML data, when the XML data was too large the compile failed. I don't need to overcome this problem in the real application. See the answer to #2, I cannot simulate the server pushing the file so I had to make a hard-coded source of data.

 

 

I would like to create a Base64Encoded string that is > 64Kb and attempt to decode it using my code. you have proved that the String wasn't causing the issue, however my problem is still unsovled.

Please use plain text.
Contributor
yoshiegg
Posts: 22
Registered: ‎06-02-2010
My Device: 8330
My Carrier: Rogers & Bell

Re: Limit on the input size for Base64InputStream.decode?

as a side note I can supply you with a working and non-working copies of Base64 encoded data if you like?

 

It's a ZIP file containing image (PNG) files that is then Base64 encoded and sent to the BB via a web service XML response.

 

I will repeat, the code works fine on ZIP files (base64 data) that appear to be < 64Kb in size, if receive and attempt to decode > 64Kb the method throws an exception. Doesn't matter if I used Strings, Byte[].

Please use plain text.
Developer
Ted_Hopp
Posts: 1,305
Registered: ‎01-21-2009
My Device: Not Specified

Re: Limit on the input size for Base64InputStream.decode?

Yeah, I hate it when I end up chasing down bugs in the diagnostic code.

 

I confirmed that there is indeed a problem decoding large byte arrays with Base64InputStream.decode. My guess is that the static decode methods use some fixed internal buffer to do their work. The solution is to read the data byte by byte. Instead of:

 

bDecodedFile = Base64InputStream.decode(m_RawData, iFileStart, iLen);

use this:

 

ByteArrayInputStream bis = new ByteArrayInputStream(m_RawData, iFileStart, iLen);
Base64InputStream b64 = new Base64InputStream(bis);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buff = new byte[1024]; // faster than byte-at-a-time
for (int len = b64.read(buff); len != -1; len = b64.read(buff)) {
bos.write(buff, 0, len);
}
bDecodedFile = bos.toByteArray();

That should work around the 64K limit that you uncovered.

 

RIM really should document this.




Solved? click "Accept as solution". Helpful? give kudos by clicking on the star.
Please use plain text.