Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Java Development

Reply
New Contributor
Posts: 8
Registered: ‎12-28-2010
My Device: Not Specified

Character Encoding is Different on Different BB Models

I have discovered (the hard way, of course -- trial and error) that there is a difference in how different Blackberry models do character encoding.  I have an application that receives data encoded in "ISO-6659-1". The multi-byte characters are just fine when displayed on one model (Storm 9550 5.0.0.517) but show as garbage on another model (Storm 9530  5.0.0.328 Verizon).  When I convert the encoding  to "utf-8" using

 

s = new String(s.getBytes("iso-8859-1"), "utf-8");

 

the reverse is true.  So one model uses iso-8859-1 encoding and the other uses utf-8.  This was identified with the simulator packages, but shows to be also true with the one physical model I've tested (the Version 9530). 

 

Who knew?

 

Does anyone know a way to identify the default encoding being used by the device so I can automate the conversion without requiring my user to select an encoding scheme?

 


Developer
Posts: 1,807
Registered: ‎04-28-2009
My Device: Z10 (STL100-4)-10.3.2.858, Z10 (STL100-3)-10.3.1.2576, Z30 (STA100-5)-10.3.1.2582, Passport (SQW100-1)-10.3.1.2576, PlayBook (16GB)-2.1.0.1917
My Carrier: Verizon

Re: Character Encoding is Different on Different BB Models

System.getProperty("microedition.encoding");

 

No offense but I'm somewhat having a hard time believing what your saying because the String class's default encoding is "ISO-8859-1". Can you post a "raw" file of the data you download then try to display?

 

Also I think you made a typo and meant "iso-8859-1" instead of "ISO-6659-1". Unless you did mean that in which case disregard what I said before, you have to pass your data into the String class with that encoding in order to get it properly formatted.

---Spends time in #blackberrydev on freenode (IRC)----
Three simple rules:
1. Please use the search bar before making new posts.
2. "Like" posts that you find helpful.
3. If a solution has been found for your post, mark it as solved.
--I code too much. Well, too bad.
Developer
Posts: 19,636
Registered: ‎07-14-2008
My Device: Not Specified

Re: Character Encoding is Different on Different BB Models

Actually I too have seen a case where a font was displayed correctly on one device and not on a 'later' OS level one.  This was however changes in the Fonts on the device rather than encoding.  If you find that characters are not being displayed correctly on any device, I recommend that you add your own font. 

New Contributor
Posts: 8
Registered: ‎12-28-2010
My Device: Not Specified

Re: Character Encoding is Different on Different BB Models

rcmaniac25 :  It was a typo.  As you can see in the one line of code in my post I did use "iso-8859-1".     

 

I understand your skepticism but the smoking gun for me is that the behavior changes as described in my post when I convert the encoding using that one line of code.

 

When I get home tonight I will try your hint at identifying the encoding scheme using System.getProperty() and then automate whether I do an encoding conversion before displaying data.  If that doesn't work I will repost with the dump of raw data as you suggest.

 

Thanks for your help.

 

peter_strange:  In this case it's not a font problem.  For one, all tests were with the identical font.  For another, without changing the font, the problem is either solved or re-instated on the platform with or without the encoding conversion -- as I stated in my post.  But thanks for your comments.

Developer
Posts: 19,636
Registered: ‎07-14-2008
My Device: Not Specified

Re: Character Encoding is Different on Different BB Models

[ Edited ]

When you say the characters display as garbage, what do you actually see?

 

I'd really be interested in a 'dump' of this happening, as

1) I am not 100% clear (bear of very little brain...)., and

2) I did not think that ISO-8859-1 was a multiple byte encoding. 

 

So is it possible for you to give us a snippet that shows the problem.  Something like the following (which I've just coded and is probably completely wrong...):

 

byte [] inputBytes = { (byte)0x11, (byte)0x98, ....

String dump = "";

for (int i = 0; i < inputBytes.length; i++ ) {

   dump = dump + Integer.toString(inputBytes[i] + ":";

}

System.out.println(dump);

String testISOString = new String(inputBytes);

dump = "";

for (int i = 0; i < testISOString.length(); i++ ) {

   dump = dump + Integer.toString(testISOString.charAt(i)) + ":";

}

System.out.println(dump);

String testUTF8String = new String(inputBytes, "UTF-8");

dump = "";

for (int i = 0; i < testUTF8String.length(); i++ ) {

   dump = dump + Integer.toString(testUTF8String.charAt(i)) + ":";

}

System.out.println(dump);

New Contributor
Posts: 8
Registered: ‎12-28-2010
My Device: Not Specified

Re: Character Encoding is Different on Different BB Models

Here's a little more you can dig your teeth into.  I wrote a simple test program that encapsulates my problem.  The data is coming from a web service (from a PHP/MySQL service), downloaded to the BB and stored in a SQLite database.   Data originally was uploaded from a .NET thick client app on a PC using an upload service.  ISO-8859-1 is specified in the SOAP encoding argument on the server. The international (maybe not multi-byte) characters look fine on the PC and on the web browser.

 

Here's the test program, which pulls the data from the SQLite database and displays it.  I've narrowed it down to pulling one field of one record that illustrates the issue.  Below the code I've pasted images of the output on two platforms.  First the 9530, then the 9550.  The correct one is the Norwegian author Nesbo -- o with a slash through it.   (Sorry for the ugly code formatting -- couldn't get the tabs down to 4 after pasting!)

 

It seems like something is different in the getString() from SQLite on the two platforms and that is what is causing the problem. 

 

Any ideas?

 

 

public class Test extends UiApplication {

Test() {
TestScreen screen = new TestScreen();
pushScreen(screen);
}

public static void main(String[] arg) {
Test app = new Test();
app.enterEventDispatcher();
}


class TestScreen extends MainScreen {

TestScreen() {
String s = getData();

add(new BasicEditField("encoding: ", System.getProperty("microedition.encoding")));
add(new BasicEditField("platform: ", System.getProperty("microedition.platform")));
add(new BasicEditField("bytes: ", getBytes(s)));
add(new BasicEditField("unconverted: ", s));
String s1 = null;
try {
s1 = new String(s.getBytes("iso-8859-1"), "utf-8");
} catch (UnsupportedEncodingException e) {}
add(new BasicEditField("converted to utf-8: ", s1));
}

private String getBytes(String s) {
byte [] inputBytes = s.getBytes();
String dump = ":";

for (int i = 0; i < inputBytes.length; i++ ) {
dump = dump +
Integer.toHexString(inputBytes[i]) + ":";
}
return dump;
}

private String getData() {
String reposLoc = "file:///SDCard/Databases/SmallDB/";
String reposName = "smallDB";
String reposSuffix = ".db";
String d = null;

try {
URI dbURI = URI.create(reposLoc + reposName + reposSuffix);
Database db = DatabaseFactory.open(dbURI);
Statement stmt = db.createStatement(
"select col1 from t1 where id = 84922684555460817");
stmt.prepare();
Cursor c = stmt.getCursor();
if (c.next()) {
Row r = c.getRow();
d = r.getString(0);
}
else
d = "Not Found";

} catch (Exception e) {
String msg = e.getMessage();
}
return d;
}

}
}

 

9530.JPG

 

 

9550.JPG

New Contributor
Posts: 8
Registered: ‎12-28-2010
My Device: Not Specified

Re: Character Encoding is Different on Different BB Models

Stop the presses! 

 

I just saw this post which addresses the SQLite encoding issue.  This seems to address my issue.

 

End of thread.

Highlighted
Developer
Posts: 19,636
Registered: ‎07-14-2008
My Device: Not Specified

Re: Character Encoding is Different on Different BB Models

Yes the post that I incorrectly added to that Thread was suppsed to go here:  To complete the picture, I have added it.....

 

Not looked seriously at your response yet, but your comment about storing in SQLite reminded my of these problems attach I have been involved in:

http://supportforums.blackberry.com/t5/Java-Development/UTF-8-Encoding-in-SQLite/m-p/557782

 

http://supportforums.blackberry.com/t5/Java-Development/UTF8-encoding-and-SQLite-database-problems/m...

 

I wonder if you have an SQLite encoding problem. 

 

The code at the end of the the second Thread should be useful for you to try to figure out where the problem is introduced.  Can you use it to see if the problem is the data coming out of the SQLite database with the incorrect encoding?