Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Native Development

Reply
Developer
zezke
Posts: 1,004
Registered: ‎12-12-2010
My Device: Z10 (red Limited Edition)
Accepted Solution

QString::fromUtf8 not giving the expected result

Hi everybody,

 

I have the following data that gets send to me via a web service.

 

{"title":"Seconde Guerre mondiale - Wikipédia","content":"Théâtre européen     L’Europe au 1er septembre 1939   Après s’être assuré de ne pas risquer une guerre avec l’URSS en signant le Pacte germano-soviétique, Hitler lance ses armées sur la Pologne, l...","favicon":"http://fr.m.wikipedia.org//bits.wikimedia.org/favicon/wikipedia.ico"}

 If I wireshark it, the raw data is:

 

[truncated] {"title":"Seconde Guerre mondiale - Wikip\303\251dia","content":"Th\303\251\303\242tre europ\303\251en     L\342\200\231Europe au 1er septembre 1939   Apr\303\250s s\342\200\231\303\252tre assur\303\251 de ne pas risquer une gu...

 However when I parse this using the following code:

 

QByteArray content = networkReply->readAll();
qDebug() << QString::fromUtf8(content.data(), content.size());

 I get this result:

 

 "{"title":"Seconde Guerre mondia
le - Wikipédia","content":"Théâtre européen     L’Europe au 1er septembre 1939   Après s’être assuré de ne pas risquer une g
uerre avec l’URSS en signant le Pacte germano-soviétique, Hitler lance ses armées sur la Pologne, l...","favicon":"http://fr.m.wiki
pedia.org//bits.wikimedia.org/favicon/wikipedia.ico"}"

 As you can see the special characters are pretty messed up. Does anyone have a clue as how to fix this? All my browsers are able to correctly decode this.

-------------------------------------------
BlackBerry Certified Builder for Native Application Development -- Proud member of the Belgian BlackBerry Developer group
Samples: Park in Ghent
Feeling generous? Nominate me for BB Elite member!
Developer
DrShavargo
Posts: 133
Registered: ‎05-10-2013
My Device: Blackberry Z10

Re: QString::fromUtf8 not giving the expected result

The problem is being caused by UTF-8 bytes being interpreted as Windows-1252 (or ISO 8859-1) bytes. The char on this page displays the problem you're having. I don't know how to set the decoding type for text recieved from a server, but hopefully someone can come along and help, or you can find something by digging a little.

----------------------------------------
Remember to mark the thread as solved at the post that solved your problem, and if you like a post, like it!
Developer
Zmey
Posts: 1,512
Registered: ‎12-18-2012
My Device: PlayBook, Z10, DAC

Re: QString::fromUtf8 not giving the expected result

[ Edited ]

Hi,

 

RSS feed is not always in UTF-8 format. First line in XML should specify the encoding used, for example:

<?xml version="1.0" encoding="ISO-8859-1" ?>

If encoding is absent, I think it's safe to assume that it's UTF.

 

If you use Qt classes for parsing the XML, it should handle the encoding automatically. Otherwise you'll need to decode the data using QTextCodec specifying a proper encoding, which can be obtained by peeking into first line of the stream:

QByteArray encodedString = "...";
QTextCodec *codec = QTextCodec::codecForName("...encoding name...");
QString string = codec->toUnicode(encodedString);

 


Andrey Fidrya, @zmeyc on twitter
Developer
zezke
Posts: 1,004
Registered: ‎12-12-2010
My Device: Z10 (red Limited Edition)

Re: QString::fromUtf8 not giving the expected result

It's not an RSS feed, it's a web service. You can find it here in case you would want to try.

 

No luck so far, using QTextCodec just seems to give the same result.

-------------------------------------------
BlackBerry Certified Builder for Native Application Development -- Proud member of the Belgian BlackBerry Developer group
Samples: Park in Ghent
Feeling generous? Nominate me for BB Elite member!
Developer
Zmey
Posts: 1,512
Registered: ‎12-18-2012
My Device: PlayBook, Z10, DAC

Re: QString::fromUtf8 not giving the expected result

I've tried to fetch it using curl:

 

curl -D header.txt "http://beread.azurewebsites.net/?url=http://fr.m.wikipedia.org/wiki/Seconde_Guerre_mondiale" > data.txt

 

and (basically does the same):

 

wget --save-headers "http://beread.azurewebsites.net/?url=http://fr.m.wikipedia.org/wiki/Seconde_Guerre_mondiale"

 

In header the charset is specified as utf8:

 

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/plain; charset=utf8
Server: Microsoft-IIS/8.0
Set-Cookie: ARRAffinity=b93b68bda46fccf89558f4cb61f103d490e49b8a60a3bc456498700b1bda1592;Path=/;Domain=beread.azurewebsites.net
X-Powered-By: Express
X-Powered-By: ASP.NET
X-Powered-By: ARR/2.5
X-Powered-By: ASP.NET
Set-Cookie: WAWebSiteSID=2f34566f50c8454893239977de500d9a; Path=/; HttpOnly
Date: Wed, 26 Jun 2013 19:30:39 GMT

The data seems to be in JSON format and has UTF-8 charset:

{"title":"Seconde Guerre mondiale - Wikipédia","content":"Théâtre européen     L’Europe au 1er septembre 1939   Après s’être assuré de ne pas risquer une guerre avec l’URSS en signant le Pacte germano-soviétique, Hitler lance ses armées sur la Pologne, l...","favicon":"http://fr.m.wikipedia.org//bits.wikimedia.org/favicon/wikipedia.ico"}

 Could you try fetching the data using curl or wget? Maybe it depends on our geographical location.

 

It's possible that the data gets damaged somewhere else in the app, because the conversion to QString looks correct. You could also try dumping the data to a file right after receiving it as QByteArray and then downloading this file to a PC and inspecting it there.

 


Andrey Fidrya, @zmeyc on twitter
Developer
zezke
Posts: 1,004
Registered: ‎12-12-2010
My Device: Z10 (red Limited Edition)

Re: QString::fromUtf8 not giving the expected result

[ Edited ]

The output of the web service doesn't differ here seems to be correct utf8 data. Here's how I request and process the data&colon;

 

QUrl serviceUrl = QUrl("http://beread.azurewebsites.net/htmlLink?url=" + this->pageUrl.toString());
QNetworkRequest* request = new QNetworkRequest(serviceUrl);
request->setRawHeader("Accept-Charset", "utf-8");
request->setRawHeader("charset", "utf-8");

 

void WebpageFetcher::onPageRequestFinished(QNetworkReply* networkReply)
{
	qDebug() << "WebpageFetcher:: onPageRequestFinished";
	QByteArray content = networkReply->readAll();
	if(networkReply->error())
	{
		qDebug() << "WebpageFetcher:: error while fetching web page: " << networkReply->errorString();
		qDebug() << "WebpageFetcher:: emitting error";
		emit fetchError(networkReply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt(), networkReply->errorString());
	}
	else
	{
		QTextCodec *codec = QTextCodec::codecForName("UTF-8");
		QTextDecoder* decoder = QTextCodec::codecForName("UTF-8")->makeDecoder();
		QString string = decoder->toUnicode(networkReply->readAll());
		qDebug() << string;
		QString contents = QString::fromUtf8(content.data(), content.size());
		bb::data&colon;:JsonDataAccess jda;
		QVariant list = jda.loadFromBuffer(content);
		if(jda.hasError())
		{
			bb::data&colon;:DataAccessError theError = jda.error();
			emit fetchError(1337, theError.errorMessage());
		}
		else
		{
			if(cancelled)
			{
				return;
			}
			QVariantMap result = list.value<QVariantMap>();
			pageTitle = QString::fromUtf8(result.value("title").toByteArray().data());
			qDebug() << "Title: " << pageTitle;
			pageContents = QString::fromUtf8(result.value("content").toByteArray().data());
			qDebug() << "Content: " << pageContents;
			QString iconUrl = result.value("favicon").toString();
			qDebug() << "Favicon: " << iconUrl;
		}
	}
}

 I am going to try to save it to a file and open it on my pc.

-------------------------------------------
BlackBerry Certified Builder for Native Application Development -- Proud member of the Belgian BlackBerry Developer group
Samples: Park in Ghent
Feeling generous? Nominate me for BB Elite member!
New Contributor
man202
Posts: 7
Registered: ‎02-24-2013
My Device: BlackBerry Z10 & Q10

Re: QString::fromUtf8 not giving the expected result

Hi,

 

Can you try

QByteArray content = networkReply->readAll();
qDebug() << (QString) content;

 and see what are the outputs?

 

Regards,

Syam

Developer
zezke
Posts: 1,004
Registered: ‎12-12-2010
My Device: Z10 (red Limited Edition)

Re: QString::fromUtf8 not giving the expected result

[ Edited ]

Bug squashed! I was doing QString::fromUtf8 twice, causing the garbage data. Also do NOT trust the output of qDebug()!

-------------------------------------------
BlackBerry Certified Builder for Native Application Development -- Proud member of the Belgian BlackBerry Developer group
Samples: Park in Ghent
Feeling generous? Nominate me for BB Elite member!