02-06-2009 08:33 AM - edited 02-06-2009 08:39 AM
Just wanted to highlight a problem I have had with a transcoder used by Vodafone (supplied by Novarra) in the UK. I’ll bet there are other carriers who use similar transcoders and I suspect they will cause similar problems.
Before I start, be aware that I am not a http expert, some of what I am saying is completely new to me. So I’m probably using the wrong terms etc. Sorry. If you find something wrong, please post a correction.
The application picks up and processes http pages from a web site. It simply issues an HTTP POST and then reads the response. This application works fine when using BES connection and using Wi-Fi. I was trying to extend it to use WAP 2.0 and Direct TCP (on a new device).
When it didn’t work, I trapped the data that came back from my POST and discovered it was an http page, which, if it had been displayed, would have started with the following lines:
“First time user
Vodafone Mobile Internet
Some sites are already mobile friendly. Many more are not and may contain features that won't work on your phone.
We can modify these sites to improve your experience.
I did some research and discovered the following links:
The first is a summary, the second I suggest you review at your leisure.
I think this means that Vodafone are attempting to transcode the mobile pages. In an effort to warn the user, and give the user the opportunity to turn this off, they send this supposedly one off warning page and give the user the opportunity to tailor the transcoding service. HOwever this assumes the actual user is initiating the request using a Browser.
The application did not expect that data and was not processing it correctly.
I reviewed the html returned and discovered that I could go to the following page using the WAP browser on the device and control the settings:
If you are a Vodafone UK user, try this page, scroll down and look for the Settings section, then click on Web preferences. I had the option "View pages with speed and layout improvements” checked. I swapped to “Viewing pages without…” and things started working.
But there are two problems with this:
1) how do you tell users to do this, and
2) even if they do it, they then loose the value of the transcoding service, which they might find useful.
In an effort to allow my applciatin to work whatever the Vodafone setting was, I tried various options, including POSTing with various cache options set, and then using the .htaccess file (it is an Apache Server, no idea of the IIS equivalent, sorry) to disable transformation. Here is an example of what I tried in the Server – this will make sense to Apache users I hope:
Header unset ETag
Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate, no-transform"
Header set Pragma "no-cache"
Header set Expires "Wed, 11 Jan 1984 05:00:00 GMT"
This actually made matters worse. With the .htaccess file in place, I got an ‘invalid Header’ Exception if I tried with the speed/layout improvements turned off and HTTP 502 error code (generic gateway error I think) with it turned on. Looking at the Headers, it was clear that the gateway was playing with them – when I tested using Wi-Fi, I got the Headers I expected.
Basically it seems that the Novarra gateway ignores the W3C standard "no-transform" cache directive which was created precisely to deal with situations like this. And it mucks round with the Headers as well.
It would seem that, if you are attempting to develop a Mobile application and considering using WAP 2.0 or Direct TCP, then you are going to have to watch out for transcoding screwing your pages up and/or a ‘first time’ page when you first connect.
But who said this was easy anyway…..
The only recommendation that I have out of this is that you must save, for debugging, the Header and contents of any request that is returned to you. If I had not been able to do this, I would never have found the problem. I was fortunate that in this case I was testing in-house. For a product that in a customer’s hands, you should have a method of getting this data sent back to you for analysis.
I will try to add to this as and when I find out more. I hope this helps someone else who gets a similar problem. And if you have already had a similar problem, I’d love to hear what you have found out….
Editted to fix a few typos....
02-06-2009 11:39 AM
Thanks Peter. This is an excellent summary of the proplems encountered when connection via gateways that use content-adaptation services. Volantis is another vendor that offers this type of product, and I can understand the motivation for these products to optimize the server responses provided that they offer a way to disable it for specific request such as those originating from a custom application rather than the standard device browser. I'm surprised that they don't offer this feature. Is this something that we can request from Novarro? I'm happy to assist from a RIM perspective.
02-06-2009 01:55 PM
Thanks Tariq, good to know.
Found some more useful links regarding this problem.
Here is good explanation of what Vodafone are doing and why.
There are also links off that page that describe how to get the real user agent, and how to set the No-Transform cache control so that your pages get passed back correctly. And I found more in the forums associated with this site:
And it works! Well sort of. Following is what I currently have in the .htaccess file and I do seem to be seeing the data correctly with the setting 'view the pages with speed and layout improvements".
# Force no caching
Header set Cache-Control "no-cache, no-store, must-revalidate, max-age=0, proxy-revalidate, No-Transform"
Header set Pragma "no-cache"
And when they say no transform, I suspect they do mean that the page is not transformed, however it is still gzipped on the way through.
Unfortunately, when I turn this setting off, I still get the header exception. I might have to try this on another device (currently testing on a Bold) to see if the header exception is a bug in the BB. However I think my next test will be a socket connection - I can't imagine what this thing will do if I try to open a socket on port 80.....
Anyway two more recommendations/suggestions:
a) Make sure that your pages are set to disable transformations
b) Make sure that your http processing code handles gzipped responses.
04-20-2009 06:44 PM
05-06-2009 12:41 PM
I have recently had the following comment from another person who was testing this:
"The Novarra server messes up content for Direct TCP (HTTP but not socket connections) and WAP2 (HTTP but not socket connections)."
06-09-2009 06:21 AM
Also see this Thread:
It appears Novarra has been implemented in Verizon network as well. The referenced link is very useful:
07-30-2009 12:29 PM
For Vodafone UK it turns out that they have explicitly set their device such that the WAP2 service book is the “preferred” network connection. The underlying implementation on the device will actually disregard your APN information in this scenario to leverage the preferred network connection mechanism. This explains the behaviour we are seeing in the posts above.
So, the way that you will work around this issue is by essentially indicating that you specifically don’t want to use the preferred network connection. You would do this by providing an “empty” Connection UID parameter on your network call.
HttpConnection conn = (HttpConnection)Connector.open( “http://www.yahoo.com;deviceside=true;ConnectionUID
11-24-2009 05:28 AM - edited 11-24-2009 05:31 AM
I know I'm posting to an old thread but realised there's no "accepted the solution" (other than logging onto vodafone and telling them no). I've tried adding a black ConnectionUID to the request and so far that's not working... I'm going to give it another go now.
Did any one find a way of resolving this on the server side to stop this from happening?
11-24-2009 05:45 AM
"Did any one find a way of resolving this on the server side to stop this from happening?"
There are two issues on this Thread
1) Content being messed up
2) Connections being forced down WAP - the blank ConnectionUID is fox for this
I presume you are talking about (1). The only information I have on this is referenced in this Thread.
What are you seeing? What connection method are you using?