Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Java Development

Reply
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

Connecting your BlackBerry - http and socket connections to the world

[ Edited ]

 

 


 

Added Nov 6, 2009:

 

Along time ago (well it seems that way), I decided that the same network related questions were being asked again and again, so I decided to try to write a post that would answer most of the questions.

 

Over the intervening period RIM has introduced a number of improvements in the networking area. And with OS 5.0, they have I think managed to bring together a lot of the pieces and provide something that should reduce the complexity significantly. I’m talking here about ConnectionFactory and the related APIs. If you are developing for OS 5.0, sorry this Thread is NOT for you, review the documentatin that comes with OS 5.0. People who need to support earlier levels, read on.

 

 


 

 

Blackberry devices have the ability to connect to the world using TCP/IP and related protocols. 

 

However the process to configure and establish a reliable connection is currently not simple.  There is not a one-size-fits-all connection method that will always work.  We can argue as to where the problem lies, but at the end of the day, we all have to work within the constraints of the Blackberry environment.

 

Hopefully in the following Posts, we will explain the options and how best to use them.  I say we, as you read you will see that I don’t know much and a lot of this has come from other people.  So your input would be useful where you see something missing, not very well explained, or plain wrong.  I’m sure you will find examples of all of these….


Rather than presenting this as one long post, I have split it up into various sections, as follows:


- Introduction (this bit)

 

- Why does it need to be complicated?

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29104#M29104 

 

- Required Reading - what you must look at

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29105#M29105

 

- Overview, summarizing what can be done

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29106#M29106

 

- What can you do in the Simulator

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29107#M29107

 

- BES Connections, i.e. options and issues for the BlackBerry Enterprise Server User when connecting

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29108#M29108 

 

- BIS Connections, i.e. options and issues for the BlackBerry Internet Server User when connecting

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29109#M29109

 

- Heuristics, including the Network Diagnostic Tool - getting the BlackBerry to decide

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29110#M29110

 

- Wi-Fi and USB/Bluetooth connections

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29111#M29111

 

- Transcoders and other things that get in the way

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29112#M29112

 

- Other Network Stuff, including links to other valuable Threads:

http://supportforums.blackberry.com/rim/board/message?board.id=java_dev&message.id=29113#M29113


I'd like to try and maintain this 'structure', so will edit each Post as more details and corrections come along (all contributions will be acknowledged!).  This makes it possible to read this like a series of chapters, rather than a disjointed set of Posts.  Feel free to add Posts, or send me a note and I will link to the Posts (so you get the kudos!) or add to the Sections.  Or perhaps I can persuade someone else to help…  Volunteers?


Hope this makes sense and you agree this is a worthwhile exercise.  If so, perhaps we can vote to make this a sticky Thread?


Before I finish the intro, a few acknowledgements.  As noted, this information has been picked up over a period of time, from various people.  I can’t remember all of them, but there are some very good people on this forum who have helped in this through their posts including: (in alphabetical order), BBDeveloper, RexDoug and simon_hain.  Thanks guys.  But my biggest thanks goes to msohm, Mark Sohm from RIM, who has been helping me for years.

 

Edit: to point out this was not all my doing and to ask for Posts, I don't want kudos for just collecting other people's work. Furhter edit to add additional Link.

Message Edited by peter_strange on 04-08-2009 11:24 AM
Moderator Edit: Fixed typo.
Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

Why does it need to be complicated?

[ Edited ]

This is speculation, I do not and have never worked for RIM or any other mobile device company. So take your biggest pinch of salt and read on...My intention here is not to explain exactly what goes on; it is to create a "model" that you can use to understand what happens. 

 

The important thing to note is that the BlackBerry (Wi-Fi devices aside, we will come to them later) is not a standard IP device.  It does not have an IP address that you can easily use.

 

But TCP/IP is all about communicating from one IP Address to another.  Any communication must start with an IP address and be sent to an IP address.  So how can the BlackBerry device speak TCP/IP?

 

The answer is indirectly, the device has to use a proxy/gateway.  A bit like a NATing IP Router hides all the PCs in your company or home behind a single IP address.

 

So who provides this proxy/gateway?

 

RIM, your carrier, or your company (for BES devices) provide the proxy depending on the connection method. 

 

I believe you can see this yourself if you use one of those services that tells you who you are (I've seen this reported in the forum, I have never done it myself).  If you log on to one of these using the WAP Browser on the phone, you will find it is an IP address owned by your carrier.  If you use the Internet Browser, it will be a Rim owned IP Address.  Log on from a BES device, it will your company IP address. 

 

So your real IP traffic is 'tunneled' from your device, to the connection point (proxy) and from there travels by standard TCP. 

 

And how does the data get between the proxy and the device?

 

Sorry, I don't actually know this.  Magic?  Unfortunately, it the specifying of this tunnel that causes most of the problems. 

 

<If anyone would like to provide some information for this, I'd appreciate it.....>

 

Summary, why is it so difficult?

 

Because it is NOT actually standard TCP/IP, the communication gets tunneled, potentially through another protocol, to the proxy/gateway.  But Rim have tried to make it easy for us to code this in a standard way, while still trying to satisfy all the requirements of the carriers and the various methods of communicating.  Many options, Many carriers, it was difficult to start with, means it is not easy to do.

 

Alternatively RIM could have just used one carrier and provided only one way to connect and then it would be simple, just extremely restricted.

 

One final point.  Because there is not really an IP address on the device, you can’t have a socket to socket connection between applications. 

 

Moderator note:  Updated with a new post written by Peter.

Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

Required Reading

[ Edited ]

So you want to connect to your or someone else’s Server, to do http or socket connections?

 

Before you even think about what you need to do, you must review the following:

 

a) Network Transports Videos

There are two.  You should start with the first one:

http://www.blackberry.com/DevMediaLibrary/view.do?name=NetworkingTransports

and then review the second one:

http://www.blackberry.com/DevMediaLibrary/view.do?name=NetworkingTransportsII

The second actually re-explains most of the first one, and also describes the enhancements in OS 5.0, which look really useful.  Hopefully these enhancements will be provided as a ‘add-on’ for older OS levels too. 

 

b) KB article

What Is - Different ways to make an HTTP or socket connection

Article Number: DB-00396

http://www.blackberry.com/knowledgecenterpublic/livelink.exe/fetch/2000/348583/800451/800563/What_Is...

 

 

Still Confused?  Don’t worry, if RIM could explain it all in one KB article and/or one video, we wouldn’t need this post!

 

However before we start talking about doing networking, we must discuss a possible confusion/problem area – the Event Thread.  As Network I/O is a blocking operation, you must do it off the Event Thread.  But I don't want to get into what is the Event Thread in this series of Posts.  So, unless you understand how to code and get things on and off the Event Thread, I would recommend that you just steal someone else's code that works, and adapt it for yourself.  So what to steal?  Simple, something from the samples:

 

- socketdemo if you are doing a socket connection

- httpdemo, if you are doing an http connection.

 

So the next required reading is the Samples.  Open these in your IDE and look at the way the code is structured.  Note the way that Threads are used and that there is synchronization back to the processing that displays on the User Interface. 

 

Now review the API documentation for:

 

Connector

HttpConnection

 

OK, you got it all now?  If you have you are doing well.  If not (that all of us that are not genius level), you know where to look for reference information.

 

Moderator note:  Updated with a new post from Peter.

Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

Overview - summarizing what can be done

[ Edited ]

Here is a typical question that gets asked

"My goal is to create a seamless connection regardless of the carrier that does not require the user to configure any settings. And where the url suffix is static for all devices.
Is this Possible?"


The answer, at this time is unfortunately, no.  The objective of this section is to explain why not.


When connecting to a socket or http web service, you supply a connecting URL, The same sort of thing that you see in a Web browser, for example:
http://www.google.com


For a Socket connection, you will also specify a port. 


In addition to this, you also specify a String that defines the connection method that is to be used.  This connection string is the critical part.  What this series of posts is really all about is getting this String correct so that your application can connect.


So what options do you have for the connection string?

1) BES/MDS - ";deviceside=false"
Note MDS means Mobile Data Service, and in recent levels is suffixed with CS (for connection Service I think) to distinguish it from the Rapid Applicatin Development tool also called MDS.
2) BIS-B - required connection suffix will be supplied from RIM to Alliance partners
I think it is called BIS-B because it is through this Service that Blackberry Browser support is provided to BIS users.
3) Direct TCP - ";deviceside=true" + APN settings in the device or added to this connection String
4) WAP 2 - ";deviceside=true,ConnectionUID=" where the ConnectionUID comes from the Service Record as specified in the standard KB article
5) WAP 1 -


So getting your connection working is all down to getting the connection suffix specified appropriately for the device.


Aside: Note in some (typically older) devices, the device itself would attempt to determine the best connection method, if you specified nothing at all.  In fact I have some devices who will only connect correctly if you do this.  However these are the exceptions rather than the rules.


So can you just try each of these in turn until one works?  Sort of...


The first issue is that BES/MDS connection only works for BES devices, i.e. corporate devices.  These people will have administrators for each device and they can control, to a certain extent, what connections the device can make.  Typically they will force all connections to come through the BES/MDS route.  So for these devices, you must use ";deviceside=false".  Nothing else will work.  For some BES/MDS users, there is only one thing to try.

 

The second method, BIS-B is only available for Blackberry Alliance members, and you need to apply for it for an application.  I have always presumed that you need to do this because this is a service provided by RIM and they need to size their infrastructure to ensure that it will support the traffic.  But that is only speculation.  BIS-B is a reliable mechanism, it has worked for me where others have failed.  It is, as I understand it, widely available in North America and Western Europe.  I have no information for other parts of the world.

 

Direct TCP depends on carrier support.  To get this going, you need the APN information for the network your device is on.  This is widely available, since Direct TCP is used to support internet access from other platforms, not just BlackBerry devices.  If you know the carrier you are on, an internet search engine should find the APN information for you.  You can code this in the Options or in your connection suffix.

 

WAP 2 should be available on most carriers, and can be used to provide a seamless connection, because the connection suffix can be determined by your program.  However experience suggests that this is not as fast as other methods.  There is also the potential problem of transcoding (see later)

 

WAP 1 should be available on all carriers, but requires some carrier specific configuration.

So back to the original question?

"My goal is to create a seamless connection regardless of the carrier that does not require the user to configure any settings"

 

The general recommendation is to determine what sort of device you have.

For a BES user, use BES/MDS connection.

For a non BES user, use BIS-B if you are allowed to otherwise try WAP 2, Direct TCP and then WAP 1.

 

Edit: To fix typo - thanks mantaker.

Message Edited by peter_strange on 04-08-2009 10:06 AM
Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

What can you do in the Simulator

[ Edited ]

From the Simulator, you can connect using three connection methods:

 

1) BES/MDS

2) Direct TCP

3) WiFi

  

If you want to test using anything else, sorry, you can't.  To test your network code, you have to run it on a real device.

 

BES/MDS

 

When using BES/MDS you are simulating an Enterprise connection through the BES and on to the MDS.  To do this with your simulator, you must have access to the MDS Simulator (i.e. the MDS-CS Simulator must be started).  What is actually happening is that the Simulator is connecting to the MDS Simulator, which then routes the TCP/IP connection like it would in a real live BES installation.

 

Direct TCP.

 

Here the device (Simulator) takes advantage of the fact it is running in a real device with a real IP address, and can so, just connect.  You do not need to specify any TCP Options information for this to work (or they are defaulted to something correct - not sure!).

 

WiFi.

 

This information provided by klyubin.  To so this, firstly the Simulator you are using has to support WiFi!  Assuming it does. Then you need to configure the WiFi network on the simulator. It's the same as on an actual device: scan for a WiFI network, connect to it, and save it. WiFi-capable simulators come with the "Test WLAN" SSID network built in. Once the simulator is connected to the WiFi network, the name of the network appears in the top part of the home screen, same as on an actual device. This is when “;interface=wifi” will work just fine, again, same as on an actual device.

  

Things to be aware of when running the Simulator:

 

1) You can't connect directly to localhost or 127.0.0.1 using BES/MDS.  So if you have your Web Server running in the same machine as your simulator and you want to connect, use a URL that will resolve to your machine's real IP address, or Direct TCP/WiFi.

 

The reason you can't do this has been provided by msohm.  The MDS Simulator mirrors the behavior of the BlackBerry Enterprise Server here, which also prevents access to 127.0.0.1 and localhost.  If this was allowed, it would be possible for any application to make a connection directly to the server hosting the BlackBerry Enterprise Server (127.0.0.1/localhost = BES server, not device).  This would be a security risk.  Preventing this forces the application to be aware of the machine name or IP address of the BES if it needs to connect to an application running on the same server as the BES.

 

2) If you want to test the Browser on the Simulator, you must run the MDS Simulator.

 

3) It seems that some applications, for example Blackberry Maps, do not need the MDS Simulator to work.

 

4) As a suggestion, to make life easier in testing, in your connection suffix logic, you can have a statement which says:

 

if ( DeviceInfo.isSimulator() ) {

    suffix = ";deviceside=true";

} else .....

 

5) The JDE is fairly Java level tolerant, you can run JDK 4.7 using Java 1.5 for example.  The simulators all seem to come with their own 1.4 level JVM.  However the MDS Simulator seems to be Java level specific, so you do need to get the level correct.  So if the MDS Simulator just starts and then stops, and you don’t see a black ‘DOS CMD’ screen displaying logging information from it (which by the way, can be very useful to watch sometimes), then check out this KB article:

  

Support - BlackBerry Mobile Data Service Simulator does not launch

Article Number: DB-00054

http://www.blackberry.com/knowledgecenterpublic/livelink.exe/fetch/2000/348583/800792/801079/Support...

Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

BES Connections

As you should have already figured out, the best way to connect from a BES device is to use the BES/MDS connection, i.e. specify ";deviceside=false" as your Connection String.


This is not guaranteed to work.  Remember that communication is routed to the MDS and it is the MDS that will then try to establish a connection to the URL you have specified.  This may or may not be inside the corporate network.  Assuming it is not, then this connection must be authorized through the corporate network (firewall and proxy server).  it could be blocked on its way.


So though it is easy to get DES/MDS connection working this does not guarantee end to end connectivity. 


BES users can, if allowed, also use any other the other connection methods that BIS users can use.  Note the "if allowed".  And in addition, this sort of processing often leads to a problem that is generically known as "Split-pipe". 


The problem described by “spilt-pipe” is simple. 


If an application can connect using the BES/MDS connection method, then it can connect to Servers that are inside the Corporate firewall.  This could be a company intranet site which posts profit forecasts, or internal sales figures and the like.  The sort of thing that the company probably doesn't want spread around.  So if an application connects in this way, it is defined to be an 'internal' application and is marked as such, for the life of the OS on the device.


However if the application connects externally, using one of the BIS connection methods, it has direct access to external Servers and can pick up, or supply information to them.  This will marked as an external application, for the life of the OS on the device.


If an application was allowed to do both, then in theory, it can copy data from an internal site to an external one, or the reverse.  This is really breaching the company firewall.  If you application attempts to do this, then in most environments, it will be terminated with an error message, saying it has attempted to connect inside and outside the firewall. 


But this is a problem.  If you try the BES/MDS connection method in your application and it fails, then you might be tempted to try one of the external connection methods. However even if this method could work, "split-pipe" might mean your application might not even start.


How do you get round "spilt-pipe"?  You can ask the BES administrators to allow it for your application.  Or you can wipe the device.  Tough choice.

Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

BIS Connections

On a BIS device we have, as we have already seen a number of connection methods that we can use.


To repeat, from the ‘what can be done post’ they are:

1) BIS-B
2) Direct TCP
3) WAP 2
4) WAP 1 -


If you were trying these methods, then the preferred order, based on various posts on this forum, seems to be:
1) BIS-B
2) WAP 2
3) Direct TCP
4) WAP 1

<further input sought for this section>


BIS Devices and split-pipe?

 

Can a BIS Device get Split-pipe problem?  I thought not, but I believe some posters have seen it.  I think the problem goes like this.  If your application attempts to connect using “deviceside=false” (or nothing at all), then I think the device marks you application and ‘internal’ before it even checks that there is in fact no BES support on the device.  Subsequently, if you try an external connection, it thinks you have already tried an internal one and will throw you out.  At least that it what it seems some fellow posters have seen, I believe with OS 4.5.  Never seen it myself. 

Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

Heuristics - getting the BlackBerry to choose

To do this, you need to detect the user environment.


RIM have provided, and extended with later OS’s, the CoverageInfo class, whose job it is to provide you with the information that you need to make this decision for yourself.


You need to review this class, at the level of OS that you are targeting (note like all BB development APIs it is forward but not backward compatible).


As an example of code that uses this, see this KB article and associated code:
 

What Is - Network Diagnostic Tool
Article Number: DB-00684

http://www.blackberry.com/knowledgecenterpublic/livelink.exe/fetch/2000/348583/800451/800563/What_Is...


I am not strong on this, I have code from a long time ago that looks directly at the Servicebook entries.  However I believe that CoverageInfo is basically doing the same thing, just packaged up and easy to use.  If so I have a word of caution.  On new devices and if someone has requested a refresh, ServiceBook entries (and so CoverageInfo) might not be present.  In this case I suspect you will get invalid information from CoverageInfo.  make sure your device is stable in terms of connection before you use CoverageInfo to detect the possibilities.  This is especially important to corporate users who might get Split-pipe issues if you choose wrong.


Note that the Network Diagnostic Tool is also a very useful tool when:

a) You have a potential customer who has no idea what connection options they have signed up for
b) You have a customer who is having network problems.

I would recommend, that if you start having serious issues like this, that you create your own version of this tool and provide it as a testing mechanism, to your support staff.

Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

Wi-Fi and other connection methods

We will leave Wi-Fi to later.


It is possible to USB connect (and I believe Bluetooth connect, though I have never done it), and make your connections via this. 


I only have experience with this in BES environments.  In its most simple case, you can USB connect the device to the BES and then all traffic can go directly to the BES via the USB connector.  In this way you can get emails to the device, Enterprise activate the device etc, without even needing a SIM card.


Of course this is typically impractical for most reasonable sized installation.


However what is also possible is to USB connect to your own PC, and then, provided you can provide the Device manager with the correct routing information to the BES on the correct port, you can do everything without a SIM.  This is different to using Wi-Fi, because in my experience, you can't Wi-Fi connect to a BES, without first having wireless connected to it (I assume to prove that the connection is there and secure).  See Wi-Fi later for more.


This mechanism is called Least Cost Routing, and it seemed that RIM were going for it big time at one stage, but I think this has been over taken by Wi-Fi.  So this is really mentioned for completeness.


I have never seen this or tried it for a BIS device.  If any one has and wishes to contribute, please do....


Wi-Fi


You can direct your connection over Wi-Fi, or you can let the device choose Wi-Fi for you.  Let us look at the Wi-Fi ‘for free’ first.


In my experience, the circumstances in which you get Wi-Fi for free (i.e. without user coding) depends on the environment you are in.

BES connection - getting Wi-Fi for free


Wi-Fi for free in a BES device, will only occur, when the Wi-Fi can connect back to the home BES.  So to get Wi-Fi, you must have an active wireless (SIM card) connection.  This will contact the BES, and then the Wi-Fi will also try to connect the BES.  If it can get through (which in most circumstances, will mean that the Wi-Fi network is within the corporate network) then the device will swap to using Wi-Fi for everything.


I believe that it should be possible to set and use VPN for this Wi-Fi connection, but I have no experience with this.


BIS Connection - getting Wi-Fi for Free

Wi-Fi for Free with a BIS device, seems to happen for any BIS-B connection, where the device has a Wi-Fi connection active.  Posts on this forum also report that Direct TCP can be routed over Wi-Fi for free.


If you want to direct your requests over Wi-Fi, then you specify another connection suffix ";interface=wifi".


Given the comments above about Wi-Fi for free, there seems little value is directing the use of Wi-Fi in a BIS environment.  However you might want to send large files (pictures/videos) via Wi-Fi only. 


In a BES environment, directly specifying Wi-Fi seems to mark you as an external application and can therefore, cause a spilt-pipe problem.  So unless you know you are only going to use data that is external to the corporate, you should not use Wi-Fi directly. 


You can detect Wi-Fi being supported directly and active from the ServiceBooks, or use RadioInfo (on later OS levels).

Please use plain text.
Developer
peter_strange
Posts: 19,526
Registered: ‎07-14-2008

Transcoders and other things that get in the way

This section discusses two things:

a) Proxy Servers/Authentication
b) Transcoders


Proxy Servers/Authentication


You will find, especially using http connections, that some organizations will not let you onto an external network, without forcing your user through authentication.  This is a pain, but relatively easy to get around.  Just follow this KB article:

How To - Implement basic HTTP authentication
Article Number: DB-00468
http://www.blackberry.com/knowledgecenterpublic/livelink.exe/fetch/2000/348583/800332/800429/How_To_...


Transcoders

 

These bits of software are designed to take web pages for larger screen devices and convert these to make them more mobile device friendly.  In fact the BlackBerry MDS is an example of this, it will shrink pictures on Web pages so that they can be displayed on the Browser on the device more conveniently.


However if you attempting to get, in your program, a web page, then these things will not give you the original web page, they will give you their reduced version of this.  Unfortunately, if you want to interpret the data you get, it might be reformatted by the transcoder.


Vodafone (and presumably there Verizon) use transcoders.  Not sure about other carriers.


To make sure that this does not effect you, you really have to have the ability to dump out the data you get from an http request, and make sure it is what the Server has sent, not what the transcoder has assumed will display correctly in a Browser.  This mainly effects http requests made via WAP gateways, though I believe I have seen it with socket connections and Direct TCP.

Please use plain text.