Java Development

[TOH_IRRD]

Hi all.

I have run into a problem with the socket demo code found in the sample programs workspace.  The original code works fine as do most of my modifications, with one exception, but first some background on the changes I have made.  I am sending contents of a file one byte at a time across a wireless connection so I have modified the socket demo code to not be dependent on knowing the length of the string you want to send.

Modification 1 to the socket demo:  The code sending and reading the byte or a string to a port has been moved to a void method called exchange so it may be easily and repeatedly called. 

Modification 2:  Instead of transferring hello and goodbye I use a loop to generate a string based on the integer counter for the loop so it is easy to track at what iteration the process failed. 

On different computers the results are slightly different, and by putting in sleep commands every couple hundred bytes transferred it prolongs the occurrence of the error which is the following:

Device Error (DE427) -BlackBerry Smartphone Simulator
Message Queue Full
Message Options are

> Exit

> Ignore   

> Ignore All DE427's      

> Save log...

This error occurs even when I am not printing to the screen and since computer speed / inserted delay times make a difference of when this pops up, I am sure it has to do with internal workings of the compiled program or garbage collection.  By pressing Ignore, the program continues normally until all bytes are done.  This problem has only been discovered on the simulator.  I haven't tested it with the device itself yet.  

Thanks for your help.

[MSohm]

It is recommmended to queue data on the BlackBerry handheld and send in larger chunks as opposed to sending a character at a time.  Every time the write method is called on a stream, a BlackBerry handheld expects a TCP acknowledgement indicating that the server has received the data.  Sending a large amount of data in very small chunks by frequently calling the write method can result in a large number of TCP acknowledgements to be queued on the BlackBerry handheld.  This can cause the buffer that queues these acknowledgements to overflow, resulting in lost TCP acknowledgements.  The connection then does not receive the TCP acknowledgements, making the BlackBerry handheld think that the packets did not reach the server so it throws an IOException stating that the connection has timed out.

To resolve this issue your application needs to lower the frequency of the calls to the write method.  The recommended approach is to queue up the data within your application and send it in larger chunks.  However, there may be cases where this is not possible due to server side limitations.  In this situation you could insert sleep commands within loop that is calling the write method.

[TOH_IRRD]

Hi Mark, Thanks for the reply to my question.

When sending larger amounts of data across a socket the problem I have encountered is the server is required to know how many bytes are being sent.  My program is designed to transfer the contents of a file to the server only when the blackberry is in range of the network, which it wont always be.  The problem this leaves me with, is the server (upon a timeout) doesn't know if it has all the data from that last "chunk" or whether the blackberry has again gone out of range.  I have been unable to write a loop that will dynamically detect the length of the string transmitted across the socket.  I would presume because it is desired to have a socket timeout that this dynamic length detection will not be possible.

If I were to use a set size of say 10 bytes to transfer at one time and the file ended after 4 of those bytes, I would need to fill 6 more characters with some sort of indicator saying that they aren't actually part of the data.  Unfortunately the file could be made up of any type of character, space, new line, etc.  (I have been unable to find the ASCII code for EOF).  So I don't think this is terribly efficient.   

My preferred solution would be to some how access this message queue in order to monitor its size and put in my wait statements accordingly so that I am not waiting when I don't need to be.  Is there anyway to access this message queue?

Thanks for your assistance. 

[MSohm]

Have you tried using '\0' to end your files (a null string terminator)?

There is no way to monitor this queue.  It is internal to the BlackBerry.

[TOH_IRRD]

The contents of the file produced is currently out of my control but I will make the recommendation to use a null string terminator for the end of the file as that would make my life easier. 

With respect to reducing the number of bogged down TCP acknowledgments is it the writer or the reader that is generating this problem.  My write and read statements are immediately after each other, I don't see how the expected acknowledgment statement could be getting bogged down when I haven't moved on to write the next byte of data until valid confirmation that the byte received is the byte sent.  I have only included the code of this process that is involved in causing the message queue error.  There is also some processing, error trapping and exception handling but its not relevant to my question. 

int i=0;
try { 
     _out.write((data+""), 0, length); 
     while(_in.ready() == false) // only try and read the input stream if it is possible to
     { 
        Wait.fracSec(i); 
        //_screen.updateDisplay("Brief pause until input stream can be read, "+ i+ " millisec"); 
        i++; 
      } 
      input[0] = (char)_in.read();// Read the first character into the input array of size 1. 
} 
catch(IOException e) { } 
catch(NullPointerException npe) {}

Thanks for your help.

[peter_strange]

I've only just skimmed this Thread, so please take the following with a pinch of salt, it might not be relevant.

I would not consider sending a file a byte at a time.  On a real wireless network, with the latency and the overheads that Mark has talked about, the file transmission will take a very long time.  I guessing you will have unhappy users with high data usage charges and slow devices. 

I recommend that you complicate your processing and design some control mechanisms.  I'm sure if you look around you will find network protocols that will help you with this, like FTP. 

Just as a simple example, you could send a block of data, up to the maximum that your Server will support, and prefix the data with control information that includes:

a) whether this is the last (i.e. end of file)

b) the length of data being sent

c) the offset in the file for this data (so that the Server know where to put it and if it has missed something).

Also I can't recommend null termination, it fails immediately you send a file that contains nulls......

Basically I think the solution to reducing the number of bogged down TCP acknowledgments is to reduce the number of transmissions.

Also in your original post, you indicated that you had a "Message Queue Full" exception.  This might be the TCP Queue.  I'm not sure but it could also be that you are trying to do some of this transmission on the Event Thread.  Can you confirm that your code is running in a separate Thread?

As noted, I might have missed something, there is lots to read.  But hopefully this helps.

[TOH_IRRD]

Hi Peter, Thanks for the reply.

So just to confirm the concept, A TCP acknowledgment is not the same as the _in.read verification step.  is there someway that I could view the contents of the acknowledgment message.  I know Mark previously told me that the queue was inaccessible, but I am now wondering about looking at the contents of the message as opposed to the entire queue.  Odds are this is also not possible. 

I have seen in other posts that this message has been seen when this is repeated from the event thread.  And I thought I had it in its own thread.  But I am not 100 % sure on how to check it.

The contents of this connectivity and exchange routines are in a class:   

public class EstablishConnectionThread extends Thread

Actually I can outline the important parts of the program I have so far. 

Class containing main method initializes the event listeners I need, and the class which creates the screen.  This class shares much of the same structure and code as SocketDemo.java.

When an option to run the program is selected a new class is called to get the initial status of all the listeners to determine whether the minimum requirements for the program have been met.  Depending on the results the thread is run using:

EstablishConnectionThread ect = new EstablishConnectionThread();

new Thread (ect).start();

So I think I have created a new thread, to handle this task. 

If this was insufficient information to determine if I was executing a new thread or if I am doing this incorrectly please let me know. 

NOTE:  I currently haven't terminated or interrupted any threads or anything yet.  (except what is already done by the event listeners)

Thanks again for your help.

Message Edited by TOH_IRRD on 03-04-2009 10:01 AM

[peter_strange]

Responding to various pats of your post.

"Mark previously told me that the queue was inaccessible"

I would go with what Mark said.  I think the queue you are talking about here is part of the protocol stack that mere application programmers like ourselves have no access too.  Have a look at the TCP Stack layers and you will see what I mean.  So you need to work on reducing your own acks (the _in.read verification) because that will reduce the requirements for the underlying acks.

"EstablishConnectionThread ect = new EstablishConnectionThread();
new Thread (ect).start();"

Looks like you are running in a separate Thread to me too.  To be absolutely sure you could test, which you can do with code like

if ( Application.getApplication.isEventThread() ) {

System.out.println(....);

}

[TOH_IRRD]

One more question in proceeding with this task. 
I have increased the number of characters being sent across the socket thus increasing the time it takes to perform

the send, and reducing the number of write commands, so TCP acknowledgments are no longer bothering me.  (on the

simulator anyway). 

Now I am looking to handle one particular scenario of data transfer interruption in which the server essentially

becomes disconnected and data transfer will fail.  For sure I know that the write command will fail, yet even though

I have it in a try Catch statement to IOException and NullPointerExceptions and RuntimeExceptions the catch statements never get entered.  Near as I can tell however, the write fails as expected but the catch doesn't get entered and the program dies and the eclipse debugger shows me this screen:

UiEngineImpl.pushModalScreen(Screen) line: 893
Source not Found.
Edit Source Lookup Path... (button)

I have seen this screen before but it was fixable by browsing to my project directory.  Additionally when I have

seen this message there hasn't even been a problem in the code, just the compiler's path to the appropriate file. 
I do not know the directory or project to browse to so that eclipse can locate the class containing this line for

the modal screen. 

If run on the simulator not in debug mode, the blackberry displays,
JVM Error 104 Uncaught: Runtime Exception
Scroll to continue

Which tells me the program won't run properly even if the debugger is not engaged. 

So if anyone knows how I can update my paths within eclipse (If I am right in my diagnosis that this is the problem) I would appreciate it. 

Thanks.

[TOH_IRRD]

One more piece of information, I don't know if it will be helpful.  This runtime exception doesn't occur when the socket, server, or coverage is interrupted and is only exchanging one single character.  It only occurs when the socket is trying to transfer a string.