Welcome!

Welcome to the Official BlackBerry Support Community Forums. This is your resource to discuss support topics with your peers, and learn from each other. New to the forum? Please visit the ‘Getting Started’ link below.
inside custom component

Java Development

Reply
New Developer
chimera
Posts: 51
Registered: ‎04-23-2009

Lower IO write missing package

Hi,

 

I tried to send large data using outputstream.write. Because the data is big, so I try to send them piece by piece. I think the outputstream write is slower than filestream read. So, the buffer may changes before the output write. Then I will have bad data on the server.

 

I am using socketconnection, is there a way to confirm the write is finish, a lock, or wait? So, I won't overwrite the un-sent data in mydatabuf.

 

Thanks.

 

Lan

 

 

  do {
                 if ( (myFileSize - off) < bufSize) 
                    len = myFileSize - off;
                 else 
                    len = bufSize;
                if (len <=0)
                    break;
                myFileStream.read(mydataBuf, 0, len);
                myOutputStream.write(mydataBuf, 0, len);
                myOutputStream.flush();
                off += len;
                i++;
      }while (len > 0);

 

 

Please use plain text.
Developer
ydaraishy
Posts: 562
Registered: ‎09-30-2009

Re: Lower IO write missing package

Off the top of my head, here is one way I can see to do this.

 

Have two threads, one writing into the buffer and another one reading from the buffer. When the Writer thread has filled the buffer, raise a flag in the other Reader thread, Object.notify the Reader thread, and have the Writer thread run Object.wait. Meanwhile, that Reader thread currently leaves Object.wait and begins to read the filled buffer. Once the Reader thread has consumed the buffer, set a flag back in the Writer thread and call Object.notify on the Writer thread and have it fill more in the buffer.

 

Rinse and repeat. The two threads should continually signal each other to do its work, and the buffer will stay pristine.

Please use plain text.
Developer
Posts: 1,474
Registered: ‎04-14-2009

Re: Lower IO write missing package

[ Edited ]

FIrstly, you write code doesn't check how much was actually read into the buffer -- the code currently writes the whole buffer which may not have been fully read. This is bound to cause corruption. I think that's the root cause of your issue. Fix this first.

 

 

while (off < myFileSize) {
int chunkLength = myFileStream.read(mydataBuf, 0, Math.min(myFileSize - off, bufSize)); if (chunkLength == -1) { // Input stream reached its end -- handle this situation appropriately. // In this case throwing an EOFException is probably a good idea throw new EOFException("Input stream closed prematurely");
} myOutputStream.write(mydataBuf, 0, chunkLength);
off += chunkLength;
}
myOutputStream.flush();

 

 

 

Since reading and writing are performed sequentially, the read rate will be automatically throttled to the write rate -- you aren't attempting to read until the previous write operation completes. The code you have right now (if you add modify it to write out from the buffer only as much as was read in) should work just fine. It will read some data into the beginning of the buffer, then write all of this data out. Only when the data has been sent to the lower layers, will it read more data into the beggining of the buffer again, and so forth.

 

If you aren't reading fast enough, TCP will automatically tell the peer to send at lower rate. No new incoming data will overwrite old data in the buffer.

 

If you want to buffer some data before writing it out, you can simply wrap the output stream into a BufferedOutputStream. Usually, this class will not write anything out to the lower layers unless its buffer is filled, or unless you explicitly tell it so by invoking flush() or close().

Please use plain text.
Developer
Ted_Hopp
Posts: 1,304
Registered: ‎01-21-2009

Re: Lower IO write missing package

 


If you want to buffer some data before writing it out, you can simply wrap the output stream into a BufferedOutputStream.

 

Except perhaps BufferedOutputStream isn't In the BB API?

 




Solved? click "Accept as solution". Helpful? give kudos by clicking on the star.
Please use plain text.
Developer
Posts: 1,474
Registered: ‎04-14-2009

Re: Lower IO write missing package

Thanks, Ted_Hopp! I completely forgot about the lack of BufferedOutputStream. It's not hard to implement it though.

Please use plain text.
New Developer
chimera
Posts: 51
Registered: ‎04-23-2009

Re: Lower IO write missing package

Thankss for all your replies. 

 

Is it write a blacking operation? My thought is, the "write" won't return before it actually write out the data. And this thread should wait for return from write and then read next chuck of data. So, it shouldn't overwrite the buffer. I am kind of surprise the buffer been overwritten. >_<

 

Yes, I can add flag or some work around. But I want to use BB API right. I think I miss understood some part of the function...

 

Please use plain text.
New Developer
chimera
Posts: 51
Registered: ‎04-23-2009

Re: Lower IO write missing package

Klyubin, I added read count check. And still have missing data. I print out the buffer which I read to and I used wireshark to capture the out data. So, on the simulator output, I saw all the package, for example, 23 packages, each has 1kb. But the wireshark only captured 16,maybe. I also compared data detail, some in between package are lost, for example 15, 17, 21. (I Also compared package 14 and 16, both simulator print and captured data are same, just missed package 15) So, server still waiting for data and hanging there.

 

That is why I think the read overwrite the buffer before the write.

 

For debug info:

 

myOutputStream.write(mydatabuf, off, chunkLength);
String teststring = new String(mydatabuf);
System.out.println("debug:len:"+chunkLength+" data:"+teststring);

 

 

Please use plain text.
Developer
Posts: 1,474
Registered: ‎04-14-2009

Re: Lower IO write missing package

Your buffer is overwritten every time you invoke read(). This is what your code does: it asks the OS to read some data into the (beginning of) your buffer/array. So, there's no surprise that the buffer gets overwritten.

 

What I suspect (same as before) is that you see data corruption on your server only because you write method sends too much data after every read -- instead of sending exactly the number of bytes that have been read by the preceding read() operation, you are sending all the bytes in the buffer/array.

Please use plain text.
Developer
Posts: 1,474
Registered: ‎04-14-2009

Re: Lower IO write missing package

Sorry, I posted my previous message just about when you posted yours.

 

The code in your last post uses "off" as the offset from which to take "chunkLength" number of bytes. You should be using 0 as the offset instead. This is because your code reads data into offset 0. Also, you should create the String as new String(mydataBuf, 0, chunkLength) instead of simply new String(mydataBuf).

 

Have you tried the code I posted before?

Please use plain text.
New Developer
chimera
Posts: 51
Registered: ‎04-23-2009

Re: Lower IO write missing package

Yes and no. I want the overwrite happens after the write. But sometimes it didn't.

 

Yes, I tried your code, added check read retuen count. Still the same errors.

Please use plain text.