Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

Java Development

Reply
Regular Contributor
walkline
Posts: 82
Registered: ‎01-10-2013
My Device: Bold 9900
My Carrier: China Unicom
Accepted Solution

Upload large file problem

package com.blackberry.util.network;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.Hashtable;

import javax.microedition.io.Connector;
import javax.microedition.io.HttpConnection;
import javax.microedition.io.file.FileConnection;

import net.rim.device.api.io.http.HttpProtocolConstants;
import net.rim.device.api.io.transport.ConnectionDescriptor;
import net.rim.device.api.io.transport.ConnectionFactory;
import net.rim.device.api.io.transport.TransportInfo;
import net.rim.device.api.ui.UiApplication;

import com.blackberry.util.Function;
import com.blackberry.util.StringUtility;
import com.blackberry.util.log.Logger;

public class NetworkThread extends Thread
{
	private static final String twoHyphens = "--";
	private static final String Boundary = "****************256176b82bde4478"; //what_hell_is_that
	private static final String lineEnd = "\r\n";

	private ObserverInterface _ourObserver;
	private String _targetURL;
	private Hashtable _params;
	private String _fileField;
	private String _fileName;
	private String _fileType;
	private String _fileURI;
	private boolean _stopRequest = false;

	private ConnectionFactory cf;
	private Logger log;
	private int[] preferredTransportTypes = {TransportInfo.TRANSPORT_TCP_WIFI, TransportInfo.TRANSPORT_TCP_CELLULAR};
	private int[] disallowedTransportTypes = {TransportInfo.TRANSPORT_BIS_B, TransportInfo.TRANSPORT_MDS, TransportInfo.TRANSPORT_WAP, TransportInfo.TRANSPORT_WAP2};
	
	private	long postSize = 0;
	
	public NetworkThread(String requestURL, Hashtable params, String fileField, String fileName, String fileType, String fileURI, ObserverInterface observer)
	{
		super();

		log = Logger.getLogger(getClass());

		cf = new ConnectionFactory();
		cf.setPreferredTransportTypes(preferredTransportTypes);
		cf.setDisallowedTransportTypes(disallowedTransportTypes);
		cf.setTimeoutSupported(true);
		cf.setAttemptsLimit(10);
		cf.setConnectionTimeout(120000);

		_targetURL = requestURL;
		_params = params;
		_fileField = fileField;
		_fileName = fileName;
		_fileType = fileType;
		_fileURI = fileURI;
		_ourObserver = observer;

		postSize = getMultipartPostBytesSize(_fileField, _fileName, _fileType, _fileURI);
	}

	public void stop()
	{
		observerError(ObserverInterface.CANCELLED, "Cancelled by User");
		_stopRequest = true;

		Thread.currentThread().interrupt();
	}

	private void observerStatusUpdate(final int status, final String statusString)
	{
		if (!_stopRequest)
		{
			_ourObserver.processStatusUpdate(status, statusString);
		}
	}

	private void observerError(int errorCode, String errorMessage)
	{
		if (!_stopRequest)
		{
			_ourObserver.processError(errorCode, errorMessage);
		}
	}

	private void observerResponse(byte [] reply) 
	{
		if (!_stopRequest)
		{
			_ourObserver.processResponse(reply);
		}
	}

	public void run ()
	{
		HttpConnection httpConn = null;
		FileConnection fileConn = null;
		InputStream input = null;
		OutputStream output = null;
		StringBuffer buffer = new StringBuffer();
		StringBuffer responeBuffer = new StringBuffer();
		
		try {
			if ((_targetURL == null) || _targetURL.equalsIgnoreCase("") || (cf == null))
			{
				if (!_stopRequest)
				{
					_ourObserver.processError(ObserverInterface.ERROR, "Target url empty or http connection initial failed!");
				}
			}

			StringBuffer urlBuffer = new StringBuffer(_targetURL);

			if ((_params != null) && (_params.size() > 0)) {
				urlBuffer.append('?');
				Enumeration keysEnum = _params.keys();

				while (keysEnum.hasMoreElements())
				{
					String key = (String) keysEnum.nextElement();
					String val = (String) _params.get(key);
					urlBuffer.append(key).append('=').append(val);

					if (keysEnum.hasMoreElements()) {
						urlBuffer.append('&');
					}
				}
			}
			
			ConnectionDescriptor connd = cf.getConnection(urlBuffer.toString());
			String transportTypeName = TransportInfo.getTransportTypeName(connd.getTransportDescriptor().getTransportType());
			httpConn = (HttpConnection) connd.getConnection();

			if (httpConn != null)
			{
				try {
					httpConn.setRequestMethod(HttpConnection.POST);
					httpConn.setRequestProperty(HttpProtocolConstants.HEADER_CONNECTION, HttpProtocolConstants.HEADER_KEEP_ALIVE);
					httpConn.setRequestProperty(HttpProtocolConstants.HEADER_ACCEPT_CHARSET, "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
					//httpConn.setRequestProperty(HttpProtocolConstants.HEADER_CACHE_CONTROL,"no-cache, no-store, no-transform");
					httpConn.setRequestProperty(HttpProtocolConstants.HEADER_CONTENT_TYPE, HttpProtocolConstants.CONTENT_TYPE_MULTIPART_FORM_DATA + "; boundary=" + Boundary);
					httpConn.setRequestProperty(HttpProtocolConstants.HEADER_CONTENT_LENGTH, Long.toString(postSize));
					output = httpConn.openOutputStream();

					buffer.append(twoHyphens + Boundary + lineEnd);
					buffer.append("Content-Disposition: form-data; name=\"" + _fileField + "\"; filename=\"" + _fileName + "\"" + lineEnd);
					buffer.append("Content-Type: " + _fileType + lineEnd);
					buffer.append(lineEnd);
					output.write(buffer.toString().getBytes());	
					observerStatusUpdate(1, "Started");

					fileConn = (FileConnection)Connector.open(_fileURI, Connector.READ);
					long totalBytes = fileConn.fileSize();
					if (totalBytes == -1) {throw new IOException("File " + _fileURI + " not available.");}
					
					long sentBytes = 0;
					int percentPre = 0;
					
					input = fileConn.openInputStream();
					byte[] temp = new byte[1024];
					
					int len = 0;
					
					while ((len = input.read(temp)) > -1)
					{
						if (_stopRequest)
						{
							observerError(ProgressListener.CANCELLED, "User canceled.");
							return;
						}
						
						output.write(temp, 0, len);	
						
						//Thread.yield();

						sentBytes += len;
						int percentageFinished = (int) ((sentBytes * 100) / totalBytes);
						percentageFinished = Math.min(percentageFinished, 99); 

						if (percentageFinished != percentPre)
						{
							observerStatusUpdate(percentageFinished, StringUtility.formatSize(sentBytes, 1) + " / " + StringUtility.formatSize(totalBytes, 1));	
						}

						percentPre = percentageFinished;
					}
					
					output.write(lineEnd.getBytes());
					output.write((twoHyphens+Boundary+twoHyphens+lineEnd).getBytes());
					
					output.flush();
					output.close();
				} catch (IOException e)
				{
					observerError(ProgressListener.ERROR, "Post data exception: \n\n" + e.getMessage());
				}

				log.info("HTTP-POST-MULTI (" + transportTypeName + "): " + httpConn.getURL());

				int resCode = 0;
				String resMessage = "";

				try {
					resCode = httpConn.getResponseCode();
					resMessage = httpConn.getResponseMessage();
					
					log.info("HTTP-POST-MULTI Response: " + resCode + " " + resMessage);
				} catch (IOException e) {
					observerError(ProgressListener.ERROR, "get respone code ioexception: \n\n" + e.getMessage());
				}
				
				switch (resCode)
				{
					case HttpConnection.HTTP_OK:
					{
						InputStream inputStream;
						int c;
						
						try {
							inputStream = httpConn.openInputStream();
							while ((c = inputStream.read()) != -1)
							{
								responeBuffer.append((char) c);
							}

							inputStream.close();
						} catch (IOException e)
						{
							Function.errorDialog("HTTP_OK ioexception: " + e.toString());
						}

						observerStatusUpdate(100, "File uploaded.");

						UiApplication.getUiApplication().invokeAndWait(new Runnable()
						{
							public void run()
							{
								try {
									Thread.sleep(1000);
								} catch (InterruptedException e) {}		
							}
						});

						observerResponse(responeBuffer.toString().getBytes());
						break;
					}
					case HttpConnection.HTTP_BAD_REQUEST:
					{
						InputStream inputStream;
						int c;

						try {
							inputStream = httpConn.openInputStream();
							while ((c = inputStream.read()) != -1)
							{
								responeBuffer.append((char) c);
							}

							inputStream.close();
						} catch (Exception e)
						{
							Function.errorDialog("HTTP_BAD_REQUEST ioexception: " + e.toString());
							observerError(ProgressListener.ERROR, e.getMessage());
						}
						
						observerError(ProgressListener.ERROR, "File transfer problems!");

						break;
					}
					case HttpConnection.HTTP_TEMP_REDIRECT:
					case HttpConnection.HTTP_MOVED_TEMP:
					case HttpConnection.HTTP_MOVED_PERM:
					{
						observerError(ProgressListener.ERROR, "File transfer moved!");
						break;
					}
					case HttpConnection.HTTP_INTERNAL_ERROR:
					{
						observerError(ProgressListener.ERROR, "Internal server error");
						break;
					}
					default:
						break;
				}
			}
				
			log.info("HTTP-POST-MULTI Body: " + httpConn.getType() + "(" + responeBuffer.length() + ")");
			log.debug(responeBuffer.toString());
		} catch (Throwable t)
		{
			Function.errorDialog(t.toString());
			log.error("New Thread Throwable: " + t.getMessage());
			t.printStackTrace();
		} finally {
			if (input != null) {try {input.close();} catch (IOException e) {}}
			if (fileConn != null) {try {fileConn.close();} catch (IOException e) {}}
			if (output != null) {try {output.close();} catch (IOException e) {}}
			if (httpConn != null) {try {httpConn.close();} catch (IOException e) {}}
		}
		
		_stopRequest = true;
		_ourObserver = null;

		observerStatusUpdate(100, "Finished"); // Tell Observer we have finished
		observerResponse("Succeeded".getBytes());
	}

	private long getMultipartPostBytesSize(String name, String fileName, String fileType, String fileURI)
	{
		StringBuffer buffer = new StringBuffer();
		FileConnection fconn = null;
		long fileSize = 0;

		/*
		 * @multipart post format
		 *	--****************256176b82bde4478\r\n
		 *	Content-Disposition: form-data; name="uploadfile"; filename="fileName"\r\n
		 *	Content-Type: txt/plain\r\n
		 *	\r\n
		 *	[content bytes of upload file]
		 *	\r\n
		 *	--****************256176b82bde4478--\r\n
		*/
		buffer.append(twoHyphens + Boundary + lineEnd);
		buffer.append("Content-Disposition: form-data; name=\"" + name + "\"; filename=\"" + fileName + "\"" + lineEnd);
		buffer.append("Content-Type: " + fileType + lineEnd);
		buffer.append(lineEnd);
		
		ByteArrayOutputStream baos = new ByteArrayOutputStream();
		
		try
		{
			baos.write(buffer.toString().getBytes());
			baos.write(lineEnd.getBytes());
			baos.write((twoHyphens+Boundary+twoHyphens+lineEnd).getBytes());

		} catch (IOException e)	{
			Function.errorDialog("flush byte data ioexception: " + e.toString());
		}

		try {
			fconn = (FileConnection)Connector.open(fileURI);
			fileSize = fconn.fileSize();
			fconn.close();
		} catch (IOException e) {
		}

		return baos.toByteArray().length + fileSize;
	}
}

 

I'm using multipart post method via httpConnection to upload file to server (a net disk service website, single file size is limited to 500MB), I tested on simulator (9800 Asia, software: 6.0.0.706, platform: 3.0.0.159, network: wifi) and successfully uploaded a large file more than 40MB, but when I signed my cod file and tested on device (9900, software: 7.1.0.1098, platform: 5.1.0.701, network: wifi), I got a interrupt during uploading almost 3MB data and thrown a ConncectionClosedException.

 

Here is my guess, when output.write() filled with any amount bytes (maybe 10240 bytes), it will upload buffered bytes to server immediately, and waiting for filled with remaining bytes and so on. Uploading the buffered bytes may takes a long time to wait, whether the ConnectionClosedException is occurred during this freeze period?

 

If I upload a file less than 1MB on device, it probably will be successful, yes, not 100% success rate, I don't know what problem I'm facing now:Dots:

Please use plain text.
Regular Contributor
walkline
Posts: 82
Registered: ‎01-10-2013
My Device: Bold 9900
My Carrier: China Unicom

Re: Upload large file problem

Is it a really big problem for everyone? or restricted topic?
Please use plain text.
Developer
peter_strange
Posts: 19,601
Registered: ‎07-14-2008
My Device: Not Specified

Re: Upload large file problem

My understanding is that the BlackBerry Http processing will not attempt to send the data to the server until it is clear the driving program requires it - which is usually when the processing requests the response code.  So my understanding is that all of your file will be sent at once, when your code reaches:

resCode = httpConn.getResponseCode();

 

I am not sure why your 9900 failed at the point it did and your Simulator code succeeded.  I would have thought they would both fail.  But it might depend on much memory each has.  If your Simulator just contains your test program, and the 9900 is full of other applications, perhaps there was not sufficient spare memory in the 9900.  It would be interesting to not if there was a detail message with your ConnectionClosedException.  Can you try this again and print it out (<exception>.toString will do it). 

 

But that said, I think sending 40 MB to a server in one go, even over WiFi, is probably not a good idea.  I would suggest breaking up your file into smaller chunks and reassembling them at your Server end.  This will mean if you do get an exception, you can restart at the point of failure (assuming you put the appropriate logic in). 

Please use plain text.
Regular Contributor
walkline
Posts: 82
Registered: ‎01-10-2013
My Device: Bold 9900
My Carrier: China Unicom

Re: Upload large file problem

Thanks, peter, thanks for your reply.

 

About your suggestion, I'm sorry, I have been said, I was used a net disk service, just like dropbox and so on, so I don't have any limits of authority to process data pieces on the server.

 

About the detail of ConnectionClosedException, I remember that eclipse thrown a message to me, and it was useless, I will try it again now.

 

And, you said that processing will not attempt to send the data to server until it is clear, but in fact, when httpconnection write amount bytes data to its buffer, the network indicator in the top right of the device screen will be flash many times, until flash stopped, the write bytes progress will be countinued.

Please use plain text.
Developer
peter_strange
Posts: 19,601
Registered: ‎07-14-2008
My Device: Not Specified

Re: Upload large file problem

[ Edited ]

"I don't have any limits of authority to process data pieces on the server"

 

You might have problems if you do this over mobile service because they may or may not cut you off if you try to upload too many bytes,  Over WiFi it should work fine   Chunking is really useful for recovery restart only, doesn't provide any additional functionality. 

 

"the network indicator in the top right of the device screen will be flash many times, until flash stopped, the write bytes progress will be continued."

 

OK, not my experience with mobile connections, perhaps it works this way with WiFi or using https which does establish the connection initially. 

 

Given this, I don't think I can help much here sorry. 

Please use plain text.
Regular Contributor
walkline
Posts: 82
Registered: ‎01-10-2013
My Device: Bold 9900
My Carrier: China Unicom

Re: Upload large file problem

peter, I have to apologize to you.

 

Just contacted with a programmer of net disk service, he told me a secret, the net disk service have a multi part upload file api used to split a large file to any pieces, and then upload them to server, finally sent a merge command to complex them to one file.

 

thanks again, peter!~

Please use plain text.
Developer
peter_strange
Posts: 19,601
Registered: ‎07-14-2008
My Device: Not Specified

Re: Upload large file problem

No Apology needed.  Glad you have this sorted!

Please use plain text.