Java Development

[kemdada]

I don't know if anyone has seen this, but I'm using a browserfield and handling cookies with a custom cookie handler.

It seems that when I try to getHeaderField("Set-Cookie")  for a server page that sets multiple cookies, it only gets the last cookie set.

Based on http://www.w3.org/Protocols/rfc2109/rfc2109

  An origin server may include multiple Set-Cookie headers in a
   response. 

A sample server page that sends multiple cookies is https://mobile.paypal.com.

This also seems to be a bug that happened in java 1.4 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4670281

Any ideas?

[virmundi]

I'm currently experiencing the same problem. Is there a way to sub class the HTTP connector? I assume the samething fails over https.

[virmundi]

Well, since we're in the same boat: how about using raw sockets and implement the HTTP connector directly? It's time consuming, but really the only option that I can see.

[cspinelive]

how about a simple loop to run through every header variable until you start getting nulls.

you could collect all the ones called 'set-cookie' and use them as needed?

this is close to what I'm suggesting, but not exact.

for (int i=0; i<100; i++){
    if (connection.getHeaderFieldKey(i) == null)
        break;

        System.out.println(connection.getHeaderFieldKey(i) + " - " + connection.getHeaderField(i) + "]");

}

I'm about to write a parsing method using this approach that takes the connection object and returns an array of custom cookie objects so please speak up if you see a problem with that approach.

here's my psuedocode

for each header in connection                  // to handle multiple "set-cookie" instances
    if header = 'set-cookie'
        cookies() = header.split(,)                 // to handle multiple cookies in a single "set-cookie" instance
        for each cookie in cookies
            cTmp = cookie.substr(0, location of first ";")  // get the first key/value pair of the cookie.  it should contain the name/value
            cookie.name = cTmp.substr(0, loction of "=")
            cookie.value = cTmp.substr(location of "=", cTmp.length)
        next
    end if
next

[johnbibs]

Hi,

The problem here is that the MDS is doing some transcoding on the http headers. So for example a website returns 5 cookies in the headers. What happens is sometimes it returns the 5 cookies in a single "Set-cookie" header with comma as a delimeter or it sometimes returns a 5 "set-cookies" header but with the same cookie name and value. you can try bypassing MDS by requesting via direct tcp.

cheers!

[cspinelive]

From what I can tell both of these scenarios are valid and can be expected from any site whether or not you're using MDS (I'm not).

Multiple “Set-Cookie” lines

Set-Cookie: atgRecSessionId=2152; Path=/

Set-Cookie: atgRecVisitorId=1462jxz3hLlPF_0Q8YFOlFA2L2ZQFZ2I626UnS4DApsFBrQD8C7; Expires=Tue, 22-Jun-2077 02:02:13 GMT; Path=/

Single “Set-Cookie” line using a comma to separate the cookies.

Set-Cookie: atgRecSessionId=2152; Path=/, atgRecVisitorId=1462jxz3hLlPF_0Q8YFOlFA2L2ZQFZ2I626UnS4DApsFBrQD8C7; Expires=Tue, 22-Jun-2077 02:02:13 GMT; Path=/

Check out the source for java.net.HttpCookie.parse() .  I know it won't work in J2ME, but it just goes to show the many different scenarios that need to be accounted for (2 specs and the Netscape implementation).

The solution I proposed has a flaw that I know of.  The 'expires' date contains a comma.  This is a problem because when multiple cookies are sent on one line, their delimiter is also a comma.

Here's what I'm working on now.  It's still untested.

public static Cookie[] getFromHttpResponse(HttpConnection conn){
	// look through all the response headers and send back an array of cookies.
	// handle the scenario where cookies are sent on multiple 'set-cookie' lines
	// also handle the scenario where multiple cookies are sent on a single 'set-cookie' line 
	
/*
	for each header in connection
		if header = 'set-cookie'
			cookies() = header.split(')
			for each cookie in cookies
				cTmp = cookie.substr(0, location of first ";")
				cookie.name = cTmp.substr(0, loction of "=")
				cookie.value = cTmp.substr(location of "=", cTmp.length)
			next
		end if
	next
*/	
	
	Cookie[] returnData = new Cookie[0];

	for (int i=0; ;i++)
	{
		//Check to see if we've found the end of the header list
		String header = null;
		try {
			header = conn.getHeaderFieldKey(i);
		} catch (IOException e1) {}
		
		if (StringUtils.IsVoid(header)){
			break;
		}
		
		
		if (header.equalsIgnoreCase("set-cookie")){
			try {
				header = conn.getHeaderField(i);

				String[] cookies = StringUtils.split(header, ',');  // in case multiple cookies are on this line
				
				for (int j=0; i<cookies.length; j++){
					String cookieString = cookies[j];
					String cTmp = cookieString.substring(0, cookieString.indexOf(';'));
					
					String name = cTmp.substring(0, cTmp.indexOf('='));
					String value = cTmp.substring(name.length() + 1, cTmp.length());
					
					Arrays.add(returnData, new Cookie(name, value));
				}
			} catch (IOException e) {}
		}
	}
	
	return returnData;
}

[johnbibs]

hi cspinelive,

the problem i encountered before is that sometimes, the header contains 5 "set-cookes" but with the same key and value. So even if you parse it correctly, still you don't have the right set of cookies that is needed by the website you are accessing. This case is true when my browser tries to login-in in yahoo mail. I'm not sure its useless to implement the 3 specs when you are getting incomplete information on your header. though i only implemented the netscape specs. 

cheers!

[cspinelive]

Here's what I've settled on for now.  It's derrived from the source of net.java.HttpCookie.parse()

It only returns a string containing name value pairs delimited using semicolons

ex: cookie1=foo;cookie2=bar;

It shouldn't be too hard to make it return a proper cookie object complete with all the attributes.

import java.io.IOException;
import javax.microedition.io.HttpConnection;
import net.rim.device.api.util.Arrays;

public final class Cookies {
	private static int VERSION_NETSCAPE = 0;
	private static int VERSION_RFC_2965_2109 = 1; 
	
    /*
     * Retrieve cookie values from a HTTP request.
     * Derived from net.java.HttpCookie.parse()
     *
     * @param header    conn HttpConnection; 
     *
     * @return          string containing name value pairs delimited using semicolons  ex" cookie1=foo;cookie2=bar; 
     *
     * @throws 
     */
	public static String getFromHttpResponse(HttpConnection conn){
		// look through all the response headers and send back a string suitable for returning with a HTTP Request.
		// ignores all cookie attrubutes such as expires, domain, path, etc.  just sends back the name and value 

		// see java.net.HttpCookie.parse() if you want to know all the scenarios required to do this correctly.
		
		String returnData = "";

		for (int i=0; ;i++)
		{
			String header;
			int version;
			
			try {
				header = conn.getHeaderFieldKey(i);
			
				if (StringUtils.IsVoid(header) || i > 1000) //bail out once we find the end of the headers or if we loop too many times.
					break;
			
				if (header.equalsIgnoreCase("set-cookie")){
					header = conn.getHeaderField(i);
					version = guessCookieVersion(header);
					
					String[] cookies = new String[0];
					if (version == Cookies.VERSION_NETSCAPE){
						// only one cookie per 'set-cookie' header
						Arrays.add(cookies, header);
					}else{
						// multiple cookies per 'set-cookie' header delimited using commas
						cookies = splitMultiCookies(header); 
					}
						
					for (int j=0; j<cookies.length; j++){
						returnData = returnData + parseInternal(cookies[j]);
					}
				}
			} catch (IOException e) {}
		}
		return returnData;
	}

	/*
     * try to guess the cookie version through set-cookie header string
     */
    private static int guessCookieVersion(String header) {
        int version = 0;
        
        header = header.toLowerCase();
        if (header.indexOf("expires=") != -1) {
            // only netscape cookie using 'expires'
            version = 0;
        } else if (header.indexOf("version=") != -1) {
            // version is mandatory for rfc 2965/2109 cookie
            version = 1;
        } else if (header.indexOf("max-age") != -1) {
            // rfc 2965/2109 use 'max-age'
            version = 1;
        }
        
        return version;
    }

    /*
     * Split cookie header string according to rfc 2965:
     *   1) split where it is a comma;
     *   2) but not the comma surrounding by double-quotes, which is the comma
     *      inside port list or embeded URIs.
     *
     * @param header            the cookie header string to split
     *
     * @return                  list of strings; never null
     *
     */
    private static String[] splitMultiCookies(String header) {
        String[] cookies = new String[0];
        int quoteCount = 0;
        int p, q;
        
        for (p = 0, q = 0; p < header.length(); p++) {
            char c = header.charAt(p);
            if (c == '"') quoteCount++;
            if (c == ',' && (quoteCount % 2 == 0)) {      // it is comma and not surrounding by double-quotes
                Arrays.add(cookies, header.substring(q, p));
                q = p + 1;
            }
        }
        
        Arrays.add(cookies, header.substring(q));
        
        return cookies;
    }
    /*
     * Parse header string to cookie object.
     *
     * @param header    header string; should contain only one NAME=VALUE pair
     *
     * @return          an HttpCookie being extracted
     *
     * @throws IllegalArgumentException if header string violates the cookie
     *                                  specification
     */
    private static String parseInternal(String header)
    {
        String cookie = null;
        String namevaluePair = null;
        
        String[] tokenizer = StringUtils.split(header, ';');
        
        // there should always have at least on name-value pair;
        // it's cookie's name
        if (tokenizer.length > 0){
            namevaluePair = tokenizer[0];
            int index = namevaluePair.indexOf('=');
            if (index != -1) {
                String name = namevaluePair.substring(0, index).trim();
                String value = namevaluePair.substring(index + 1).trim();
                cookie = name + "=" + value + ";";
            } else {
                // no "=" in name-value pair; it's an error
                throw new IllegalArgumentException("Invalid cookie name-value pair");
            }
        }else{
        	throw new IllegalArgumentException("Empty cookie header string");
        }
        
        /*
         * All I needed were the name/value pairs.
         * All attributes are being ignored.
         * If you need attributes, you'll need to finish making this code J2ME compliant.
        
        // remaining name-value pairs are cookie's attributes
        while (tokenizer.hasMoreTokens()) {
            namevaluePair = tokenizer.nextToken();
            int index = namevaluePair.indexOf('=');
            String name, value;
            if (index != -1) {
                name = namevaluePair.substring(0, index).trim();
                value = namevaluePair.substring(index + 1).trim();
            } else {
                name = namevaluePair.trim();
                value = null;
            }
            
            // assign attribute to cookie
            assignAttribute(cookie, name, value);
        }
        */
        	
        return cookie;
    }    
}

Here's the stringutils

public class StringUtils {
	
	public static boolean IsVoid(String s) {
		return s == null || s.trim().length() == 0;
	}

	public final static String[] split( String str, char separatorChar ) {
	      if ( str == null ) {
	         return null;
	      }
	      int len = str.length();
	      if ( len == 0 ) {
	         return null;
	      }
	      
	      Vector list = new Vector();
	      int i = 0;
	      int start = 0;
	      boolean match = false;
	      while ( i < len ) {
	         if ( str.charAt( i ) == separatorChar ) {
	            if ( match ) {
	               list.addElement( str.substring( start, i ).trim() );
	               match = false;
	            }
	            start = ++i;
	            continue;
	         }
	         match = true;
	         i++;
	      }
	      if ( match ) {
	         list.addElement( str.substring( start, i ).trim() );
	      }
	      String[]  arr    = new String[list.size()];
	      list.copyInto( arr );
	      return arr;
	   }

}

[marchywka]

many header handlers are moving to hashtables so you will need to accomodate

the valid transofmrations that put all same-key headers into a single entry. We bothered RIM about this for quite a while before discovering that we didn't handle this as it seems most sites don't bother. Indeed, the BIS/MDS transofmration reduces headers size somewhat and makes things much easier.

Also, google and webkit have some discussion on cookie code. I had to reverse engineer from things like curl and wget.