HTTP POST from a J2ME midlet

NOTE - this blog has now moved to codetrips.com
It is highly recommended that you read the original post there.

-------------

Although there are several examples and tutorials about how to connect to an HTTP server from a J2ME Midlet, I found that they all seem to concentrate a lot on the details of a GET and the POST is always assumed as an "exercise to the reader."

Now, although it is indeed rather straightforward to work out how to send POST requests to a server, I found (the hard way) that there are certain "minor details" that are best not forgotten.

I have thus decided to post here a brief example of how to send a request to a server, using a form-encoded body, so that either a JSP or a servlet can retriever the parameters, but one can benefit from the many advantages of POSTing, as oppposed to GETting.

The source code follows, I believe it is reasonably self-explanatory and comments should guide even the most novice reader through, but please do let me know if you feel that this is still too cryptic:

private void OpenConnection(String server) 
throws java.io.IOException {
// TODO: the actual "resource" part of the URL (/test in this case)
// should be either passed as a parameter or obtained at application
// level (maybe even set up at runtime).
// To test this code, I have written the simple Servlet below, however
// this would also work with a JSP page that echoes back the contents
// in text/plain format for the midlet to display.
//
  String url = "http://"+server+"/test"; 
  byte[] data = null;
  InputStream istrm = null;

  HttpConnection http = (HttpConnection)Connector.open(url);
  http.setRequestMethod(HttpConnection.POST);

  // This allows a JSP page to process the parameters correctly
  // This format (or content type) is the same as when forms are
  // submitted from a web page.
  http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

  // You may want to indicate to the server side (eg servlet) that 
  // this request is coming from a specific application.  This would
  // allow generation of appropriate format for the content.
  http.setRequestProperty("User-Agent", "HttpMidlet/0.2");

  // You can send any custom property as part of the HTTP header
  // This would matter for the "infrastructure" code, as opposed
  // to "body parameters" that would matter to the application
  // (eg, user=marco)
  http.setRequestProperty("Custom-Property", 
      "MyCustomProperty/1.0; AnotherProperty/debug_0.1");

  // You MUST create the body of the post BEFORE actually opening
  // the stream, otherwise you won't be able to set the length of
  // the request (see next step).
  // In this example, I am sending coordinates to a mapping server
  String msg = "x="+location.getX()+"&y="+location.getY();

  // THIS is important! without it a JSP won't process the POST data
  // it would also appear that CASE MATTERS (using "Content-Length" -note 
  // the capital 'L'- caused my servlet to return -1 from the 
  //    HttpServletRequest.getContentLenght() method
  http.setRequestProperty("Content-length", ""+msg.getBytes().length);

  // After this point, any call to http.setRequestProperty() will
  // cause an IOException
  OutputStream out = http.openOutputStream();
  out.write(msg.getBytes());
  out.flush();

  if (http.getResponseCode() == HttpConnection.HTTP_OK) {
    int len = (int)http.getLength();
    istrm = http.openInputStream();
    if (istrm == null) {
      log("Cannot open stream - aborting");
      throw new IOException("Cannot open HTTP InputStream, aborting");
    }
    if (len != -1) {
      data = new byte[len];
      int bytesRead = istrm.read(data);
      addProgressMsg("Read "+bytesRead+" bytes");
    } else {
    ByteArrayOutputStream bo = new ByteArrayOutputStream();
      int ch;
      int count = 0;

      // This is obviously not particularly efficient
      // You may want to use a byte[] buffer to read bytes in chunks
      while ((ch = istrm.read()) != -1) {
        bo.write(ch);
        count++;
      }
      data = bo.toByteArray();
      bo.close();
      addProgressMsg("Read "+count+" bytes");
    }
    response = new String(data);
    addProgressMsg("finished");
  } else {
    log("Response: "+http.getResponseCode()+", "+http.getResponseMessage());
    response = null;
    addProgressMsg("failed: "+http.getResponseMessage());
  }
  // This is critical, unless you close the HTTP connection, the application
  // will either be consuming needlessly resources or, even worse, sending
  // 'keep-alive' data, causing your user to foot unwanted bills!
  http.close();
}

To test this code, all that is required is a simple Serlvet that runs, for example, in an Apache Tomcat JSP container:

public class ConnectivityDiag extends HttpServlet {

  private static final String REVISION = "0.0.04";
  private static Logger log;

  public ConnectivityDiag() {
    if (log == null) {
      log = Logger.getLogger(getClass().getName());
      BasicConfigurator.configure();
    }
  }

  public void doGet(HttpServletRequest req, HttpServletResponse res)
      throws ServletException, IOException {
    doProcessRequest(req, res);
  }

  public void doPost(HttpServletRequest req, HttpServletResponse res)
      throws ServletException, IOException {
    doProcessRequest(req, res);
  }

  public void doProcessRequest(HttpServletRequest req, HttpServletResponse res) {
    BufferedReader reader = null;
    Writer writer = null;

    try {
      // Note this call seems to be case-sensitive
      // if the midlet does not set exactly "Content-length"
      // as the property, -1 will be returned
      int len = req.getContentLength();
      String contentType = req.getContentType();
      log.debug("Len: " + len);
      log.debug("Type: "+contentType);

      int bytesRead = 0;
      String remoteHost = req.getRemoteHost();
      String userAgent = req.getHeader("user-agent");
      log.info("Accessed at " + (new Date()) + " from " + remoteHost
          + ", using " + userAgent);

      // This simply echoes the parameters back to the client.
      // A JSP would use these in a  tag
      // and the ${param} macro.
      ArrayList strings = new ArrayList();
      Enumeration e = req.getParameterNames();
      while (e.hasMoreElements()) {
        String param = (String)e.nextElement();
        String value = (String)req.getParameter(param);
        strings.add(param+" = "+value);
      }
      res.setContentType("text/plain");

      // This is a custom property, that the remote client 
      // could query
      res.setHeader("Sent-at", new Date().toString());

      writer = res.getWriter();
      writer.write("Diagnostic servlet - Rev. "+REVISION);
      writer.write("\nBytes received: " + len);
      writer.write("\nFrom: " + remoteHost);
      writer.write("\nUsing: " + userAgent);
      writer.write("\n=== Parameters ===\n");
      for(String s:strings) {
        writer.write(s);
        writer.write('\n');
      }
      writer.write("=== Parameters end here ===\n");
      writer.flush();

    } catch (IOException e) {
      log.error(e.getMessage());
    } finally {
      try {
        if (reader != null)
          reader.close();
        if (writer != null)
          writer.close();
      } catch (IOException ex) {
        log.error(ex.getMessage());
      }
    }
  }
}

Microsoft Cabs

Apparently:

If you hail a black cab you might be getting your ride to work for free, courtesy of Bill Gates. OK, it’s not as simple as that - nothing ever is.

But if you happen to find a cab between 7.30 and 9.30 a.m. in London zone one or two with a Microsoft logo – you’re in the money. Apparently this is all done to help workers have a better day in the office as the Tube gets less and less bearable in summer.

(Rainier PR Briefs of April 27, 2007)

A thought occurred to me: wouldn't we all have a (much) better day at the office, if Microsoft were to put a tiny little more effort in releasing products with less bugs? and waste less of their (and our) time on pointless marketing initiatives?

And maybe a half-decent Operating System that does not crash on you for unknown reasons, that is secure, that does not require re-booting every time a ill-designed application crashes?

Ah, I hear you saying, but such an OS already exists: it's called Linux!

J2ME Multi-Threading and HTTP

Update: This is Part I of a two-part series about multi-threading in the J2ME environment, Part II can be found here.
I have uploaded the source code to BitBucket, you can easily download it using Mercurial (see here for an intro) by simply using:

https://marco@bitbucket.org/marco/j2me_threads

...

In my experience, one of the most difficult things to achieve in software (and even
more so in "limited" environments, such as J2ME) is to harness certain "long-running tasks" so that:

• they have enough time to complete their task, yet they do not
  hog the system so as to make it unresponsive to users (who would
  then have the impression the application is 'hung');
• it is easy to both check on whether they are still running or are, indeed, 'hung';
• it is relatively straightforward to set a timeout, expired which, they will be, rather uncerimoniously aborted;
• it is relatively straightforward to enable a 'Cancel' option for the user to press if they get bored before the timeout expires.

The hortodox answer to this has been (and quite rightly so) to have the long running task run in a separate thread from the main (GUI) thread, with a boolean variable to control execution.

Something like this:

public class MyLongTask implements Runnable {

  private volatile boolean stopped = false;
    public void stop() {
    stopped = true;
  }

   public void run() {
     boolean finished = false;
     // do some setup here
     // ...

     while (!stopped && !finished) {
       doAChunkOfWork();
       finished = amIDone();
     }
   }
}

or something similar to that.
The controlling thread (typically the one that runs the GUI too, which has probably a "Cancel" button hooked to a cancelTask() method) looks something like this:

public class Controller {
  private MyFrameView theView;
  private MyLongTask task = null;

  public Controller(MyFrameView theView) {
    this.theView = theView;
  }

  // various other stuff...

  public void doStartTimeConsumingTask() {
    task = new MyLongTask();
    new Thread(task).start();
  }

  public void doCancel() {
    if(task != null)
      if (! task.isFinished())
        task.stop();
  }

  // other Controller "stuff"

}

This is, in fact, what the good folks at Sun in the NetBeans team have essentially done, by defining the CancellableTask interface and a generic SimpleCancellableTask class (although, the latter, quite sadly has not implemented... the cancel() method!).

This was mostly done to supporting NetBeans' WaitScreen component and, even then, I found it not terribly compelling. (see Lukas Hasik's blog here).

Initially, when confronted with the problem, I simply extended SimpleCancellableTask, and used the stopped/stop() mechanism to enable the cancel() mechanism - that enabled my application to stop a long running task that had overstayed its welcome; however, at the price of having to throw a RuntimeException to communicate back to the WaitScreen that it was done, but had not finished its task.

So far, so good.

However, there is still no easy way to set a timeout task, nor we have implemented any mechanism to let MyLongTask tell Controller they're done (let alone let the latter communicate back to the the View, which, most likely, will display something to the user to tell her we're done).

The latter problem is easily solved with the Observer pattern (see my other post - to be published later): this can be implemented by either the view or the controller - my preference has always been for the Controller to act as the observer (again, I seem to be in broad agreement with most practitioners - those times when, for expediency, I have implemented that in the View, I have always come to regret it... Thank God for Refactoring!).

Setting a timeout and generally keeping tabs on the long-running task, in fact, requires that one uses an additional, controlling thread.
In other words, to properly manage a time consuming task, you need two additional threads - just one won't do.

I decided then that a more general solution was needed, one that would allow an easy, programmatic way of controlling a long running task and was, ideally, also "pre-wired" with an observer.

My approach uses a ControllerThread, a LongRunningTask interface and an Observer interface.

The code can be downloaded here, please note that this is written specifically for 'microedition' environments(J2ME), hence no generics or any other Java 5 niceties are used.

The LongRunningTask interface is reviewed below, the ControllerThread will be analysed in Part 2 of this article.

LongRunningTask is a relatively straightforward interface, that defines a couple of 'check-me' methods (isDone() and isStopped()) the obvious cancel() one and an optional progress() method, that returns a percentage value to completion (the actual semantic of the value returned by progress(), is worth noting, is entirely application-dependent, although I like to use it as a % value, to update, for example, a Gauge UI indicator).

The 'quirk' here is that LongRunningTask does not extend Runnable: the reason for this will be clearer when we'll come to look at ControllerThread (that does implement Runnable). All the work is done in execute() that throws a TaskException (unlike run() in Runnable) and can thus indicate abnormal termination to a 'wrapper' class.

To use the framework, one simply implements LongRunningTask for a class with a time-consuming method, and checks regularly on a boolean variable that may be set by cancel() (or any other sensible way to communicate to a running thread that it's time to quit it, enough is enough).

public class RecordTask implements LongRunningTask {

  private boolean stopped;
  private boolean done;
  private int progressIndicator;

  private byte[] audioData;

  /** Creates a new instance of RecordTask */
  public RecordTask() {
  }

  public boolean cancel() {
    stopped = true;
    return true;
  }

  public boolean isStopped() {
    return stopped;
  }

  public boolean isDone() {
    return done;
  }

  public int progress() {
    return progressIndicator;
  }

  /**
   * This is where all the time-consuming work is done:
   * here we record a speech segment, long at most maxDuration
   * or until the user presses a "Stop" command (that in turn will cause
   * the controller to call cancel()
   *
   * @param observer the observing class
   */
  public void execute(Observer observer) throws TaskException {
    observer.addMsg("Record started");
    try {
      done = false;
      stopped = false;
      Player player = Manager.createPlayer(getAudioMimetype());
      if (player != null) {
        player.realize();
        RecordControl rc = (RecordControl) player.getControl("RecordControl");
        if (rc != null) {
          observer.addMsg("RecordControl: OK");
        } else {
          observer.addMsg("No RecordControl, exiting");
          return;
        }

        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        rc.setRecordStream(bos);
        rc.startRecord();
        player.prefetch();
        player.start();

        long start = System.currentTimeMillis();
        while (!isStopped() && ((System.currentTimeMillis()-start) < getMaxDuration())) { 
          calcProgress(start);
          observer.announceProgress(progress());
        }
        rc.commit();
        player.close();
        audioData = bos.toByteArray();
        observer.announceProgress(100);
      }
      done = true;
    } catch (Exception ex) {
      observer.addMsg("Recorder exception: "+ex.getMessage());
      // The following is necessary to communicate back to ControllerThread
      // that an error condition was encountered
      throw new TaskException("Recorder exception: "+ex.getMessage());
    }
  }
  // Other utility methods follow....
}

Then, an instance of this class is passed, at construction, to a newly created instance of ControllerThread:

public class Multimedia implements Observer {

// Observer interface implementation omitted here
// ref. source code & Part II of this article

public void startRecording() {
  task = new RecordTask();
  thread = new ControllerThread(task, 1000L, 10000L, this);
  new Thread(thread).start();
}

// Other stuff goes here...
}

And that's pretty much about it: with the code above, we have created and started a new task which we'll check every second (1000L msec) and will allow at most 10 sec (10000L msec) to complete.
All the magic is done in ControllerThread - explained in part 2 of this series.