Wednesday, September 16, 2009

Programming sockets using temporary ports

In an earlier posting, I showed where there was code in Apache Harmony that had some unsafe parameter checking logic, and I gave a pattern for how to do it right.

Another "anti-pattern" that I see recurring in the Harmony test cases is around client-server socket programming. The typical scenario is that the tester wants to exercise some socket code, so they spin up a new Thread to act as the server, and run the tests on the main thread. The tester either picks a port that they assume will be free, or use some horrible code that tries to find a free port:


/*
* Returns a different port number every 6 seconds or so. The port number
* should be about += 100 at each 6 second interval
*/
private static int somewhatRandomPort() {
Calendar c = Calendar.getInstance(TimeZone.getTimeZone("UTC"));
int minutes = c.get(Calendar.MINUTE);
int seconds = c.get(Calendar.SECOND);

return 6000 + (1000 * minutes) + ((seconds / 6) * 100);
}

Guessing port numbers like this is simply awful and hopeless. Once the port has been established, the poorly written test case opens a server socket to accept connections in the server Thread, while to get the timing right the client goes into a sleep for a while to give the server time to start up. Apart from the fact that the code is going to fail intermittently, it means the test case always takes a minimum length of time to execute as it spends time sleeping.

There is no need for guessing ports or separate threads. In Java, just like other programming languages that expose the underlying platform's TCP/IP stack behaviour, binding to port zero instructs the stack to allocate an ephemeral port.

Furthermore, server sockets have a listen backlog queue capable of remembering 50 outstanding connect requests by default. Here is a example of simple client-server code using an ephemeral port and a single thread to exchange a simple message over TCP/IP sockets.

// Set-up
ServerSocket server = new ServerSocket(0);

Socket client = new Socket();
client.connect(server.getLocalSocketAddress());

Socket worker = server.accept();

// Do some stuff
client.getOutputStream().write("Hello world!".getBytes("UTF-8"));
byte[] buffer = new byte[1024];
int length = worker.getInputStream().read(buffer);

// Tidy-up
client.close();
worker.close();
server.close();

System.out.println(new String(buffer, 0, length, "UTF-8"));

Hopefully the code is simple enough to understand without further comment. I'll just point out that the server socket is created with an argument of "0" to mean the listening socket should be opened on any network adapter, with a stack allocated port number, and supporting up to fifty pending connections. Then the client connects to the actual interface and port that was used using getLocalSocketAddress().

The same simple example can also be written using the NIO APIs, which requires passing null to the bind() method, like this:

// Set-up
ServerSocketChannel server = ServerSocketChannel.open();
server.socket().bind(null);

SocketChannel client = SocketChannel.open();
client.connect(server.socket().getLocalSocketAddress());

SocketChannel worker = server.accept();

// Do some stuff
client.write(ByteBuffer.wrap("Hello world!".getBytes("UTF-8")));

ByteBuffer readBuffer = ByteBuffer.allocate(1024);
worker.read(readBuffer);
readBuffer.flip();

// Tidy-up
worker.close();
client.close();
server.close();

System.out.println(Charset.forName("UTF-8").decode(readBuffer));

Of course, the simple examples still ignore a few return codes and exceptions that should be considered to make the code safer, but the purpose here is to show the structure for tests and applications that need to use sockets to exchange information.

Monday, September 07, 2009

Shoddy reporting from El Reg :-(

I stop by The Register website quite frequently to keep up on the gossip, and quite enjoy their laid back reporting style, but the September 4th article titled "Oracle should relax Sun's Java Community control grip" was a very poor piece of reporting.

It hit my radar with the reference to Apache Harmony, where Clarke writes:

For all its evangelism - and its initial decision to open source Java - Sun has refused to open the TCKs, infuriating and frustrating the open-source community.

This has led to accusations that Sun is hindering - not helping - open-source Java projects such as Harmony from the Apache Software Foundation (ASF), backed strongly by IBM.

While Apache's has been able to build an implementation of Java Standard Edition under Project Harmony thanks to the opening of Java, Harmony cannot be certified because the TCKs contain proprietary code the open-source code cannot touch. Harmony, therefore, remains stuck in a limbo of having been built but being uncertified.
Unfortunately this just serves to illustrate that Clarke doesn't understand the situation, or history, around open source Java SE.

Apache have never asked, or expected, Sun to open source their TCKs. While that would be a fine outcome in itself, there is no reason why Sun cannot maintain their tests as proprietary code and make them available under a variety of license terms as they do with other Java specification test suites. Apache have asked for license terms that allow the code we have written to be released under an open source license, that is to say Apache will not entertain restrictions placed on our code that passes the TCK.

http://www.apache.org/jcp/sunopenletter.html

Furthermore, Apache Harmony and GNU Classpath existed well before OpenJDK, so it can hardly be said that OpenJDK was a prerequisite to alternative implementations being created!

The story attempts to cover a great deal of ground, the possible effects of the acquisition, the Apache dispute, JCP reform, and so on -- each of these is a major story in itself and this shoddy attempt does each a disservice.

Improving parameter checking in Apache Harmony

I'm going through some Apache Harmony code at the moment, and spotting a few places where there is a common pattern of code that needs fixing.

Consider this method:


public synchronized int read(byte[] target, int offset, int length) throws IOException {
if (length + offset > target.length || length < 0 || offset < 0) {
throw new ArrayIndexOutOfBoundsException();
}
if (0 == length) {
return 0;
}
ByteBuffer buffer = ByteBuffer.wrap(target, offset, length);
return channel.read(buffer);
}

Can you see the problem?
Hint: the parameter checking in the if() statement is wrong.

The original author rightly wants to ensure that the offset and length represent plausible values for the data to be put into the buffer; that is, the method should not attempt to place data outside the bounds of the 'target' byte array.

The problem is that in Java, like a number of other languages, (int) values overflow without any exception being thrown. In the code above we are checking for illegal values by computing (length + offset) and throwing an exception if it is greater than the buffer's length. This ensures we don't "write off the end" of the target buffer. However, if the sum overflows an int representation then it may indeed pass this safety check, and permit a value for offset or length that is too large.

As a practical illustration, consider that the 'offset' actual parameter value is Integer.MAX_VALUE = (int)2147483647, and the 'length' value is, say 42. In this case the sum of these positive values is a negative value '(int)(2147483647 + 42) = (int)-2147483607' so the less-than test passes and the method is broken!

The fix is to avoid the sum, and convert it into a subtraction, e.g.
if (length < 0 || offset < 0 || length > target.length - offset) {
throw new ArrayIndexOutOfBoundsException();
}
Now we can be assured that there will be no int overflow. The target.length cannot be less than zero, so the subtraction can only produce values in the range (0 - Integer.MAX_VALUE = Integer.MIN_VALUE + 1) to (Integer.MAX_VALUE - 0 = Integer.MAX_VALUE).

I know this is not news for Java developers. Josh Bloch and others have been highlighting the problem for a number of years, yet I've found about four places in the code so far that exhibit this potential problem. They'll all be fixed in time for the next milestone build.

Wednesday, July 08, 2009

Queen introduction

I got two new queen bees in the post a short time ago. They come in normal padded envelopes via Royal Mail. The postman was wondering why the envelope was buzzing!

Within the envelope each queen and her attendants is protected in a 'puzzle cage'. The cage has some candy to to keep the queen alive, and since the queen never feeds herself there are worker bee attendants to feed her and keep things calm during travel. If you look closely you can see the queen bee on the right, she has a green dot on her back showing she is the latest 2009 model.

To introduce a queen you can either use the puzzle cage they came in which has knock-out tabs enabling the bees to walk out one the candy is consumed, but I used a Butler cage. The Butler cage is a simple wire cage permanently blocked at one end, and which you seal at the other end using newspaper and elastic bands.

The knack is to take the queen out (let the attendants fly away) and put her into a separate cage. I did it on a sheet near a window since the queen may well fly off too, and at £20 each you don't want to loose her! The attendants will sting, but the queen doesn't sting you so you can pick her up by her thorax and pop her in.

After carefully putting the queen into the Butler cage and sealing it up it is ready to go into the hive. Opinions vary, but I take out the old queen about two days before introducing the new queen. The period of queenlessness makes the workers more inclined to accept the new smell. I kept the old queen in a jar with some attendants and candy in the case the introduction failed and I needed to back out of the procedure.

As you can see from the photo I have put a short nail in the cage's block so that I can push it into the comb. It is put in position so that the newspaper end of the cage is centred on the middle frame, right in a patch of brood where the workers would expect to find the queen.

The receiving colony's workers can feed and lick the queen through the cage, but if they try to attack they cannot ball her in there, and she can retreat into the newspaper end if they try to sting her. Over time the new queen pheromone permeates the colony and the workers will be ready to accept her.

The queen is eventually released by workers eating through the newspaper, and I gave two pin pricks on the end for them to get hold of with their mandibles.

Once the cage is in place another recommendation of Professor Ratnieks is to heavily smoke the colony. Not sure if that is a distraction technique or simply another way to mask the new incoming smell. I put in loads of smoke so that the bees were fanning furiously.

Then leave the colony alone for at least two days. After that time I checked to ensure the queen had been released, and removed the cage. The new queen was wandering over the comb without a care in the world, probably quite oblivious of her travels through the postal system a few days earlier.

I left them alone for another week, then looked again and there were eggs a plenty, so she was working away happily. Some people report as low as 50% success rate at introducing new queens. I only did these two, one into a colony that had been queenless for a while and the other where I removed an existing queen. Both worked for me so I'm happy with that.

Organised volunteerism

As part of a longer discussion on a private mailing list[1] Sam Ruby wrote a succinct description of the Apache volunteerism ethos. He wrote:

"The prevailing attitude within Apache is that releases will be done when they are ready, and that such releases will contain only the functions for which there exists volunteers who have an interest in doing the work. At times, there are people who would prefer a more predictable schedule and specific function. There are organizations which provide such assurances. This isn't one of them."
This resonated with me as a participant in the Apache Harmony project which is undertaking a full implementation of the Java SE specification. There are some modules that attract lots of attention in completeness and performance (such as the core LUNI, beans, security, and so on) and others that don't attract so much effort (such as Swing, print, and RMI).

I'm totally fine with that situation since it represents the technological equivalent of the adaptive market hypothesis of financial markets. In our world, people will tend and care for code that is important to them, and the other code by definition is not so important to people.

I've been a contributor to numerous open source foundations and working groups with different styles of working, and there is no "one true way" -- having a variety of organizations with different styles of working ensures that there is going to be a place for a wide variety of people to be comfortable innovating in the advancement of Java technology.

I'm comfortable with those who want to make some code open, and keep other code to themselves. I object to those who claim there is only one way to behave, and try to enforce that on others through restrictive licensing or organizational rules.

[1] Sam kindly gave me permission to take that quote and make it public.

Monday, June 22, 2009

WebSphere Application Server for Developers

IBM's WebSphere Application Server (WAS) is the company's flagship Java EE 5 product used in production environments worldwide.

As you would expect for a product representing over 10 years investment from Big Blue, it comes with a license fee to match. However, if you are willing to forgo formal support, and can live with the developer license offered, you can now download the full application server at no cost!

The "WebSphere Application Server for Developers" package for Windows or Linux is available as a download directly from IBM. Bear in mind that WAS weighs in at 788 MB, so downloading over a 1.5Mbps connection is likely to take about one and a half hours.

Go and grab it, and try your software out on a runtime that matches the production environment. In place of formal support there are WAS developers answering questions on the open newsgroup.

Monday, June 15, 2009

JavaOne 2009


The attendance figures put around for JavaOne 2009 state that there were "around 15,000" people at the Moscone Center -- hmm, that seems generous and it certainly felt a bit quieter than previous years. Not altogether unexpected with the economy impacting on travel, maybe people avoiding large congregations over concern of flu, and so on.

The main hall was arranged slightly differently, with the stage set up at the 'side' of the hall, with large back drapes hung across each end. Ushers did a good job of corralling people to the front, and there was a noticeably large representation from Sun themselves.

Like last year there was much talk of Sun's proprietary technology called "JavaFX", with demos of JavaFX running on a disparate variety of hardware (set top box, TV, phone, desktop). There was discussion of Java performance, as realized by the contributions made from new machine advances from Intel. Of course, it wasn't called out that the Nahelem numbers were published on IBM's Java implementation! There was a demo of the new Java market place where Sun, like others, are aiming to reproduce the success of an iTunes style store front to end users.

There was not much talk about Java SE 7 (Jonathan Schwartz did mistakenly announce it's availability, but was corrected by his Chief Engineer on stage the next day), and not much talk about OpenJDK or the JCP which were prominent topics of calls for community participation in previous years. NetBeans was also notably absent from the opening session.

The opening session was concluded by a moving 'good-bye' speech from Scott McNealy and a symbolic hand-over to Larry Ellison. During the hand-over Larry gave a challenge to the OpenOffice team to rewrite in JavaFX. He also said that Sun could be expected to produce notebooks running a Java-based OS in competition with Android.

Later in the week I gave a demo of the Apache Harmony runtime during the IBM general session. I showed Harmony running Eclipse and developing the NIO module using the Eclipse PDE, then went on to show Harmony running Geronimo and the Roller blogging webapp. Craig Hayman had slides of Harmony running the WebSphere eXtreme scale server too. With the existing modular architecture of Apache Harmony, it is trivial to build right-sized runtimes for each of these.

Many people have speculated that this may be the last JavaOne conference, but I don't think so. It is a well organized shop window for Sun/Oracle, and while it remains important to associate Java the ecosystem with a particular vendor I believe that the marketing event will survive.

Thursday, April 02, 2009

Not so fast Dagastine

Earlier this week David Dagastine was quick off the mark reporting HotSpot as:

"the fastest, most reliable, and most widely deployed, open source Java Runtime in the world"
running on an Intel Nehalem. Unfortunately a bit too quick, since he didn't wait for the results to come in from IBM's J9 VM which posted a faster result as reported by Derek (who is, allegedy 8-10% better looking too)!

The HotSpot results were partially attributed to improvements in the class library, which David (when challenged by Mark Wielaard) tells us "need additional work to integrate into OpenJDK, but that work is in progress now" -- I assume that they are in the same queue as the TreeMap performance changes that are apparently in the process of being contributed back to Apache Harmony.

So today "Sun Java powered by the HotSpot Server VM" is not the fastest and not open source. Looks like some catching-up to do again.

Saturday, March 21, 2009

For best results ...

This sticker was on the back of a Jeremy Clarkson DVD in Sainsbury's!

Remember to remove all outer packaging first.

Friday, March 20, 2009

Sun welcome at EclipseCon

I'm delighted to see that Sun are Gold sponsors of the EclipseCon event next week in Santa Clara. It's sure to be a great event, and encouraging to see Sun able to demo their Eclipse support for Solaris, Glassfish and JavaFX.

I wonder if the Netbeans girls will be back again too?

Thursday, March 19, 2009

The "wrong type" of nature

Here in the South of England spring seems to have arrived, and that means getting back out to the apiary to see how things have faired over winter.

I was pleased to see the colonies in both my hives are still alive and well (sadly not the case for many beekeepers' colonies this year).

You can see from the photo on the left that I left my hives with some winter protection. If you look closely you can see a metal "mouse guard" to keep the hibernating critters out, a super of honey for the bees' winter food, a solid roof which you can't see but has insulation inside, and the whole lot is surrounded with chicken wire -- which I had lying around in the garage.

The chicken wire is there to stop woodpeckers. In winter, when food is scarce, the woodpeckers figure out that there are bees in those big "lunch boxes", and they cling on the side to peck a hole and help themselves. Apparently, they will even tap around a bit and figure out that the handles are the thinnest part of the hive, and will attack there.

Well that was a good move, because the birds had helped themselves to some friends' bees that were placed close by:
Apart from the annoyance of having the woodwork damaged, and having bees eaten and wax damaged - having an extra entrance means the hive is harder to defend for the remaining bees and the reduced colony may well be "robbed" of any remaining food by stronger colonies. Once the woodpeckers learn there are free bees, they also tell their friends and you get them all round for a feast.

Luckily the damage was spotted quite quickly, and repairs made. Besides using chicken wire you can also tack plastic sheeting on the sides to stop the birds getting a hold.

With both my colonies growing again it will be time to restart the regular inspections and get back into the intriguing world of the honeybee.

Tuesday, February 17, 2009

The Magic Android powered phone

The BBC are reporting a new Android-based phone announcement at Mobile World Congress. The HTC "Magic" will go on sale in various European countries through Vodafone.

Judging by the job advertisements, you can also get a reasonable salary with Android programming skills compared to Java ME experience. It's still a tiny percentage of the demand but maybe a sign of some traction in larger companies. Watch out for the onset of non-phone devices powered by Android.

Saturday, December 13, 2008

Bops, tree hugging, and safe wagers

Back in July I wrote about how Apache Harmony's TreeMap implementation had been used (with modifications) to deliver improvements in Sun's SPECjbb2005 score by "a solid 3-5% depending on the platform" according to Dave who also promised that they were "making the necessary steps to give back [the] code changes to Apache Harmony"; even though they are not obliged to do so according to the Apache License.

Did you expect us to sit and wait?!

IBM has just published its latest SPECjbb2005 scores with, amongst other things, an enhanced version of TreeMap from Harmony that gives them a 10% boost, and thereby move ahead in many reporting categories. Derek has all the details, and I encourage you to read them.

It would be interesting to look back and compare the changes Sun and IBM made to Harmony's TreeMap, and merge them for the common good.

While I'm sure Dave's intentions are pure, I doubt the suits will let him contribute his changes back to Apache and all the talk of "requiring innovation sharing in the commons" only applies in one direction. In fact, I'm so sure, that if Sun do successfully contribute their TreeMap changes back to Apache Harmony I promise to give £500 of my own hard-earned money to The Woodland Trust to benefit some real trees.

Friday, December 12, 2008

Debugging mixed-language code in Apache Harmony + Eclipse

I've been hacking in managed runtimes for more years than I'd care to mention, and plenty of that time has been spent in debuggers of one sort or another!

...and that's the point. Implementing a runtime requires that you have a suite of debuggers that cover the different target platform native code and the dynamic late-bound language du jour. You have to switch from Eclipse to WinDbg/gdb and back again frequently.

Intel's Software group have (for some unknown reason) been quite low key about their innovation in this space, with the development of an integrated debugger for Java and JNI environments. This advance has been enabled by the combination of support in the Eclipse Java Debug Tools and Apache Harmony runtime.

Now, as you step into a native method from Java, the Eclipse debugger shows a logical representation of the mixed Java thread / C thread stack frames and allows you to inspect variables etc. in either language. They also support the JNI methods that call back into Java so you get full roundtripping. Cool.

For most people who need to debug a system comprising Java and native code this is exactly what you want to do, and its a real time saver to be able to manage the breakpoints, variable watches and flow of control from a single IDE.

Kudos to Chris, Vasily, Gregory, Ilya, et al who made this work. Clearly people who live in the real world!

Go watch the screencast demo, download it and try it out!

Friday, November 14, 2008

EVP of software leaves Sun - that makes 6,001

Rich Green, the long time Executive Vice President of Software has quit Sun according to press reports about the new round of job cuts affecting 6,000 workers.

Seems that the Sun website was quick to notice, his webpage went offline almost immediately (but can still be found in the Google cache). I guess there is no love lost with the software guys and girls running the website ;-)

Not a good time to be made redundant (if ever there was). Good luck to all those smart people who will be out there in the market place now looking for new opportunities.

Tuesday, October 28, 2008

Sun get a visit from Uncle Sam


Things really are not looking too good at Sun right now.

The recent SEC filing shows that Southeastern Asset Management (a.k.a. Uncle "SAM") have spent $2,115,932,199 to bring their acquired stock of Sun up to 21% of the company's total outstanding shares.

The aggregate number and percentage of Securities to which this Schedule 13D relates is 160,566,828 shares of the common stock of the Issuer, constituting approximately 21.2% of the 757,953,410 shares outstanding.
But Uncle Sam is no sugar daddy. He wants to move in for a while and "help" Sun to manage the company...
As the result of investment analysis or the occurrence of events, Southeastern may desire to participate in discussions with the particular portfolio company's management or with third parties about significant matters in which Southeastern may suggest possible courses of action to assist in building corporate intrinsic value per share or to cause the Company's true economic value to be recognized. In such situations, Southeastern may elect to convert a filing on Schedule 13G to a filing on Schedule 13D in order to be more active in corporate governance and management matters, and to have the ability to enter into discussions with third parties concerning proposed corporate transactions of a significant nature.
Hmm - what do you read into the expressions "suggest possible courses of action", "be more active in corporate governance and management" and "enter into discussions with third parties concerning proposed transactions of a significant nature" ?? That in conjunction with news that co-founder Bechtolsheim has found new interests outside Sun points to some loss of control by the current management team.

This comes hot on the heels of Sun's preliminary results for 1Q09 in which, beyond the earnings warnings, there was the notice of imminent "goodwill impairment" which essentially means that the premium they paid on their acquisitions probably won't be earned back. Which acquisition(s) they overpaid for is a matter of debate. I expect the results call will be tough on Thursday.

We can only speculate about what Uncle Sam will think of Sun's software business, and the Java group in particular. Now it is open sourced it is hard to see how Sam will realize it's "intrinsic value" for Sun's shareholders. Let's hope Sam sees the value of freedom over no-cost -- and let's hope that Sun can turn things around in this period of global economic downturn.

Tuesday, September 23, 2008

T-Mobile G1 (tm) with Google (tm) with Harmony

The Press Release is out, and T-Mobile G1 is officially launched. I'm delighted that T-Mobile are making the platform available in the US and Europe simultaneously.

Of course, the real difference is that unlike some other 3G /WiFi feature phones you can buy today, Android is a truly open platform built with software available under the Apache License. This makes it a real contender for businesses to invest their effort and value and compete.

The core class libraries come from Apache Harmony -- the same source that IBM are using for its server class certified Java offerings [>], and Sun Microsystems are using in their performance release [>]. It's a testament to the innovation and flexibility embodied in the Harmony project that the code can span from mainframe zSeries machines to mobile phones.

Interesting that the selling points for the phone are the applications it runs and how well they are integrated and adapted to the mobile device. I believe it will give the young incumbent a run for its money -- and that is good news for the consumer.

Monday, September 22, 2008

Ubuntu - now with added fizz

It took me about two seconds to put Ubuntu on my laptop this morning :-)

This bottle caught my eye yesterday, so I had to buy it. I guess if you are looking for ways to shift large quantities of Fair Trade sugar, then selling cola has to be as good a way as any.

Now if Pepsi & Coke switched to Fair Trade sugar I bet that would make a difference to the growing economies far beyond the cola-nization of offering free umbrellas to advertise their wares.

Ubuntu : "I am because we are".

Posted by Picasa

Tuesday, August 19, 2008

The first honey crop!

Last weekend, after getting back from holiday, I took eight frames from the bee hive at home, and extracted the beautiful golden honey.

The bees have been doing a great job, and they are fascinating to watch and learn about. It's a little civilization in your backyard. There are little victories as you see bees laden with nectar and pollen return safely to the hive, and concerns as you see hornets and wasps trying to get in to seal the pupae and honey.

The honey we got is very light coloured. The bees are collecting from a variety of sources in our neighbourhood, and although I don't like honey I'm told it has a wonderful delicate flavour.

Here's a slideshow of the hive and honey. From eight frames we got about 23 lbs (~10kg) of honey and there are another thirty frames to go! Obviously way more honey than we'll ever be able to use.


Friday, July 25, 2008

Microsoft sponsors The Apache Software Foundation

At OSCON today, Microsoft announced that they are becoming a Platinum level sponsor of The Apache Software Foundation, joining the existing sponsors at that level, Google and Yahoo!

It's great to see Apache's no-nonsense, technology focused approach to open source being clearly recognized as a place where everyone can collaborate as equals.