Netuality

Since September 17 (three long days and nobody broke the silence ?) the 3.0.1 maintenance release of Eclipse is available for download. There is a respectable list of bugfixes and the same announcement contains the links to latest versions for core plugins such as GEF, EMF, CDT and (surprise, surprise ?) UML2 (v1.0.1).

Yes, you can increment your SDK version via Eclipse update, but it doesn't work for GEF and EMF, so it's time to fire up those download managers …

PS: Well it seems that the automatic upgrade does work for GEF and EMF, if they're correctly installed, with features & all.

“More and more pressure is on Microsoft to rush Longhorn. Apparently, a number of Microsoft licensees will get some sort of massive refund if the product isn't delivered in 2006, and the word on the street is that the code keeps breaking. My guess is that at the last minute the company will kludge together a workable system missing a lot of features.”

John C. Dvorak, Now What Do We Do? Dept. 13 July 2004

“In order to make this date (of 2006), we've had to simplify some things, to stagger it. One of the things we're staggering is the Windows storage work,” Jim Allchin, Microsoft's vice president in charge of Windows development, said in an interview with CNET News.com

CNET News.com Microsoft revamps its plans for Longhorn August 27, 2004

While the REST API is mentioned in the Top 10 Reasons to use Jira, I can hardly see any reference to such an external API in the documentation (I mean, besides XML-RPC and upcoming SOAP support) and even Google can't clear the issue. But I can confirm you, it is possible to fully(?) automate Jira using HTTP requests.

Recently, I was asked to programatically login a user into Jira and open a browser with the results of an advanced search. This is part of a really slick integration between one of my employer's products and a JIRA server – mainly helping our testers to avoid bug duplication and also providing insightful statistics for QA dept.

I've started doing it the hard way, trying to obtain and pass JSESSIONID and other useless stuff between my application and the browser, until I realized that Jira can be fully controlled through HTTP in a REST-like manner. Let me explain. Normally, if you are not logged into Jira and launch a browser with a carefully crafted search request (well, you know how to reverse enginner POSTs into GETs don't you ?) – then a graceful invitation to log in is everything you'll ever obtain. But, if you add at the end of your request “&os_username=” + USER + “&os_password=”+ PASS bang ! not only you obtain the desired search results, but you are automagically logged into Jira for the rest of the browsing session. Yes, yes yes : here I come, integration ! A couple of hours later, I am able to programatically insert bug reports, extract bug details, compute statistics and open customized advanced searches.

To quote a classic : Jira ! Jira ! Jira !. Docs would be nice, though.

PS I'm testing this on a Jira Professional 2.6.1-#65. YMMV.

This is the summary of a nice thread spotted on eclipse.platform.rcp. The initial question was I am constantly having to deal with resistance from my developers, which mostly boils down to resistance to SWT, and hesitance from peers in the company. There are compelling arguments to counter nearly every criticism that I've encounted. However, I'm having difficulty with one question, has anyone else based a non-IDE product on Eclipse?. A few answers later, boy do we have some nice eye candy to show !

	The XMF Mosaic Development Environment from Xantium provides “platform for engineering semantically rich languages and tools” (errr, whatever this means, just looks nice).
	Eclipse Trader is not quite a fully-fledged app, but a set of plugins for the Eclipse platform, dedicated to the building of an online stock trading system. This is an open-sourced, very recent project, with interesting features such as charts built with data extracted from Yahoo! Finance.
	EPIC from Incremental stands for a nice looking management software for the ship repair industry. You'll spot some gorgeous Eclipse Forms in the screenshots !

Recently, doing some SWT research, I've stumbled over a few open-sourced apps, not very popular but still interesting to disect. Thought you'd like me to share, so:

	Quezen is an open source administration utility for the OpenJMS messaging system and is an implementation of the OpenJMS 0.7.6.1 Administration API.
	Freeciv-swt is a Freeciv client using SWT widgets, port of the jfreeciv client. It's a rather corny app, using SWT2 as far as I can tell, but exhibiting interesting features, so worth a look.
	Argonaute (French) is a file manager developed by a team of 6 students from Université de Marne-la-Vallée. A bit slow on big filesystems (such as mine) but it has autocompletion, display filters and drag and drop between multiple instances.

Enjoy …

NB: Here's my older post about SWT apps …

Very very busy lately, but I'd like to share some knowledge about a few useful Java OSS gems that were not easy to find. Mr. Google, please 'index this':

1. Aspirin is a self-contained SMTP server (send only) written in Java, open-sourced and free. It simplifies configuration and deployment by allowing your app server to send emails without passing through an external SMTP server. The project is heavily inspired from Apache James code (thus its licencing terms). The few problems I see right now are : possible performance issues when sending big volumes of mail, behavior still erratic (sometimes sending fails without plausible reason), failure reports which do not provide reasons of failure. However, the thingie works pretty well and is a big time saver because, well, configuration is not the most pleasant part of a complex server.

2. If you produce a lot of reports and want to send them automatically on a remote printing server you may use JIPSI (quickstart in English, but site in German) which implements CUPS as a Java Print Service API. This little beauty was found by one of my coworkers and the 'report guys' seem to be making good use of it.

3. You're in for some serious processing on OpenOffice documents using the freely available DTD's (downloadable from the OOo CVS server) ? Then hold your horses ! I've tried to make sensible use of them and failed abruptly. Let's just say that those DTDs are a big pain in the a**: to begin with, no tool is able to transform them into a schema. I've tried XmlSpy and a few other exotic softwares, without success. Even basic stuff like parsing with a validating parser does not work. So much for the usefulness of open standards. Eventually, I have ended up by using the excellent Writer2Latex. Don't be fooled by the name, you may do all sorts of conversions with it, including Writer to XHTML, which I was interested in. You can even write your own plugin to boot some exotic formats, because Writer2Latex is built around a modified version of XMerge. Officially, XMerge is the solution for visualizing documents on 'Small Devices', but it really is a fancy plugin-based document converter. Most probably (too lazy to check the sources), SAX-based with a nonvalidating parser. Go figure.

4. The Eclipse download site has now links to a BitTorrent tracker. I just used it succesfully in order to download RC3 at a reasonable download rate (anyway, being on wifi right now I wasn't expecting blazing speed). I found interesting that all the other peers were using Azureus, a torrent client written in Java+SWT. Azureus is a fantastic source of knowledge, choke full of tips and techniques for writing professional-looking and very responsive SWT apps. But not only : Azureus is also a great example about how to write a plugin-ready app, which performs automatic updates from the net. Not bad, at all.

NOTE : unfortunately, this trick is again unavailable starting with Eclipsev3M9, the last version which made that possible was v3M8

• both teams have grown = more activity on the repository,
• vast majority of developers is integrating frequently, aka every little bit of functionality is committed as soon as it's stable and working. Of course, before committing, there's the mandatory update to check for consistency against the most recent codebase. This means that at least a few synchronizations will be performed by each member of the team, each day.
• Code generators. It's true that common sense dictates that nothing generated should be stored in CVS, because this may be source of frequent conflicts and loads the repository in an inefficient manner. However, when one routine generation (for each of the few modules) may take between 3 and 10 minutes and produces thousands of files, it becomes pretty obvious that it should not become part of the build process. What gives : in the days when the model has some important changes, a few different subsequent versions of 3-15MB jar files are committed on the repository. Update process starts to slow down and soon a part of the team is lagging, waiting to download the update. The irony is that usually everybody is downloading slowly and inefficiently the SAME big file. Of course, there are also other kinds of traffic via the VPN, such as Netmeeting (but you can't really have a conversation when everybody in the team is updating the codebase and slows the VPN to a crawl).

This boils down to having a 'read-only' local repository, perfect mirror of the main CVS server, which will be used only for updates. Both the server and its mirror are Linuxes. My choice for mirroring was good old rsync. While cvsup seems to be all the rage nowadays, I headed towards rsync because a) it comes pre-installed within any decent Linux distro and b) I am using Gentoo on my home desktop, so rsync is a tool that I learned to use [and abuse] almost daily, via portage.

We won't lose a lot of time explaining how to set up an rsync server, since it's very well explained in this rather old but useful tutorial. There's just a small twist : we were planning to synchronize frequently so we ran rsync daemon independently, not via xinetd. The config file on the server:

2;root@dev  /etc> more rsyncd.conf

motd file = /etc/rsyncd.motd

log file = /var/log/rsyncd.log

pid file = /var/run/rsyncd.pid

lock file = /var/run/rsync.lock



[cvs]

   path = /opt/cvsroot

   comment = CVS Rsync Server

   uid = nobody

   gid = nobody

   read only = yes

   list = yes

   hosts allow = 10.0.10.100/255.255.255.0

is a classical read-only, anonymous (we're on VPN, right ?) mirroring setup. Next step is making a rsync service in /etc/init.d or appending the line “/usr/bin/rsync –daemon” to /etc/rc.d/rc.local, to be sure that the daemon restarts after reboot [especially when you are NOT administrating the server].

On the client, there are some small tricks to make it work. First one, setup your client CVS repository in the same location as on the server, '/opt/cvsroot' in our case (because you are going to synchro CVSROOT as well, which contains absolute paths in some if its files).

The mirroring script is something in the spirit of:

#!/bin/bash



source /etc/profile

cd /opt/cvsroot



RUN=`ps x | grep rsync | grep -v grep | wc -l`

if [ "$RUN" -gt 0 ]; then

    #already running

    exit 1

fi



rsync -azv --stats 10.0.3.193::cvs/CVSROOT/history /home/cvs/history >/tmp/histo



sum1=`sum /home/cvs/history`

sum2=`sum /opt/cvsroot/CVSROOT/history`



if [ "$sum1" = "$sum2" ]; then

    #nothing to do

    date > /tmp/empty

    exit 0

fi



date > /tmp/started

cat /dev/null > /tmp/ended

rsync -azv --stats --delete --force 10.0.3.193::cvs /opt/cvsroot > /tmp/status

cd /opt

date > /tmp/ended

This (badly written) script is heavily inspired from a script found on a Debian user mailing list:

• exits in the case of a long running rsync process
• assures that the full rsync is triggered by changes in CVSROOT/history. This is a neat trick which minimizes the server activity if you are syncing frequently, as we will do.
• outputs stuff in some files in the /tmp directory. This has a double purpose. First is avoiding root mailbox pollution (because we're 'cron-ing' it, the output will be visible as a few hundreds of mails each day). Second is providing data for a web page on the client machine which tells at a glance what is the synchronization status. AFAIK rsync will not write incomplete files, but the usual sound advice is to make an update between successive synchronizations.

The small 'synchronization status' page (left as an exercise for the reader ) just prints the dates and the last few lines of the output files; this is a dumbed-down sample :

The only step left is to create a file in cron.d containing the line:

0-59/5 7-23 * * * cvs your_mirroring_script.sh

(meaning sychronization each 5 minutes from 07:00 to 23:00) and you may start enjoying blazingly fast CVS updates in Eclipse.

Anybody looking for OSS reporting solutions in Java usually has to make a choice between Apache FOP and Jasper Reports*. While having somewhat different feature sets and addressing distinct reporting solutions, the two APIs boil down to the same basic thing : generate a report from an XML file (or stream/string/whatever). FOP has a clear advantage of standardization (based on XSL-Formatting Objects) while Jasper plays more in the pragmatic field of obtaining those 80% results with a minimum of effort and uses a proprietary XML format.

But FOP is not a standalone reporting solution : it's just a way of transforming XSL-FO files into a report. In order to fill the report with the necessary data, the obvious choice is a templating engine such as Jakarta Velocity. Thus a FOP report creation is a two-step operation :

• create the XML report via Velocity
• feed the XML stream to FOP

Jasper alleviates this problem by including its own binding engine, the only restriction being that input data should support some constraints (such as putting your 'rows' inside a JRDataSource).

Both Jasper and FOP allow inclusion of graphic files inside, usual formats (GIF, JPEG) are supported, however FOP has a nice bonus of rendering SVG inside reports. Unfortunately, this comes with the price of using Batik SVG Toolkit, which is a bulky (close to 2MB) and rather slow API. While processing your dynamic charts as XML files (Velocity again) is a seducing idea, the abysmal performance of SVG rendering will make you give up in no time. Unfortunately, I speak from experience.

At first sight, FOP has a lot more options for output format, compared to Jasper Reports. Of course there's PDF and direct printing via AWT, but also Postscript, PCL, MIF as well as SVG. These choices are quite intriguing, since Postscript and PCL are printing formats (easily obtained by redirecting the specific printer queue into a file), MIF is a rather obscure Adobe format (for Framemaker) and SVG … well, a SVG report is too darn slow to be useable (yes, I was foolish enough to try this, too). Jasper makes again a pragmatic choice by allowing really useful output formats such as HTML, CSV and XSL (never underestimate the power of Excel); and of course: direct printing via AWT and PDF.
While FOP's latest version (0.20.5) was released almost a year ago (summer 2003), Jasper Reports is bubbling with activity – Teodor releases a minor version each one or two months (latest being 0.5.3 at 18.05.2004).

I've decided to use as a 'lab rat' one of the apps developed during my 'startup days': the client GUI is written in Swing and features a few mildly complex reports generated using Velocity+FOP. FOP version is 0.20.4 (the current version back in Q1-2003, when we had to quit dreaming about the 'next financing round' and development halted) but as I already told you FOP has evolved little since then. Though, it's perfectly reasonable to use this implementation as a witness for comparison with Jasper (on the opposite, Jasper has evolved a great deal since Q1-2003).

Back then, the report development cycle was quite simplistic. In fact, the XSL-FO templates were written by hand inside a text editor and the application code was run (via a Junit testcase and some necessary configuration and business data mocking) in order to generate a PDF report. In the case of errors, we had feedback by examining the error traces. Visual feedback was given by the PDF output. While simple to perform, this cycle was extremely tiresome after a while as there was an important overhead : start a new JVM, initialize FOP, fire Acrobat Reader (plus we were using some crappy – even by the standards of 2003 – 1GHz machines w 256/512MB RAM). A WYSIWYG editor would have been nice, so one of my coworkers has made some research and the only solution he found was XMLSpy (Stylevision not available back then) – but, at 800USD/seat this was 'a bit' pricey** for us (only the Enterprise flavor covers FO WYSIWYG editing !?). Another interesting idea was to use one of the conversion tools (from RTF to FOP) such as Jfor, XMLMind or rtf2fo (of these products, only Jfor is free, but feature-poor). What stopped us from doing it was that the generated FO was overly complex : we needed comprehensible cut_the_crap files because we were going to integrate inside Velocity templates. And when you have tens of tags and blocks inside blocks and not the slightest idea which one is a row, which one is a column and which one is a transparent dumbass artefact, it's a gruesome trial-and-error task to integrate even simple VTL loops. And you'd have to do this each time you change something in the report : yikes ! Conclusion : the report development cycle was primitive for FOP and there was no way we could change it.

Things are quite different for Jasper Report : there are a lot of available report designers, and some of them are free. While the complete list is on Jasper Report site, I'd like to note at least three of them :

• iReport is a Swing editor and very interesting because it's not only covering the basic Jasper functionality but also supplementary features such as barcode support (which is admittedly as easy as embedding a barcode font in Jasper with two lines of XML, but much easier to make it via a mouse click). iReport is free, which is excellent, but is a standalone app without IDE integration, and as any complex Swing app is quite slow and a memory hog.
• if you are a developer using Eclipse, you'd appreciate two graphical editors based on Eclipse GEF, available as Eclipse plugins : JasperAssistant and SunshineReports. None of them is free and, at least on paper, the functionality seem identical, but SunshineReports has only the older 1.1 version downloadable, which is free but does NOT work with recent builds of Eclipse 3. How the heck am I supposed to test it ? On the contrary, Assistant has a much more relaxed attitude allowing the download of a free trial for the latest version of their product. Maybe too relaxed, though, because – even if (theoretically) limited in number of usages – you can use the trial as much as you want to***. But if you are serious about doing Jasper in Eclipse you should probably buy Assistant, available for a rather decent 59USD price tag. I am currently using it and it's a good tool.

So much for the tools, let's get the job done. The bad part : if you're experienced with FO templates, don't expect to be immediately proficient with Jasper, even with a GUI editor. The structure of an FO document has powerful analogies with HTML : you have tables, rows, cells, stuff like that, inside special constructs called blocks. It's relatively easy to use a language such as VTL in order to create nested tables, alternating colors and other data layout tricks. You can even render a tree-organized data via a recursive VTL macro, and everything is smooth and easy to understand. Jasper is completely different and at first sight you'll be shocked by its apparent lack of functionality : only rectangles, lines, ellipses, images, boilerplate text and fields (variable text). Each one of this elements has an extensive set of properties about when the element should be displayed, stretch type, associated expression for value and so on. Basically, you'd have to write Java code instead of Velocity macros and call this code from the corresponding properties of various report elements. If at the beginning it feels a little awkward, after a while it comes quite natural and simple. As for nesting and other advanced layouts, there is a powerful concept of 'subreport'. And yes I've managed to render a tree using a recursive subreport, but given the poor performance the final choice was to flatten the data into a vector then feed it into a simple Jasper report. So pay attention to the depth of 'subreporting'.

Once the reports were completely migrated, I've benchmarked a simple one (without SVG, charts, barcodes or other 'exotic' characteristics). The test machine is a 2.4GHz P4 w 512MB Toshiba Satellite laptop. In the case of FOP, the compiled velocity template and the FOP Driver are cached between successive runs. In the case of Jasper, the report is precompiled and loaded only on first run, then refilled with new data before each generation. The lazy loading and caching of reporting engines is the cause of important time differences between the generation of the first report and the subsequent reports. Delta memory is measured after garbage collection. The values presented are median for 10 runs of the 'benchmark report'.

While I am totally pro-Jasper after this short experiment, it is important to note that commercial and well-maintained FO rendering engines such as RenderX XEP claim improved performance upon FOP. Depending on your requirements, environment and reporting legacy apps, an FO-based solution might be better, especially when report generation is only on server-side.

Of course, usual disclaimer apply: this benchmark is valid only for my specific report in my specific application so YMMV.

* While I am aware that other OSS solutions do exist for Java, I consider these two as 'mainstream'.

** Did I mentioned that we were a startup with financing problems ?

*** No, I'm not going to explain here how it can be done.

Mr. Jesus M. Rodriguez asked fellow Java bloggers where are all those SWT apps. Here's a possible answer.

First, do not forget that Eclipse is a big ecosystem. There are a bunch of software development shops making a living from selling Eclipse plugins : Omondo, Genuitec, Instantiations, Exadel, etc. Add here the vast majority of the older/bigger tool producers (such as Borland/Together, Visual Paradigm …) which ported their products as Eclipse plugins. Count also important product suites from (of course) IBM (WSDD, Rational, etc.), (for sure) Lotus and (surprise ?) Novell and SAP. There is life in here and a great deal of it is due to SWT. There are a few strange Swingonian hybrids, but most of the plugins and products are purely SWT-based.

SWT is still young, so unfortunately there are only a few interesting opensourced projects based on SWT. Among them there's Azureus, a BitTorrent client which is consistently in the top 5 downloads at SourceForge. I have noticed an interesting fact : Azureus is THE ONLY desktop Java app (which is not an IDE or a development tool) and which I consistently find installed on different systems (friends, coworkers). Add to the list Sancho, a P2P multiprotocol client, SQLAdmin another rdb administration tool and RSSOwl plain simple RSS feedreader. These are all decent performing apps and their sources are ripe for grabbing and inspiration.

But this is just the tip of the iceberg. The Eclipse newsgroups are full of people discussing RCP development strategies and asking SWT questions. And – you know – they're not all unemployed ex-dotcommers with Yahoo emails. But let's not make speculations about people email addresses – let's just look at an interesting existing product : GDFSuite is a whole suite/platform targeted at geobusiness applications. In their presentation from EclipseCon 2004, Frank Gerhardt (SENS) and Chris Wege (DaimlerChrysler AG) shared a few development experiences related with GDFSuite rich client, built upon the Eclipse platform. I bet there are a few other important RCP-based products which will be announced during 2004.

Last, but not least, I have some personal experience to share as well. My current employer sells a production supervision tool targeted at food industry, with a SWT-based rich client. When I say “sells” I mean “already sold quite a few licenses and currently successfully installing at big customers”. The SWT-client is deployed in its Motif version on P3/500Mhz machines running Linux. It's fast and slick. It's not very aesthetically pleasing and you can not play Solitaire on it, but this doesn't seem to interest the guys cutting meat or the girls which are inspecting the barcodes on the package of your next 1.99$ steak. Some other SWT-based products are undergoing development in our Java department, but of course they are really really really secret, they will be announced as soon as they are ready and they'll kick some serious ass There are probably quite a lot of other companies capitalizing on SWT to make fast (or 'rich') clients for their commercial products.

Swing is beautiful. Swing is conceived with MVC in mind. Swing has Jgoodies. Swing is also slowly-moving and memory-bulimic. When all I have to do is a simple interface with a text displayed in big fonts and a “Print barcode” button, does it really have to pass through 10 nested layers of listeners ? What shall I do with Swing on a CPU-handicapped computer in production environment, where 200ms is considered a big delay ? But of course, if you want intellectual challenges and nicely architectured clients, go for Swing. If all you need is to be efficient and please the customer, take SWT for a ride.

I know, I know – I may sound like the VB advocates a few years ago. But it's not the point. It's a matter of doing the right compromise between a rich functional API and a simple fast API, between generalization and the need for custom coding. My opinion is that Swing has – for the moment – failed to reach the good compromise. SWT it's just the next candidate.

PS Jess is a dual-licensed inference engine for Java. The next version (7 aka 'Charlemagne') includes an Eclipse-based IDE. Unfortunately, not being a licensed Jess user I do not have access to latest beta versions. Maybe somebody could shed some light upon this question.

PPS There's some more stuff at SWT Community Page

PPPS A bit of history : Swing (JFC) was made public in april 1997. SWT became available to developers, along with the Eclipse donation, in november 2001.

According to David Heinemeier Hansson, the RAILS 1KLOC (!) web framework written in Ruby was used to develop Basecamp, a mildly complex project management webapp. What's fascinating is the Ruby code used to develop Basecamp is only 4KLOC (according to the RAILS document) and was developed in less than 2man*month (including 212 test cases and all the bells and gingles). Although I strongly suspect that there was a lot of templating going on behind the hoods and I doubt that template development time was included, the efficiency is amazing, especially if you consider that they have started almost from scratch. Makes you wonder what would be possible to perform in Ruby with a comprehensive library such as PHP's PEAR (well, duh, ignoring the fact that it should be called REAR which is a very very very nasty name) ?