Netuality

Taming the big, bad, nasty websites

Archive for the ‘linux’ tag

(Almost) distributed CVS with Eclipse

leave a comment

NOTE : unfortunately, this trick is again unavailable starting with Eclipsev3M9, the last version which made that possible was v3M8



In Eclipse 3.0, in the CVS repository properties it is possible to separate the 'read' and 'write' access locations (see picture). AFAIK, this feature was meant for developers using extssh for CVS connection. They would use an anonymous account for updates in 'clear', and their respective accounts for commits over encrypted link. The basic idea is that clear connections are considerably faster than encrypted ones. However, you may change the whole connection string, not only the login and protocol, thus enabling the usage of two completely different repositories for 'read' and 'write' actions on CVS.

This might come handy in situations like the one we've recently been confronted with, in my team. The main CVS repository is located at a certain geographical distance (in the headquarters, in France) and the VPN bandwidth is nothing to brag about. Working with CVS was decent, but things have started to got meaner lately, mainly because of three reasons:

  • both teams have grown = more activity on the repository,
  • vast majority of developers is integrating frequently, aka every little bit of functionality is committed as soon as it's stable and working. Of course, before committing, there's the mandatory update to check for consistency against the most recent codebase. This means that at least a few synchronizations will be performed by each member of the team, each day.
  • Code generators. It's true that common sense dictates that nothing generated should be stored in CVS, because this may be source of frequent conflicts and loads the repository in an inefficient manner. However, when one routine generation (for each of the few modules) may take between 3 and 10 minutes and produces thousands of files, it becomes pretty obvious that it should not become part of the build process. What gives : in the days when the model has some important changes, a few different subsequent versions of 3-15MB jar files are committed on the repository. Update process starts to slow down and soon a part of the team is lagging, waiting to download the update. The irony is that usually everybody is downloading slowly and inefficiently the SAME big file. Of course, there are also other kinds of traffic via the VPN, such as Netmeeting (but you can't really have a conversation when everybody in the team is updating the codebase and slows the VPN to a crawl).

This boils down to having a 'read-only' local repository, perfect mirror of the main CVS server, which will be used only for updates. Both the server and its mirror are Linuxes. My choice for mirroring was good old rsync. While cvsup seems to be all the rage nowadays, I headed towards rsync because a) it comes pre-installed within any decent Linux distro and b) I am using Gentoo on my home desktop, so rsync is a tool that I learned to use [and abuse] almost daily, via portage.

We won't lose a lot of time explaining how to set up an rsync server, since it's very well explained in this rather old but useful tutorial. There's just a small twist : we were planning to synchronize frequently so we ran rsync daemon independently, not via xinetd. The config file on the server:

2;root@dev  /etc> more rsyncd.conf
motd file = /etc/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock

[cvs]
path = /opt/cvsroot
comment = CVS Rsync Server
uid = nobody
gid = nobody
read only = yes
list = yes
hosts allow = 10.0.10.100/255.255.255.0

is a classical read-only, anonymous (we're on VPN, right ?) mirroring setup. Next step is making a rsync service in /etc/init.d or appending the line “/usr/bin/rsync –daemon” to /etc/rc.d/rc.local, to be sure that the daemon restarts after reboot [especially when you are NOT administrating the server].

On the client, there are some small tricks to make it work. First one, setup your client CVS repository in the same location as on the server, '/opt/cvsroot' in our case (because you are going to synchro CVSROOT as well, which contains absolute paths in some if its files).

The mirroring script is something in the spirit of:

#!/bin/bash

source /etc/profile
cd /opt/cvsroot

RUN=`ps x | grep rsync | grep -v grep | wc -l`
if [ "$RUN" -gt 0 ]; then
#already running
exit 1
fi

rsync -azv --stats 10.0.3.193::cvs/CVSROOT/history /home/cvs/history >/tmp/histo

sum1=`sum /home/cvs/history`
sum2=`sum /opt/cvsroot/CVSROOT/history`

if [ "$sum1" = "$sum2" ]; then
#nothing to do
date > /tmp/empty
exit 0
fi

date > /tmp/started
cat /dev/null > /tmp/ended
rsync -azv --stats --delete --force 10.0.3.193::cvs /opt/cvsroot > /tmp/status
cd /opt
date > /tmp/ended


This (badly written) script is heavily inspired from a script found on a Debian user mailing list:

  • exits in the case of a long running rsync process
  • assures that the full rsync is triggered by changes in CVSROOT/history. This is a neat trick which minimizes the server activity if you are syncing frequently, as we will do.
  • outputs stuff in some files in the /tmp directory. This has a double purpose. First is avoiding root mailbox pollution (because we're 'cron-ing' it, the output will be visible as a few hundreds of mails each day). Second is providing data for a web page on the client machine which tells at a glance what is the synchronization status. AFAIK rsync will not write incomplete files, but the usual sound advice is to make an update between successive synchronizations.

The small 'synchronization status' page (left as an exercise for the reader :) ) just prints the dates and the last few lines of the output files; this is a dumbed-down sample :

CVS synchronization started at Mon Feb 23 09:45:18 EET 2004
, ended at Mon Feb 23 09:46:00 EET 2004

Last empty synchronization recorded at Mon Feb 23 09:40:05 EET 2004

No knowledge about recent deleted files

Last synchronized
_/modules/postpreparation/tools/
_/modules/postpreparation/tools/velocity/
_/gateway/
_/src/mailer/
_/src/old/
_/src/tools/
_/src/tools/Attic/
_/tools/
CVSROOT/history
wrote 60767 bytes read 471522 bytes 13142.94 bytes/sec
Last added files
_/ihm/AlignementChamps.java,v

The only step left is to create a file in cron.d containing the line:

0-59/5 7-23 * * * cvs your_mirroring_script.sh

(meaning sychronization each 5 minutes from 07:00 to 23:00) and you may start enjoying blazingly fast CVS updates in Eclipse.

Written by Adrian

June 2nd, 2004 at 1:44 pm

Posted in Tools

Tagged with , , , ,

Re: Where are all the SWT applications?

leave a comment

Mr. Jesus M. Rodriguez asked fellow Java bloggers where are all those SWT apps. Here's a possible answer.

First, do not forget that Eclipse is a big ecosystem. There are a bunch of software development shops making a living from selling Eclipse plugins : Omondo, Genuitec, Instantiations, Exadel, etc. Add here the vast majority of the older/bigger tool producers (such as Borland/Together, Visual Paradigm …) which ported their products as Eclipse plugins. Count also important product suites from (of course) IBM (WSDD, Rational, etc.), (for sure) Lotus and (surprise ?) Novell and SAP. There is life in here and a great deal of it is due to SWT. There are a few strange Swingonian hybrids, but most of the plugins and products are purely SWT-based.

SWT is still young, so unfortunately there are only a few interesting opensourced projects based on SWT. Among them there's Azureus, a BitTorrent client which is consistently in the top 5 downloads at SourceForge. I have noticed an interesting fact : Azureus is THE ONLY desktop Java app (which is not an IDE or a development tool) and which I consistently find installed on different systems (friends, coworkers). Add to the list Sancho, a P2P multiprotocol client, SQLAdmin another rdb administration tool and RSSOwl plain simple RSS feedreader. These are all decent performing apps and their sources are ripe for grabbing and inspiration.

But this is just the tip of the iceberg. The Eclipse newsgroups are full of people discussing RCP development strategies and asking SWT questions. And – you know – they're not all unemployed ex-dotcommers with Yahoo emails. But let's not make speculations about people email addresses – let's just look at an interesting existing product : GDFSuite is a whole suite/platform targeted at geobusiness applications. In their presentation from EclipseCon 2004, Frank Gerhardt (SENS) and Chris Wege (DaimlerChrysler AG) shared a few development experiences related with GDFSuite rich client, built upon the Eclipse platform. I bet there are a few other important RCP-based products which will be announced during 2004.

Last, but not least, I have some personal experience to share as well. My current employer sells a production supervision tool targeted at food industry, with a SWT-based rich client. When I say “sells” I mean “already sold quite a few licenses and currently successfully installing at big customers”. The SWT-client is deployed in its Motif version on P3/500Mhz machines running Linux. It's fast and slick. It's not very aesthetically pleasing and you can not play Solitaire on it, but this doesn't seem to interest the guys cutting meat or the girls which are inspecting the barcodes on the package of your next 1.99$ steak. Some other SWT-based products are undergoing development in our Java department, but of course they are really really really secret, they will be announced as soon as they are ready and they'll kick some serious ass ;) There are probably quite a lot of other companies capitalizing on SWT to make fast (or 'rich') clients for their commercial products.

Swing is beautiful. Swing is conceived with MVC in mind. Swing has Jgoodies. Swing is also slowly-moving and memory-bulimic. When all I have to do is a simple interface with a text displayed in big fonts and a “Print barcode” button, does it really have to pass through 10 nested layers of listeners ? What shall I do with Swing on a CPU-handicapped computer in production environment, where 200ms is considered a big delay ? But of course, if you want intellectual challenges and nicely architectured clients, go for Swing. If all you need is to be efficient and please the customer, take SWT for a ride.

I know, I know – I may sound like the VB advocates a few years ago. But it's not the point. It's a matter of doing the right compromise between a rich functional API and a simple fast API, between generalization and the need for custom coding. My opinion is that Swing has – for the moment – failed to reach the good compromise. SWT it's just the next candidate.

PS Jess is a dual-licensed inference engine for Java. The next version (7 aka 'Charlemagne') includes an Eclipse-based IDE. Unfortunately, not being a licensed Jess user I do not have access to latest beta versions. Maybe somebody could shed some light upon this question.

PPS There's some more stuff at SWT Community Page

PPPS A bit of history : Swing (JFC) was made public in april 1997. SWT became available to developers, along with the Eclipse donation, in november 2001.

Written by Adrian

April 4th, 2004 at 9:52 am

Posted in Tools

Tagged with , , ,

Eclipse 2.1 workspace deadlock – and a dirty but small workaround

leave a comment

It happened also on older versions but it does happen more frequently on the “final” 2.1 version. FYI : Gentoo Linux, Eclipse gtk, seems to be related somehow with bug 33138 (don't have the time to dig further).
Sometimes the monster simply hangs during a [take your pick : refactoring, new class generation] with an empty progress bar in the dialog box and a completely useless “Cancel” button. Been there, done that : kill -9 …
Then, trying to re-start Eclipse leads to a deadlock – while recovering workspace : dialog box, empty progress bar and useless “Cancel”. Freezed !
I have a lot of settings and projects so deleting the whole .metadata directory is just too painful. Therefore, I had to find out a smaller workaround : just delete the file .metadata/.plugins/org.eclipse.ui.workbench/workbench.xml and Eclipse restarts with a clean workspace. Some adjustments are lost but hey – my metadata is still there.

Written by Adrian

March 1st, 2004 at 5:21 pm

Posted in Tools

Tagged with ,

Using jython to internationalize a PHP app

leave a comment

At first, this might seem a mind-boggling combination. What do
jython and PHP have in common (excepting the fact that I am a Python fan
and my current consulting task is in a PHP project) ?
Well, internationalizing a PHP app is pretty much a trivial task.
If you are a sensible PHP programmer insisting to use PEAR instead of randomly choosing a script from the tons of snippets
populating the “scripting websites”, I18N is probably the
safest choice.
Maybe – for you – application maintainability and performance are not exactly important concerns.
For me, they are. This is why I chose to store internationalized texts in files rather than database.
I'd rather keep the database for real data, which is created, modified, aggregated and such.
And I'd rather like to have an internationalized error message on the screen even if the database is down.
Now we know that we'll use I18N and text will be kept in some php files. However, I am no professional translator and
have no desire to translate or to manually maintain the correspondence between translators files and PHP files
(no, translators won't modify PHP code, stop this nonsense right away).
Code generation comes immediately in mind.
Basically, my first idea was to investigate wether the files used by the translators can be quickly transformed to PHP,
and if I am able to generate their formats from my own files (aka. “roundtrip internationalization process” ?).
Unfortunately, this is not an easy task – as the only clue was that the translators use Office tools such as Word or Excel, because they
rely upon some specialized translation software integrated with these products.
The easy choice is Excel, since it allows a better organization of data than having to search for tables in a Word document.
The hard choice is the tool that I'd use for automatically reading and even generating Excel files.

The difficulty comes from the fact I don't have Windows with Office installed on my desktop, just Gentoo Linux and OpenOffice.
Thus, I am unable to write a simple Python script which could perform my generation tasks via automation.
Fortunately, this is not the first time I am confronted with the issue.
I happen to know that there is a very nice Java tool that I wholeheartedly
recommend for your Excel processing needs :
JExcelApi.

Still, Java is a heavyweight programming language – it would be a really bad idea to fire up the

monster just for some easy processing of Excel files.
Here's why Jython comes naturally into equation. Four hours and about
100 lines of debugged code later, here I am sitting on top of a perfectly functional internationalization tool which :

  • generates PHP code from a big xls file (the root vocabulary) which centralizes all the internationalization texts
  • generates 2-language xls files for translators usage
  • updates the root vocabulary starting from the files modified by the translators
  • Automation scripts are already in cron and there's also a nice text document explaining translators where to get
    their files and where to put them after modification. The resulting script is not exactly fast, but this is tooling
    and not production so this should not be a problem after all.

    Whatever your project contraints are, give Jython a try and you'll be amazed … As they put it on the
    Useless Python site – If it were any simpler, it would be illegal.
    Finally there's a trick not quite related with Jyhon, nevertheless interesting.
    There is an easy way of solving the problem of translating phrases with real data inside them, with easy parameter swapping.
    We'll use the good old sprintf but not directly. We'll pass through a not so popular but extremely useful function,
    call_user_func_array. Suppose that our example needs the
    user name and authorization profile description to display inside a nice message. All you have to do is to define placeholders
    in I18N files which would fit as the first argument for sprintf. The following example should make it clearer:

    localization/en/login.php
    $messages = array(
    'loggedin'=>'You are authenticated successfully as user %1$s with profile %2$s.'
    );
    $this->set($messages);
    
    localization/fr/login.php
    $messages = array(
    'loggedin'=>'Vous avez le profile %2$s en tant qu'utilisateur %1$s.'
    );
    $this->set($messages);
    
    Simple passing of multiple parameters to I18N in PHP. Example function without error processing or data domain checking.
    #this is the multiple parameter function
    function complexTranslation($i18n, $label, $params)
    {
      return call_user_func_array('sprintf',array_merge(array($i18n->_($label)),$params));
    }
    
    Then, you have to initialize your I18N object. This can be done in a generic manner for all pages.
    #specific I18N initialization stuff
    require_once 'I18N/Messages/File.php';
    $g_language_dir = dirname($_SERVER['PATH_TRANSLATED']).'/localization/';
    $i18n =& new I18N_Messages_File($g_langCode,$script_name,$g_language_dir);
    
    Finally, use the function.
    #translate the successfull login message
    $loginbox = Tools::complexTranslation($i18n,'loggedin',array($operator->name,$profile->description));
    

    Written by Adrian

    March 1st, 2004 at 5:20 pm

    Posted in Tools

    Tagged with , , , , , ,

    Shy Java, silent Java, sad Java

    leave a comment

    Last, week I got to visit Web Services & XML Integration Forum in Paris. It was an interesting day and I’ll probably come back with some more info, but for now I’ll just talk about what really impressed me. There was a nice triagle of expo booths somewhere in the ‘North-East’ area of the Forum, three little companies you might’ve heard about : Microsoft, Sun and Borland.

    The difference was striking. Microsoft is by far more prepared for this public event. They weren’t using presentation rooms, but instead conferencing right there in the booth all day long (with a lunch break of course). Although the presentation rooms were at least 4 times larger than the space they prepared (3 benchesx2m apx), I think they’ve got the best coverage at the Forum. They and Adobe, but Adobe was slightly off-topic (PDF is king at Adobe, and for a reason).

    Instead of using a classic projector, casting large shadows every time they wanted to point something on the screen, MS people installed a large (plasma, I think) flat-screen in their booth. They were the only ones I saw with wireless microphones, which gave them a lot of freedom to walk, wave hands and hypnotize the audience. But wait, there’s more … The Microsoft guys were talking “agile enterprise and processes” and one of them said once “you can put any backend database here, say MySQL for instance”. No, there is no confusion with “MS SQL”, the guy standing next to me was as stunned as I was. And to complete a superb marketing image, they wear no suits, no ties – everybody was relaxed and smiling. The only glitch was that the latest rage among MS employees seems to be cloning Bill’s haircut. I mean wake up guys, you can cut your hair in differents lenghts an shapes, you know…

    Just a few meters away, in the much smaller Borland booth, there was also a plasma display. However, no real presentations showing. Just a loop of images, logos of products and the word SPEED written in large, red letters. Um, ok. After 30 seconds I had to walk, it was incredibly boring.

    No crowd was gathered at Sun’s booth, neither. Some LCDs were displaying what looked like a non-interactive Project Rave demo and some screenshots of a Windows’look’alike Linux distribution, their famous Sun Desktop blabla whatever. WHY were they showing this at a Web Services & XML Forum ? But, probably Sun’s intentions were to scare away the visitors so they could watch undisturbed Microsoft presentations. In that case, they succeded extremely well. Plus, their unique positioning gave them a perfect view of the presentations – or was it Microsoft which strategically placed their flat screen in Sun’s direction ? Hmmm, makes you wonder.

    Anyway, to conclude, I watched two Microsoft speeches (one about their Web services strategy and one about Biztalk). It was fun, it was interesting, it was well explained. But it was not Java – and I am a Java developer. Should I be – hmm, ashamed or something ? For not watching instead : what, the bored faces of Sun staff ? Or : Jbuilder logo and words in big red letters ? Java has what we call an image problem. Some steps were made (the java.com site for instance) but there’s still a long way to go. It’s a matter of ‘packaging’ and might seem useless and frivolous. But how will Sun impose ‘friendlier’ development tools if the language itself still has a cold, silent, sad, geeky image, competing with .Net platform which has a much “sexier” look ?

    Written by Adrian

    December 4th, 2003 at 6:16 pm

    Posted in AndEverythingElse

    Tagged with , , , , , ,