Netuality

You have to know that I’ve tried. Honestly, I did. I hoped to be able to read each and every page of “Java persistence with Hibernate” (revised edition of “Hibernate in action”), by Christian Bauer and Gavin King. But, I gave up before reading a third of it, then I continued only reading some sections. First, because I know Hibernate, I’ve used Hibernate in all the Java projects I’ve been involved with – in the last 5 years or so. Second, because the content from the first edition is more than familiar to me. And third, because this second edition is a massive > 800 pages book (double the number of pages in the first edition). At that rate, the fourth edition will be sold together with some sort of transportation device, because a mere human will not be able to carry that amount of paper. How did this happened ?

Hibernate is the perfect example of a successful Java open-source project. Initially started as a free alternative to commercial object-relational mapping tools, it quickly became mainstream. Lots of Java developers around the world use Hibernate for the data layer inside their projects. It’s very comfortable, just set some attributes or ask for a business object instance and Hibernate does all the ugly SQL for you. As a developer, you are then comfortably protected from that nasty relational database, and gently swim in a sea of nicely bound objects. Right ? No, not exactly. Each object-relationship mapping tool has its own ways of being handled efficiently, and this is where books like “Java persistence with Hibernate” come into play. This book teaches you how to work with Hibernate, with a “real-world” example: the Caveat-Emptor online auction application.

Since the first edition of the book was written, lots of things happened in the Hibernate world and you can see their impact in “Java persistence with Hibernate”. Most important is the release of the 3.x version line and its different ameliorations and new features: code annotations used as mapping descriptors, package naming reorganization inside the API, but most important the standardization under the umbrella of JPA (Java Persistence API) for a smooth integration with EJB 3 inside Java EE 5 servers. And this, is a little bit funny. Yesterday, Hibernate was the main proof that it is possible to make industrial-quality projects within a “J2EE-less” environment, today Hibernate has put a suit and a tie, joined the ranks of Jboss, then Redhat, and it lures the unsuspecting Java developers towards the wonderful and (sometimes) expensive world of Java EE 5 application servers. Which is not necessarily a bad move for the Hibernate API, but it’s a proof that in order to thrive as an open-source project, you need so much more than the Sourceforge account and some passion …

Enough with that, let’s take a look at the book content. Some 75% if it is in fact the content of the first edition, updated and completed. You learn what object-relational mapping is, the advantages, the quirks, the recommended way of developing with Hibernate. For a better understanding, single chapters from the initial book were expanded into 2, sometimes more, chapters. The “unit of work” is now called “a conversation” and you’ve got a whole new chapter (11) about conversations, which is in fact pretty good stuff about session and transaction management. Christian and Gavin done some great writing about concurrency and isolation in the relatively small 10-th chapter – which is a must read even if you’re not interested in Hibernate, but you want to understand once and for all what are these concurrent transaction behaviors everyone is talking about. The entire 13th chapter is dedicated to fetching strategy and caching, which is a must read if you want performance and optimization from your application. There is also a good deal of EJB, JPA and EE 5 – related stuff scattered in multiple chapters. And finally, a solid 50-pages chapter is pimping the JSF (Java Server Faces) compliant web development framework, Jboss Seam. I have only managed to read a few pages of this final chapter, so cannot really comment. Note to self: play a little bit with that Seam thing.

To conclude, is this a fun book ? No. Is this a perfect book to convert young open-source fanatics to the wonders of Hibernate API ? Nope. Is this a book to read cover to cover during a weekend ? Not even close. Then, what is this ? First, it’s the best book out there about Hibernate (and there are quite a few on the market right now), maybe even the best book about ORM in Java, in general. It has lots of references to EJB, JPA and EE, it will help you to easily sell a Hibernate project to the management. Even if the final implementation uses Spring … And finally, it’s the best Hibernate reference money can buy. When you have an issue, open the darn index and search, there are 90% chances your problem will be solved. And that’s a nice accomplishment. Don’t get this book because it’s funny, because it’s a nice read, about a new innovative open-source project. Buy it because it helps you grok ORM, write better code, deliver quality projects.

Java Caching System is one of the mainstream opensource and free Java caches*, along with OSCache, EHCache and JbossCache. Choosing JCS may be the subject of an article by itself, since this API has a vastly undeserved reputation of being a buggy, slow cache. Exactly this reputation has motivated the development of EHCache, which is in fact a fork of JCS. But, JCS has evolved a lot lately and is now a perfectly valid alternative for production; it still has a few occasional bugs, but nothing really bothersome. I've recently had this interesting experience of cleaning up and tuning a website powered by JCS. This dynamic Java-based site is exposed to a healthy traffic of 0.6-1.2 Mhits/day, with 12.000-25.000 unique visitors daily, and caching has greatly improved its performance. This article is a collection of tips and best practices not comprised (yet?) in the official JCS documentation.

Where to download JCS

This is usually the first question when one wants to use JCS. Since JCS is not a 'real' Jakarta project, but a module of the Turbine framework, there is no downloading link available on the main site. If you search on Google, this question has popped many times on different mail lists or blogs and it usually has two kinds of answers, both IMHO wrong:

• download the source of Turbine and you'll find JCS in the dependencies. No, you won't, because Turbine is build with Maven, which is supposed to automagically download all the needed dependencies and bring them to you on a silver plate. Meaning: tons of useless jars hidden somehwere in the murky depths of wherever Maven thinks is a nice install location. Uhh.
• build it from scratch. Another sadistic advice, given that JCS is also build with Maven. So you'll not only need to checkout the sources from CVS, but also install Maven. Then try to build JCS. And eventually give up. Like for instance in my case, I installed the monster^H^H^H^H^H^H wonderful build tool, then ran 'maven jar'. Instead of the expected result [you know, building the jar !] Maven performed a series of operations like running unit tests, washing teeth, cooking a turkey. Well, I suppose it was doing this, because I couldn't read the huge gobs of text running quickly on the screen. At the end, it miserably failed, with no logical explanations (too many explanations is the modern equivalent of unexplained). So I gave up. Again.

Fortunately, some kind souls at Jakarta (think of these developers as of a sort of secret congregation) provide clandestine latest 'demavenized' binary builds in obscure places; for JCS, the
location is here. I used the last 1.1 build without problems for a few weeks and I strongly recommend it.

Using the auxiliary disk cache

There's a common misconception that one doesn't need no stinkin' disk cache. Even on Hibernate site the example JCS configurations has the auxiliary disk cache commented out. Maybe this comes from the fact that JCS disk cache suffered from a memory leak (not true any more) or from the simplistic reasoning that disk access is inherently slower than memory access. Well it surely is, but at the same time it's probably much faster than some of the database queries, which could benefit from caching.

Also, it is interesting to note that incorrectly dimensioned 'memory' caches will make the Java process overflow from main memory to the swap disk. So you'll use the disk anyway, only in an un-optimized manner !

I wouldn't advise you to activate the auxiliary cache on disk without limiting its size, otherwise, the cache file would grow indefinitely. Controlling cache size is done by 2 parameters (MaxKeySize and OptimizeAtRemoveCount) example:

jcs.auxiliary.DC.attributes.MaxKeySize=2500
jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=2500

Only MaxKeySize is not enough, since it will only limit the number of keys pointing to values which are in disk cache. In fact, removing a value from the disk cache will only remove its key. But, the second (OptimizeAtRemoveCount) parameter will tell the cache to recreate a new file after a certain number of 'removes'. This new cache file will keep only the cached values corresponding to the remaining keys, thus cleaning all obsolete values, and of course will replace the old cache file. The size of disk cache and the remove count is of course subject of tuning in your own environment.

Tuning the memory shrinker

Although one of the JCS authors specify that the shrinker “is rarely necessary”, it might come handy especially in memory constrained environments or for really big caches. With one exception: be careful and specify the MaxSpoolPerRun parameter (undocumented yet, but discussed on the mailing list) otherwise the shrinking process might lead to spikes in CPU usage. I am using the shrinker like that:

jcs.default.cacheattributes.UseMemoryShrinker=true
jcs.default.cacheattributes.ShrinkerIntervalSeconds=3600
jcs.default.cacheattributes.MaxSpoolPerRun=300

YMMV.

Cache control via servlet

Again, undocumented, but people seem to know about it. The servlet class is org.apache.jcs.admin.servlet.JCSAdminServlet but do not expect it to work out of the box ! This servlet uses Velocity thus you'll need to :

• initialize Velocity before trying to access the servlet (or lazy, but you'll have to modify the servlet source)
• copy the templates into the Velocity template location. The templates (JCSAdminServletDefault.vm and JCSAdminServletRegionDetail.vm) are not (bug ? feature ?) in the jar, so you'll have to retrieve them from the CVS repository. For the moment, they are at this location.

These are my findings. I would have really appreciated to have these few pieces of info before starting the cache tuning. If anybody thinks this article is useful and/or needs to be completed, write a comment, send an email, wave hands. I'll try to come up with more details.

*For a complete list, see the corresponding section at Java-source.

Disclaimer : this review is based on the MEAP draft. Things might be (a little) different in the final version.

From a documentation point of view, Hibernate is one of the most notable exception in the world of open-source LGPL'ed projects. Its website offers a plethora of information, from solid documentation (the reference has no less than 141 pages) and various FAQs to sample projects and third-party resources. The forum is quite active and you may get answers to tricky questions. Or a little bit of rough treatment in case you haven't RTFM – but that is understandable, given the number of questions that the authors have to answer every day.

Under these circumstances, one might wonder what Gavin King (Hibernate founder) and Christian Bauer (documentation/website maintainer and Hibernate core developer) can add in order to be able to write a 400-pages book about Hibernate. I mean – sure – only by joining the reference documentation, different FAQs and guides, one can easily 'extract' a hefty 'manuscript' with more than 200 pages.

Well, I am extremely glad to tell you that this is not the case. The book not only gets you up to speed with Hibernate and its features (which the documentation does quite well). It also introduces you to the right way of developing and tuning an industrial-quality Hibernate application. I consider myself a pretty seasoned Hibernate developer, being familiar with the API since its 1.2 version in Q1-2002 (if I remember well the first app when we used Hibernate). However, I was proved wrong by “Hibernate in action” which describes best practices and even API features that were unknown or vaguely known to me. That is, until now.

The first chapter, in the good tradition of all first chapters in the world, is an introduction. It's a very well written introduction about why do we need ORM solutions in OO applications. The chapter explains the O/R impedance mismatch, while declaring quickly that OODB suck (immature and not widely adopted). Wel'll also find out that EJB also suck from a persistence point of view (for various reasons). Which can be quite a surprise knowing that Gavin is one of the authors of EJB3.0 specs. Or, on the contrary, this will explain a lot of things in the new EJB specs.

Now that we have cleared the “why Hibernate” issue, let's continue to the second chapter. Which – tradition obliged – is a “Hello, world” and a “Let's get started” chapter. Here you go, almost 50 pages later you should be able to write simple Hibernate-based persistence layers and integrate within an application server, like for instance … Jboss ! Humm, well, why not ? They are sponsors of the Hibernate project, after all.

In the 3rd chapter, our fresh knowledge will be put to good use by starting the development of an online auction application called CaveatEmptor. This app will follow our reading progression and will grow bigger and smarter chapter by chapter. But for the moment, we are at the inception phase. What gives : a little bit of analysis, a stylish class diagram of the domain model and the resulting mapping file. And if you thought (based on 2nd chapter) that the mapping file is very intuitive and simple, you're in for a big surprise : it is, indeed, intuitive and simple ! Quite bizarre for an open-source project. As a matter of fact, the mapping file is one of the pivotal elements of Hibernate, since it addresses directly the O/R impedance mismatch, a recipy for transparent linking your POJOs and the constrained relational model. No wonder that a big part of this chapter is aimed at explaining why and how the mapping works in Hibernate. You'll see how class associations and inheritance translate at the metadata and mapping level. You'll start to understand the things that you took for granted in the previous chapter and you'll have that pleasant “uuh, I see” chain reaction. Hold on, it's just the beginning.

Because chapter 4 is going to explain once and for all the lifecycle of persistent object in Hibernate, their behavior from a persistence point of view as well as the available fetching strategies. And if you thought you already knew everything by heart from the documentation … well, maybe you do know everything by heart. Nevertheless, it's very well synthetized in chapter 4 and I'll recommend it anytime to a coworker eager for Hibernate knowlege.

In the next chapter (the 5th) the rollercoaster slows down a bit. That is, if you already know the behavior associated with the four possible isolation modes in transactions, what are the different types of locking, what (the hell) MVCC means and the importance of transaction scopes. Chances are you already know some of this stuff quite well, but everybody needs a refresher from time to time, especially when it's well explained and when it comes with versioning and caching (1st and 2nd level) in Hibernate as a desert. By the way, I thought that OSCache supports clustering, not only SwarmCache and JbossCache, as stated in the book. There's even a thoroughly explained example of using JbossCache as a level 2 clustered cache for Hibernate, but it shouldn't be too hard to convert to other types of caching systems.

Now, if I were the author of the book, I would have placed chapter 6 before chapter 5. But I am not the author, which is quite fortunate for you dear readers since Christian and Gavin are much more competent than me at writing books about Hibernate (and probably at some other unrelated domains). They have decided to go back to mapping in chapter 6, after the short transaction/caching intermezzo. Well, they should know better… it's time for a serious dose of advanced mapping. This chapter is attacking interesting subjects such as custom mapping types (simple or composite) and (finally) the mapping of collections. Special guests stars: the whole gang of “sets, bags, lists and maps”, together with explanations about their relational equivalent (associations, associations and associations !). Oh and yes “polymorphic association” (section 6.4.3) – I wasn't even aware that Hibernate is able to do that… guess I'm not that 'seasoned' (as a Hibernate developer) after all.

The 7th chapter is about “Retrieving objects efficiently” : about 45 pages for the 'retrieving' part and 6 pages for the 'efficiently' part. Fair enough ! You'll learn how to master basic HQL queries (parameters, pagination …). You'll get a grip on the query by criteria API, as well as on advanced stuff such as dynamic queries, filters, subqueries and native SQL (very powerful). At the end of the chapter there's the Hibernate-specific solution for the n+1 selects problem, query caching and result iterators.

Following this wealth of useful knowledge, the 8th chapter starts a bit dry. Nevertheless, after a short introduction about Hibernate in managed environments, you'll find yourself again in the land of advanced programming techniques : application-level transaction implementation ! This is mostly new stuff (at least for me) – a great collection of best practices for transactional behavior management in industrial-quality apps. Somewhat unrelated but still interesting, the chapter ends with legacy schemas integration and a smart implementation example for audit logging.

The 9th (and last) chapter is about the roundtrip development in Hibernate using the classical toolset : Middlegen and/or hbm2java and/or XDoclet. All the available techniques are presented in a very detailed, step-by-step manner.

Wait : don't close the book, there's more ! Ignore Appendix A (a short and rather uninteresting document about SQL fundamentals – that is, if you know SQL). Appendix B contains mildly un-fascinating ORM implementation strategies pour les connaisseurs (come on guys, I'm just a dumb user). But – Appendix C is a great collection of real-world stories and by all means read them all ! Especially the last one, a treasure of hard to find knowledge (no spoilers, please…).

In the end, I have to confess that there is something truly interesting about 'Hibernate In Action' : albeit very technical, it reads astonishingly easy – and this kind of books is unfortunately very rare nowadays. My congratulations to the authors for this excellent piece of work – it was worth the wait.

As for you dear potential reader, if you already know all the information detailed in the book, I bow before you, great Hibernate wizard. But if you don't, what are you waiting for ? Because, if you're going to read only one technical book this summer, make sure that it's 'Hibernate In Action' (or, at least chapters 6,7 and 8, if you are that good !).

My first contact with Tapestry was more than 18 months ago. Back then, I was interested to find a web framework for integration with our custom Avalon-based (using the now-obsoleted) Phoenix server*. The web interface was ment to be backoffice stuff, for simple administration tasks as well as statistics reports. Given that the data access and bussines logic were already developed, we were looking for something simple to plug into a no-frills servlet container such as Jetty. we managed very easily to integrate Jetty as a Phoenix service and pass data through the engine context. But when we finally integrated Tapestry [into Jetty [inside Phoenix]] and make it display some aggregated statistics, the project funding was cut and the startup went south. But, that’s another story and rather uninteresting one.

Meanwhile, things have changed a bit. Tapestry had become a firsthand Apache Jakarta project, the Tapestry users list is more and more crowded, and again I see it used in my day work (by Teodor Danciu, one of my coworkers and incidentally author of Jasper Reports) and doing some moonlighting by myself for an older web project idea. And there is exceptional Eclipse support via Spindle plugin. While the ‘buzzword impact’ on Tapestry on a Java developer CV doesn’t yet measure up with Struts, this framework has obviously gained a lot of attention lately.

So, what’s so special about it ? If I’d have to choose only one small phrase I’d quote Howard Lewis Ship, Tapestry lead developer, from the preface of his book ‘Tapestry in Action’:

The central goal of Tapestry is to make the easiest choice the correct choice.

In my opinion this is the weight conceptual center of the framework. Everything, from the template system which has only the bare minimum scripting power, passing through the componentized model, up to the precise detailed error-reporting (quite unique feature in the opensource frameworks world) gently pushes you (the developer) to Do The Right Thing. To: put logic where it belongs (classes not templates), organize repetitive code in components, ignore the HTTP plumbing and use a real, consistent, MVC model in your apps (forms are readable and writable components of your pages). You don’t need to be Harry Tuttle to make a good Tapestry webapp, just a decent Java developer is enough. That’s more than I can tell about Struts …

Coming from a classic JSP-based webapp world, Tapestry is really a culture shock. The most appropriate way to visualise the difference is to imagine a pure C programmer abruptly passing to C++, into the objects world**. For a while, he will try to emulate the ‘old’ way of work, but soon enough he’ll give up and start coding his own classes. However, this C programmer will have to make some serious efforts, not necessarily because OOP is hard to leard, but in order to break his/her old habits.

“Tapestry in Action” is your exit route from the ugly world of HTTP stateless pages and spaghetti HTML intertwingled with Java code and various macros. It’ one of the best JSP detoxification pills available on the market right now.

The first part of the book (‘Using basic Tapestry components’) is nothing to brag about. It’s basically an updated and nicely organized version of the various tutorials already available via the Tapestry site, excepting probably some sections in chapter 5 (‘Form input validation’). By the way, the chapter 5 is freely downloadable on the Manning site and is a perfect read if you want a glimpse of the fundamental differences between Tapestry and a classic web framework (form validation being an essential part of any dynamic site). However, if you want to go over the ‘Hangman’*** phase you really need to dig into the next two book sections.

The second section ‘Creating Tapestry components’ is less covered by the documentation and tutorials. I’m specifically pointing here to the subsections ‘Tapestry under the hood’ (juicy details about pages and components lifecycle) and ‘Advanced techniques’ (there’s even an ‘Integrating with JSP’ chapter !). While it is true that any point from this chapter will generally be revealed by a search on Tapestry user list or (if you’re patient) by a kind soul answering your question on this same list, it’s nethertheless a good thing to have all the answers nicely organized on the corner of your desk.

The third and last chapter (‘Building complete Tapestry application’) is a complete novelty for Tapestry fans. It’s basically a thorough description of how to build a web application (a ‘virtual library’) from scratch using Tapestry. While the Jboss-EJB combination chosen by the author is not exactly my cup of tea (I’m rather into the Jetty+Picocontainer+Hibernate stuff) I can understand the strong appeal that it is suppposed to have among the J2EE jocks. Anyway, given the componentized nature of Tapestry, I should be able to migrate it relatively easily if I feel the need for it. The example app is contained in a hefty 1Meg downloadable archive of sources, build and deployment scripts included.

To conclude, ‘Tapestry in Action’ is a great book about how to change the way you are developing web applications. The steep learning courve is a little price to pay for a two or three-fold improvement in overall productivity. And this book should get you started really quick.

*Which AFAIK is still used at our former customer.
**There were some posts on Tapestry user list on about a certain conceptual ressemblance with Apple’s WebObjects. I can’t really pronounce upon this because I do not know WebObjects, but the name in itself is an interesting clue.
***’Hangman’ is the Tapestry ‘Petshop’ (although there is also a ‘real’ Tapestry Petshop referenced in the Wiki).

There is a certain contradiction which appears in modern projects concerning the unit testing strategy. On one hand, there is a powerful assertion stating that business logic testing should be completely disconnected from the database. This makes perfect sense in a certain way : the tests should check the business logic, not the database and/or the persistence layer. Then, generally the persistence layer is a fully-fledged product (such as the excellent Hibernate) or other JDO-esque solution which has its own testing suite – no need to check that it really works. Usually, the link between business objects and persistence is “faked” using mock objects. Basically, this means that testing the code doesn't need a running database (well, code testing doesn't need a database at all).

The database schema should also be tested – the only tool I am aware of is the excellent DbUnit. Although more targeted towards data testing, it copes quite well with schema testing. Nicely integrated with Ant, DbUnit is the right solution for your database testing needs. And yes you do need to test your database since it is supposed to evolve along with the code (there's a great article about Evolutionary database design on Martin Fowler's site).

Somehow, we instinctively feel that something is missing from this picture. We are testing the code, disconnected from the database – and also the database, in a independent manner. But how can we be sure that the persistence layer between the application model and the database is ok ? And I'm not talking about the persistence mechanics, but the data model itself. Basically, this goes down to mapping testing. I am aware of the fact that some special O/R bindings do not need mappings and there is a direct object-table correspondence, but I feel that this is generally a BadIdea^TM since it hampers the flexibility of both the application model structure and database schema.

In the small-to-medium-sized projects I've been working lately we didn't feel the need of mapping testing. This has a very simple reason : the person which is performing the change on the database schema is usually the same person which needs a certain modification in the application model. After performing the modification, quite often this same person starts the application and makes a functional test which implicitly checks the mapping. Most of the time this works just fine.

However, some nasty problems might appear when the project starts to grow :

• changing the mapping is more difficult, some kind of testing might give indications about the nature of the problem.
• there is a certain “schema decay” when some foreign keys cannot be created at a certain point, then their creation is forgotten when the data is finally consistent. Further with schema evolution, more and more objectual data model relations will not be backed up by integrity constraints.
• you may sometimes end up with unmapped and unused tables/views/columns.

A really useful testing tool should be able to check one or multiple mapping files against a database schema (via DbUnit, why not). The tool should :

• a) recognize different mapping formats (Hibernate, Castor, etc.) and different database types
• b) match the mapping declarations with the tables from the database, check their existence also the type of primitive columns
• c) warn if some constraints are wrong or missing (based on simple aggregation, cardinality or other hints from the mapping structure).
• d) warn for unmapped tables/views/columns.

Here's the good news : a tool which is able to perform a) and b) does exist ! And the bad news (purists will jump with disgust) : just for a moment, you should forgot about testing your code without the database. The solution is quite simple, build a unit test which fires up the persistence layer and retrieves at least one of each type of mapped object from a test database. If no exceptions are encountered, the test is ok. This is a basic but effective approach and :

• be prepared to have a testing database different from the development database but with schema automatically synchronized.
• harden your test case by inserting the most “exotic” test data you can find. If the data goes in via SQL (dbunit) but you have problems retrieving it via persistence layer, then look for missing schema constraints and sometimes some subtle mapping problems.

You could go one step further by performing update and deletion operations and check them via dbunit, but we have found that if the retrieval works, the persistence layer is perfectly able to perform updates and deletions. Now if your data layer is more complex, then just use some mock objects to test it – because it's a code issue and not a mapping issue.

If you are interested in the topic, just let me know by mail (still waiting for comment integration with FreeRoller). And yes, I'm still looking for a tool able to do a), b), c) and d).

Note : There is a simple technique that we are using currently. The idea is that, when the application starts, a simple retrieval is performed via the persistence layer for some objects that we know for sure must exist in all test and production datbases. It this succeeds you may be sure of two things : that the database connection really works and that the mapping is probably fine. This way, you don't have to wait the first persistence operation in order to see an error. Coupled with a nightly build and rerun, this little trick proved quite effective at keeping the mapping clean.