Hypes and Disruptors

On of the most interesting challenges in software development is a proper analysis on a new piece of technology, process of theoretical framework, hereafter referred to as a concept. Is the concept a hype or is it a disruptor?

Probably stating the obvious, but a hype is a concept where the marketing value surpasses the actual value and generates a bubble of activity that can only end in a bursting sometime or later.

Now, a disruptor on the other hand, is a concept with the ability to displace existing concepts. A disruptive technology has the power to upset existing industry models and do so quickly as well, so it is wise to keep tabs on it. That is, if it can be identified as such.

Nearly every discussion on new concepts is clouded because -- let's face it -- most people do not know what they are talking about. Instead of being upfront about this, smoke and mirrors is applied, and conversation is muddled. I think any discussion on a new concept should be an open-minded exploration with the sole aim of going to the heart of the matter and identifying that what makes the concept valuable. Or not, of course.

To further complicate matters, new concepts rightly often focus only on a subdomain of that in which it can be applied, so that it only fixes its particular problem. Therefore most of the time, a new concept "forgets" about acquired wisdom and experience from previous technology.

I think that it is an acceptable price to be pay for a disruptor. A small group of people can hardly be expected to take on the whole domain. In the discussion those gaps need to be accounted for, and extensions or future concepts can build on the Shoulders of Giants, as it were.

Sometimes a concept has a real price attached, one that is fundamental to the idea. The price could well be too high to pay. Here we are being harassed by religious proponents of the concept, sometimes tagging themselves as evangelists, as if proud of their obnoxious nature, immunity to reason and good conversation, and loud-mouthedness in proclaiming their one-way gospels. A detestable group.

Finally, there are the forces of conservation. These people have learned the hard way that most of what they see is hype and have henceforth insulated themselves to new ideas. Any new concept is struck down well before proper discussion has taken place by applying the dreaded "Hype" label.

Ignorance, Extremism and Conservation -- these are the enemies of innovation.

We have seen a lot of disruptors over time. And a lot of hypes as well. Seek out the people with whom you can explore the concept and avoid the ones that cannot. Or, if they can be rescued from ignorance, extremism and conservation, educate them how to be open-minded in the matter. Our industry would benefit from a broadly adopted Enlightened mindset.

Case "Dynamic Order By" When?

The other day I ran into a nicety of SQL that I was not aware of. Not that that is special, but I think this nicely illustrates how the Old World has taken into account aspects which are not (yet) secured in the New World.

Let us suppose you have a table that has a column called state. Within that column you have the following distinct values:
- pending_approval
- approved
- rejected

Let us also assume that you want to order retrieved rows in the exact same order as I listed the items.

A regular order by won't do. It will show the results alphabetically, either ascending or descending.

Usually, I am inclined to prepend the values with sortable pieces of text such as 1_, however this messes up the semantics of the value. Doing this makes me feel dirty.

Well, what do you know, SQL actually took this scenario into account:

  select * from table
  order by case state
    when 'pending_approval' then 0
    when 'approved' then 1
    when 'rejected' then 2
  end;

What a gem!

The functionality goes quite deep, giving you even the option to fetch values from other columns.

Regrettably, this functionality cannot be accessed through JPA/Hibernate. My good colleague and database prima donna, Ron Smeets, advises to lay a view upon the table. Interesting idea, though for the specific use case I am looking at, this is not an option. This probably means going back to using JDBC.

As far as I am concerned, certainly goes to show that the database layer still holds treasures for application developers to discover.

Why Maven is a Schmuck (part II)

This evening I had an excellent dinner with my friends Anthony and Erik, both of whom I hold in high esteem. There is nothing like a full-blown argument in good company and while we broached several topics, the one most noteworthy was that of Maven.

Anthony opened his broadside fire by claiming that a lot of the problems he saw in projects nowadays were Maven-related. I couldn't believe what I heard: a kindred spirit with sharp criticism on the untouchable Maven!

Maven adds considerable complexity by adding a layer of abstraction, obfuscating a lot of the dependency magic. Developers like Erik have no problem grasping the whole playing field, but the impact of this complexity on the team must not be underestimated. After all, team members must be facilitated by the tools, not obstructed or forced to learn more than they need to know, which in a way is what Maven imposes.

Anthony went so far as to claim that for large projects a dedicated Maven expert is required. I feel a lot of sympathy for that claim.

We concluded on the following regarding Maven's upsides:

• IDE support; it has solid support in setting up project structures -- think Eclipse and IDEA plugins which generate from the POM. With the right POM, you are up and running literally in minutes
  
• Reports; the built-in support of Maven for many types of reports is way more advanced than what its peers offer
  
• Intermediate product support; any project which produces intermediate JARs to be reused by other projects is bound to be a dependency nightmare, but Maven takes care of that mess best. Question is: aren't these the niche projects?
  

One thing we dealt with resolutely was the convention to split up a project in many sub-projects. A sub-project which does not result in its own deployable, nor is about to be reused in any way, does not deserve its own sub-project. The argument that future architecture splits are thus enabled is an academic train of thought which is rarely exercised in real life, though imposes heavy costs throughout the life cycle of the project. Keep it simple. Most projects belong in just one project folder.

At the end of the argument, both Anthony and I were more in favor of internalization of the dependencies (see my other article on Maven), while Erik remained firmly on the side of Maven dependency management.

For internalization, there is a price to pay though: internalization means that one developer goes through the pain of "getting it right", but after that it works for all other team members and for deployment to the target environments.

Excellent discussion of which I hope to see a lot more in the future.

Leaving Independent IP

As of September 1st, I will no longer be working as a full-time employee of Independent IP. I will still be involved in an advisory role.

The last three years have been a wild roller coaster ride and I am incredibly proud at the things we achieved together.

I stood at the cradle of little Fuga and saw it grow. I saw it take its first little steps by delivering content to webshops. I smugly noticed how the little kid outran all competing products, gained lots of compliments all around and was generally hailed as THE digital delivery solution for music. I watched the company change to accomodate the maturing of Fuga.

It is a kind of magic to nurture and grow a product like this. Very intensive, but very rewarding, more so if customers are happy and benefits are visible to everyone.

Professionally, when I started out on this venture I had lots of ideas how I wanted to develop the software, such as:

• test-driven development
  
• high test coverage
  
• development team works from home
  
• demanding recruitment criteria
  
• weekly release cycle (short sprints)
  
• clear split development / operations
  
• component-based design
  
• issue tracking
  
• integration / staging / production (similar to OTAP)
  
• proper testdata on staging
  
• testing by developers on integration
  
• testing by non-developers on staging
  
• external code reviews (I heartily recommend Springsource)
  
• process for functional requirements
  
• open culture where process and results can be challenged
  
• topic complexity; easy=text, medium=voice, hard=face-to-face
  
• utilize the strengths of team members to knit a tight team
  
• wrong people off the bus, right people on the bus (read Good to Great if you did not already!)
  

It is good to see that most of these ideas contributed to the success of the product. The value of all of these ideas, however, pales in the shadow of the wonderful and talented people whom I had the honor to serve with. It is a rare occasion to participate in a team that makes you feel like you can take on the world and yet that was exactly what I felt. For all of you, please allow Henry V to put into words my feelings:

And Crispin Crispian shall ne'er go by,
From this day to the ending of the world,
But we in it shall be remembered-
We few, we happy few, we band of brothers;
For he to-day that sheds his blood with me
Shall be my brother; be he ne'er so vile,
This day shall gentle his condition;
And gentlemen in England now-a-bed
Shall think themselves accurs'd they were not here,
And hold their manhoods cheap whiles any speaks
That fought with us upon Saint Crispin's day.

Fare well, my friends.

The Black Art of SEO

I have been dabbling in the superstitious side of IT recently; Search Engine Optimization, or SEO for short.

Google guards it search engine secret jealously so know one knows for sure what is hot and what is not. Like any half-decent government organization that desires a certain behaviour from its citizens, it is glad to tell people how they should behave, but refrains from telling the exact way in which the appraisal takes place. Better to let people act in the spirit of an idea, than to give them the exact parameters and inner workings, which tends to provoke abuse.

On the downside, and for this reason, the SEO market looks a lot like the paranormal community -- no one knows for sure what is happening, so anyone can claim to have divined The Algorithm (tm). Proof is sketchy and scientific evidence hard to come by, more so if the search engine's crawlers take some time to revisit your website, leaving a big gap between a change and its effect. What seems to be sure is that Google has succeeded in striking fear into the hearts of SEO experts. Ever since it degraded the ranking of sites meddling with link farms, people seem to be watching their steps more carefully, aiming to please, not to anger. What more could the Great G ask for?

Having said that, SEO has its own scale of near-certain knows to urban legends. On the near-certain side are the HTML tags on a page which are important for Google to determine what a page is about. Keywords with a higher importance are taken from the following tags:

• URL; give your pages names which are meaningful in relation to the content


• title; it is advised to stick the important keywords to the start of the sentence


• h1/h2/h3; in order of importance


• bold/strong; emphasis might denote importance of keyword


• image; the name of the file and especially the alt text


• link; the title of the link and the anchor text

As I am naturally disinclined to human-crawl a website looking for these tags, I hacked together a Java program based on HtmlUnit (my latest love!) that does the job for me. I give it a URL and it parses the site, following all links to pages on the same site. Of every page it checks the important HTML elements. The outcome is a report where you can see what keywords are being picked up by a crawler. It does not do SEO for you, but it certainly helps if you do not see the keywords that you would have expected in there.

If you want the app (disclaimer: no unit tests, all-round error checking, build etc in place), are interested in a report for your site or can add valuable SEO divining capacities to the tool, please drop me a line.

Why Gems is mere carbon and Maven is a schmuck

The first time you run Ruby gems to install components, you have to admit it all looks pretty slick. The application is set up and the developer goes to work, adds a gem here and there

Then the downside becomes painfully obvious.

All the components that were installed by the developer have to be installed by each and every other developer who wants to work on the application.

All the components have to be installed on all the servers where the application is to be deployed.

Truly, I am not kidding.

So for every component added to the application you will have to run this gauntlet. Since component changes are in flux and need to be dealt with as such, instability is bound to result.

To add insult to injury, the concept out-of-the-box further imperils Ruby's Cloud-ability, because it requires root access to deploy gems. As long as this issue is not addressed, Ruby is bound to miss out on the movement.

The Ruby community soon enough saw the downsides of the gems approach, but as solutions go, the proposed practice is lamentable. Gems need to be unpacked (similar to extracting content from a ZIP file) and added to a subfolder in the product. Then the application will have to be customized so that it reads the gems' content as part of the project structure.

I am sure the Ruby community would have much preferred a cleaner solution like Java offers. Instead they chose the enticing route of setting up their own disruptive technology. Too bad the disruption was for the user, not for competing technologies.

The Java community will not be left off the hook, however, having introduced and hailed its own piece of well-intended, but flawed reasoning and I will gladly go against the grain to make my case.

Maven has the concept of a repository where dependencies can be stored (usually JAR files). Some reasons are given on the Maven website why those dependencies must not be stored as part of the project; takes too much room, impedes checkouts and does not require to be versioned.

Can you believe that those are actually the real reasons for Maven's repositories? You better believe it, because there is not much other value to the concept, even downsides.

For one, there is the downside of having a compile-time dependency on external sites for which I wager most organizations have no Service Level Agreement in place. Unacceptable to any serious company. Though there are ways to bypass the problem by setting up a local repository, most of the time that is overkill, introducing more complexity than the value you get out of the deal.

And let's not forget the extra hassle you get when setting up your development project. Normally I select the JARs in the library subfolder of a project and be done with. Not so with a Maven repository. You will actually have to go through the POM and harvest the dependencies one by one.

So here is Maven with its fancy repository, saving you disk storage and checkout time. Would you consider that a good trade-off for an extra layer of complexity and more hassle?

If you build your application, always internalize your dependencies whenever you can. You keep your teammates, your systems operator and yourself much happier this way.

Ruby, the End of the Honeymoon

At the risk of being threatened, ostracized or even lynched by an angry mob, I am boldly stating here that the Ruby movement is past its heyday.

"Why?", you rightly ask.

The signs were there all along. It started with the usual pioneering enthousiasm that comes with a new technology, and admittedly, I joined in the choir. Ruby, or more accurately, Ruby on Rails, sure played its part in upsetting the existing order, especially in the field of webclient development. The copy-cats soon followed, with Grails being the most honest about where it borrowed its idea from.

For some time now, I have noticed that quality Ruby people are hard to come by. For some time too, I noticed, that a number of Ruby pioneers start to lose interest and drift to other promising languages such as Scala, or even back into the fold of Java.

Ruby on Rails applications are slow. I have had the luxury to have a Ruby crack in my team who knows how to tune these little beasts and crack that whip. Grails applications, on the other hand, are not slow. They run directly on the JVM. An enviable position, that Ruby believers can only glare at without offering a solid alternative.

Aha, I hear you say, that's where JRuby, Ironruby and the likes come in. Now I would like you to heed this -- we are talking here about hand-crafted Ruby interpreters that make sure Ruby can be run on the JVM. Tools that have their own issues (we had a huge concurrency problem with an older JRuby version) and introduce one extra element of complexity into the Technology stack. Everything that people think up in the Ruby interpreter needs to be faithfully duplicated in JRuby, but then written up in Java. DRY? Not really. The notoriously sloppy documentation and audit trail that unfortunately appears to be a convention in the Ruby community only adds to the task. I have great respect for these brave developers, but less faith to build essential software on such a platform. And don't even get me started on how JRuby-hosted Ruby code is its own island (yes, you can ferry between the islands, but you will get seasick), contrary to Groovy and Scala applications which interact freely with Java code.

Finally, the commoditization of hosting web applications, especially by Google, pushes the knife home. Google's cheap hosting offer only allows you to run on the JVM, effectively adding a premium to your service if you want to run full-fledged Ruby on your own servers.

Ruby and Rails have brought a lot of good to software development, but alas, their time in the limelight has come to an end.

A Tribute to the Bug Hunt

There are few experiences in software development quite as humbling and as satisfying both as the Bug Hunt. The search for an elusive problem that causes systems to crash, data to be lost and bits to be unwantingly inverted is a wild chase where one is peeling off several layers of proximate causes in order to get to the ultimate one.

The interesting aspect is how the Hunt is often opened by having a preconception of what the problem is and where it resides. Often a hunt ends there. The interesting ones are those where the preconceptions get shattered several times as they are revealed as mere proximate causes or worse, false leads.

The humbling part, for me, consists of how I sometimes have an idee fixe about what the cause is and how that assumption turns out to be untrue. I daresay that this aspect made me more conscious in my interactions with other people and prevent me from jumping to conclusions.

There are many out there who are happy to have found the proximate cause and stop there. The symptom is solved. For example, a warning light flickers in an irritating way, so take out that light. Symptom solution is in my experience also typical when an electrician comes to repair something in my house; they observe the symptom, declare that the cause and proceed to fix it. They don't ask that critical question: why does this light flicker?

It amazes me to see this is often the prevailing mood in software development as well. If you are in a position to question someone who has just solved a problem, ask them why the problem occured and keep on asking why until you have satisfyingly heard the ultimate cause. If this is not business-as-usual, I predict you will find the level of understanding to be lacking.

The lack of understanding will ultimately come back and bite you when your dirty-fixed applications will be plagued by problems that start cooperating in ways unimagined to cause serious mischief, multiple root causes making the hunt even harder. Your application is more likely to reach the end of its life prematurely.

The Bug Hunt consists of five parts:
- arrange a problem description
- reproduce the problem in a separate environment
- determine the root cause by isolating the problem
- propose a concept solution
- implement the fix

The only part that should be subjected to economical choice is the fourth one, ie the type of solution. Here you can choose a proper solution, a quick-fix or no solution at all as the situation allows. Do not save costs on the hunt itself! You will be sorry that you did.

The end result of the hunt should be a narrative that explains exactly what has happened. If you are using an issue tracking system, demand that the hunt is documented meticulously, so knowledge is shared and valuable information on the patient preserved.

Doing all this right, will keep knowledge about your application with your team on a high level and the application itself is kept in good shape. You are better positioned to scale up, address new problems, refactor old code and add new functionality. The Bug Hunters are the heroes of your team; honor them!

Remember, the most powerful weapon of the Bug Hunter is that one little word: why.

Good Hunting!