Hypes and Disruptors

On of the most interesting challenges in software development is a proper analysis on a new piece of technology, process of theoretical framework, hereafter referred to as a concept. Is the concept a hype or is it a disruptor?

Probably stating the obvious, but a hype is a concept where the marketing value surpasses the actual value and generates a bubble of activity that can only end in a bursting sometime or later.

Now, a disruptor on the other hand, is a concept with the ability to displace existing concepts. A disruptive technology has the power to upset existing industry models and do so quickly as well, so it is wise to keep tabs on it. That is, if it can be identified as such.

Nearly every discussion on new concepts is clouded because -- let's face it -- most people do not know what they are talking about. Instead of being upfront about this, smoke and mirrors is applied, and conversation is muddled. I think any discussion on a new concept should be an open-minded exploration with the sole aim of going to the heart of the matter and identifying that what makes the concept valuable. Or not, of course.

To further complicate matters, new concepts rightly often focus only on a subdomain of that in which it can be applied, so that it only fixes its particular problem. Therefore most of the time, a new concept "forgets" about acquired wisdom and experience from previous technology.

I think that it is an acceptable price to be pay for a disruptor. A small group of people can hardly be expected to take on the whole domain. In the discussion those gaps need to be accounted for, and extensions or future concepts can build on the Shoulders of Giants, as it were.

Sometimes a concept has a real price attached, one that is fundamental to the idea. The price could well be too high to pay. Here we are being harassed by religious proponents of the concept, sometimes tagging themselves as evangelists, as if proud of their obnoxious nature, immunity to reason and good conversation, and loud-mouthedness in proclaiming their one-way gospels. A detestable group.

Finally, there are the forces of conservation. These people have learned the hard way that most of what they see is hype and have henceforth insulated themselves to new ideas. Any new concept is struck down well before proper discussion has taken place by applying the dreaded "Hype" label.

Ignorance, Extremism and Conservation -- these are the enemies of innovation.

We have seen a lot of disruptors over time. And a lot of hypes as well. Seek out the people with whom you can explore the concept and avoid the ones that cannot. Or, if they can be rescued from ignorance, extremism and conservation, educate them how to be open-minded in the matter. Our industry would benefit from a broadly adopted Enlightened mindset.

Case "Dynamic Order By" When?

The other day I ran into a nicety of SQL that I was not aware of. Not that that is special, but I think this nicely illustrates how the Old World has taken into account aspects which are not (yet) secured in the New World.

Let us suppose you have a table that has a column called state. Within that column you have the following distinct values:
- pending_approval
- approved
- rejected

Let us also assume that you want to order retrieved rows in the exact same order as I listed the items.

A regular order by won't do. It will show the results alphabetically, either ascending or descending.

Usually, I am inclined to prepend the values with sortable pieces of text such as 1_, however this messes up the semantics of the value. Doing this makes me feel dirty.

Well, what do you know, SQL actually took this scenario into account:

  select * from table
  order by case state
    when 'pending_approval' then 0
    when 'approved' then 1
    when 'rejected' then 2
  end;

What a gem!

The functionality goes quite deep, giving you even the option to fetch values from other columns.

Regrettably, this functionality cannot be accessed through JPA/Hibernate. My good colleague and database prima donna, Ron Smeets, advises to lay a view upon the table. Interesting idea, though for the specific use case I am looking at, this is not an option. This probably means going back to using JDBC.

As far as I am concerned, certainly goes to show that the database layer still holds treasures for application developers to discover.

Hiring people

If you are a manager and you are in a position to hire people, there is something you can do right really easy and really fast.

Do not be dismayed by people whom you think might be more talented than you yourself are!

Instead, relish the thought of hiring those people, even though the mere thought gives you shivers and breaches your comfort zone. You are now truly laying the foundations for a stronger organization.

In the end, you will be repaid with something much more valuable than a kingdom of mindless zombies, willingly assuming the position, nodding aye to everything you say and laughing to every stupid joke you make. Instead, you are in a position to experience being part of a team that can shape history.

Not only that, but there is no better way to keep you sharp and on-edge than surrounding yourself with those that are likewise equipped and honed.

So next time you are evaluating a candidate and you feel slightly threatened -- hire the person!

On Strategy

World War I is, bare none, the most tragic war fought the last thousand years. Some say the war started in the 1870's era and ended on the eve before World War II. From a technology and strategy point of view, this is true. Nothing quite boosts human mental prowess more than a deep and enduring crisis; the war started with outdated ideas and technology and ended with all the concepts that were used to wage the second tragedy in the same century.

Take, for example, the approach on the battlefield -- long lines of young men, trodding slowly towards their doom. And even those who arrived at the other end of No Man's Land could not hold on. There were no supplies, no backup, no mates to back up their positions. Seldom was terrain conquered for long. Reconquests of trenches bought dearly with the blood of many were the norm.

The British did not innovate their fundamental tactics. Their society at that time was highly stratified with little room for initiative in the lower strata. On the other hand, and contrary to popular belief, Germany was almost the inverse; it had a strong middle class. It invented a concept that nearly blew the Allies away -- Stormtroopers. Soldiers with decentralized decision making authority, often by non-commanding officers (NCOs in jargon, ie those below officer level). The tactics were aimed at deep penetrating units, circumventing tough defenses. New weapons were added to the arsenal; flamethrowers and grenades. The units had a high degree of self-coordination and self-reliance, without overbearing staff interference up to the level that they did not, and indeed, could not, know about. The unit was very successful and mostly due to incompetence of the over-hyped general Ludendorff (suffering from a nerve breakdown after his failed Peace Offensive), was the advantage thrown out of the window and could the Allies win the war.

Strategy permeates everything we do, even though we may not be conscious about it. It is about how we deal with everyday aspects. What do you do if someone treats you unkindly? Will you put up a fight? Offer friendship? Walk away? Strategies are most often not designed. Indeed, they evolve. You try something; doesn't work? Toss it out and try something else.

The best examples are those seen in nature. Think about the arms race between prey and predator, or how parasites and influenza viruses keep on adjusting to outrun what medicine can throw against it. The relentless pressure of evolution is the best evolver of new strategies.

Nor is strategy a single thing. It often consists of a number of characteristics that strengthen a generic theme; in companies, these could consist of culture, processes, skills, technology. The best companies in the world have strategies that work. If you break those down, you will find many, many parts that all interrelate and enforce each other. Do not even bother cherry-picking; it's bound to fail, because you do not have the overall strategy.

It makes me think. Can a really good strategy actually be designed? Can it come into existence without the pressure to change? Can it evolve into the target strategy withouth the cultural preconditions being in place?

I think pressure is essential. And so is the holistic view. Without pressure, no one will feel the need to change and without that energy nothing will happen. Without the holistic view, all the measures are shrapnel shots going nowhere. It needs to be part of a bigger picture, the parts reinforcing the whole.

As for evolution or design; strategy suffers from the same problem as software development. Everyone accept nowadays that the waterfall model of software development does not work and smart people invented the concept of iterative development, which for sure works a lot better. So it is with strategy. I firmly believe that it is not possible to design a 100% sound strategy upfront. You use your instincts and experience to aim as good as you can and then evolve the concept as you go.

Culture might well be the hardest one. How do you know you are not shooting for an impossible target? The Will to Succeed may well be blind to impossibilities. Do you have the right people on board? Are you yourself the right person to pull this through?

Personally, I stick to these rules:

• design and look at the big picture, see how it all interacts
  
• know what not to do and block that out
  
• relish a crisis; it offers the opportunity for change
  
• don't be too harsh on yourself for not seeing the entire picture (yet)
  
• be fair in appraising weaknesses
  
• learn from others, read biographies of successful people
  

Despite an intensive, lifelong fascination for anything that has to do with strategy, I still cannot claim to have mastered the subject, though I do claim to derive a lot of intellectual pleasure just thinking about it.

Geeh... migration

Last saturday we migrated two separate email domains and our Google Apps domain into one Google account. Besides the mailboxes, we therefore also had to set up Google Talk buddy lists, export/import the calendars and copy the Google documents.

We started by having a general reset on all passwords, so we could execute all phases easily.

The mailboxes went very smooth. Google has the right tooling to enable this. The only hiccup was the transformation from folders to Google's labels. Deeply nested folders come out really ugly, but hey, what can you expect? I prefer labels to folders, especially since folders do not cater for multiple annotations like labels do. And let's face it, like an animal that is both bird and fish, some mails refuse to be defined by a single label.

Google Talk is easy as well. There is a global setting which automatically makes domain users accept any buddy request -- this is key! After this setting is enabled, it is just a matter of copying the entire email list into the invitation text field for a user and the deed is done. Google could improve its service considerably by having a setting which automatically connects all domain users.

Calendar has the same ease of migration as Google Talk. Just export the calendar and reimport it, mapping to the right user names.

Google Docs is a bit of a pain. There is no way to change ownership of a document to someone outside of the domain (@Google -- improvement?). Luckily, the Google Docs API is splendid. I made a custom script that does the following:

• analyze all documents and flag those that are owned by a user
  
• check the access rights on that document
  
• download the document to a temporary location
  
• upload the document to the new Google Docs account
  
• reassign the access rights to the uploaded document (also using new email accounts)
  

The script performed well. Noteworthy is that on uploading a document in an automated way, a lot of garbage is created by Google. I think it has something to do with uploading a high number of documents in quick succession. Anyway, the script also cleaned up the Google garbage, which was not hard since the garbage documents still had the original filename used for downloading.

After the fact, I noticed I forgot about document labels and email notifications when a document changes. Not a big deal for people, but if you want to be all-inclusive, you should have this as well.

The migration took less than a working day with a minimum number of complaints.

Here's to Google!

How the Mighty Fall

Jim Collins, author of Good to Great, is back with another book. Whereas Good to Great focused on what makes companies stand out, How the Mighty Fall inverts the equation by looking at the prime reasons for companies not making it.

The author tries to apply the same sound formula; i) gather a lot of scientific data, ii) discern the patterns, iii) write a good book about it.

Oh, how Jim Collins must have bewailed his fate when he found out there are many, many more ways to fail than to make it.

Good to Great comes back with solid and applicable advice. This made the book an instant hit among the target audience. And rightly so.

How the Mighty Fall's main contribution to understanding failure is the 5-phased lifecycle of a company:

• Hubris born of success; the company grows and starts to adopt an arrogance stance ("We won, because we are better")
  
• Undisciplined pursuit of more; growth becomes the mainstay and strategy (in business mainly the art of not doing something) is thrown out of the window
  
• Denial of risk and peril; warning indicators of imminent doom are ignored
  
• Grasping for salvation; doom is upon the company and anything which might reverse the situation is tried
  
• Capitulation to irrelevance or death; the company is sold or goes bankrupt
  

In other words, a very generic way of describing failure, which has mainly to do with character and culture. Given that there are so many ways to fail, the author falls back on safe ground by invoking Good to Great lessons. The one addition he makes is what you can read from any biography of a great person, like Lincoln or Churchill -- never give up, never give in.

If you are new to Jim Collins' works, skip this one and honor him by starting with Good to Great instead.

Why Maven is a Schmuck (part II)

This evening I had an excellent dinner with my friends Anthony and Erik, both of whom I hold in high esteem. There is nothing like a full-blown argument in good company and while we broached several topics, the one most noteworthy was that of Maven.

Anthony opened his broadside fire by claiming that a lot of the problems he saw in projects nowadays were Maven-related. I couldn't believe what I heard: a kindred spirit with sharp criticism on the untouchable Maven!

Maven adds considerable complexity by adding a layer of abstraction, obfuscating a lot of the dependency magic. Developers like Erik have no problem grasping the whole playing field, but the impact of this complexity on the team must not be underestimated. After all, team members must be facilitated by the tools, not obstructed or forced to learn more than they need to know, which in a way is what Maven imposes.

Anthony went so far as to claim that for large projects a dedicated Maven expert is required. I feel a lot of sympathy for that claim.

We concluded on the following regarding Maven's upsides:

• IDE support; it has solid support in setting up project structures -- think Eclipse and IDEA plugins which generate from the POM. With the right POM, you are up and running literally in minutes
  
• Reports; the built-in support of Maven for many types of reports is way more advanced than what its peers offer
  
• Intermediate product support; any project which produces intermediate JARs to be reused by other projects is bound to be a dependency nightmare, but Maven takes care of that mess best. Question is: aren't these the niche projects?
  

One thing we dealt with resolutely was the convention to split up a project in many sub-projects. A sub-project which does not result in its own deployable, nor is about to be reused in any way, does not deserve its own sub-project. The argument that future architecture splits are thus enabled is an academic train of thought which is rarely exercised in real life, though imposes heavy costs throughout the life cycle of the project. Keep it simple. Most projects belong in just one project folder.

At the end of the argument, both Anthony and I were more in favor of internalization of the dependencies (see my other article on Maven), while Erik remained firmly on the side of Maven dependency management.

For internalization, there is a price to pay though: internalization means that one developer goes through the pain of "getting it right", but after that it works for all other team members and for deployment to the target environments.

Excellent discussion of which I hope to see a lot more in the future.

The 5 Temptations of a CEO

The other day, Erik pushed a little book into my hand, called The 5 Temptations of a CEO.

The book is written in what is called the business fable style. Ie, storytelling as a way of teaching. More correctly, I would term it a parable, but I think the inventors of the term business fable wanted to steer clear of that one for its religious connotations.

The book follows the soul searching of the protagonist, a failing CEO, who not so much reverses his course, as identifies what the reasons for his failing are.

The author goes forth to summarize these failing under appealing titles:

• Status over Results; when personal agendas, career paths and status symbols prevail over what the company requires to subsist. Suddenly there is a lot of "I" in the team.
  
• Popularity over Accountability; when personal ties are more important than keeping people to their promises, thus holding them accountable. People slacken off and don't take their responsibilities serious anymore. Complacency sets in. This one also has to do with using your reports as allies and venting ground.
  
• Certainty over Clarity; the need for ever more information and procrastrination by a decision maker before making the call. In board game terms, I would call this Analysis Paralysis.
  
• Harmony over Conflict; if people refuse to slug their arguments out to prevent others from getting flushed, there will be a lot of suboptimal decisions. Those who can work the dynamic best by emotionally blackmailing the group hold sway
  
• Invulnerability over Trust; working to be one-up will push others to raise their defences as well. Any argument will now become one of status and survival, immediately tied to personal interests.
  

The text behind the headers is my own interpretation, which will show you that I concur with the author in his conclusions. Powerful stuff for anyone in a leadership position to consider.

And The Answer Was 42

I just started at 42. In true Hitchhiker's style, I got equipped with various handy galactic hiker equipment, first and foremost being the towel, of course. Most interesting was the six-volume literary bundle and the threat that "a verbal will be taken soon". Glups. I guess I have to cancel my appointments and get down to it.

Leaving Independent IP

As of September 1st, I will no longer be working as a full-time employee of Independent IP. I will still be involved in an advisory role.

The last three years have been a wild roller coaster ride and I am incredibly proud at the things we achieved together.

I stood at the cradle of little Fuga and saw it grow. I saw it take its first little steps by delivering content to webshops. I smugly noticed how the little kid outran all competing products, gained lots of compliments all around and was generally hailed as THE digital delivery solution for music. I watched the company change to accomodate the maturing of Fuga.

It is a kind of magic to nurture and grow a product like this. Very intensive, but very rewarding, more so if customers are happy and benefits are visible to everyone.

Professionally, when I started out on this venture I had lots of ideas how I wanted to develop the software, such as:

• test-driven development
  
• high test coverage
  
• development team works from home
  
• demanding recruitment criteria
  
• weekly release cycle (short sprints)
  
• clear split development / operations
  
• component-based design
  
• issue tracking
  
• integration / staging / production (similar to OTAP)
  
• proper testdata on staging
  
• testing by developers on integration
  
• testing by non-developers on staging
  
• external code reviews (I heartily recommend Springsource)
  
• process for functional requirements
  
• open culture where process and results can be challenged
  
• topic complexity; easy=text, medium=voice, hard=face-to-face
  
• utilize the strengths of team members to knit a tight team
  
• wrong people off the bus, right people on the bus (read Good to Great if you did not already!)
  

It is good to see that most of these ideas contributed to the success of the product. The value of all of these ideas, however, pales in the shadow of the wonderful and talented people whom I had the honor to serve with. It is a rare occasion to participate in a team that makes you feel like you can take on the world and yet that was exactly what I felt. For all of you, please allow Henry V to put into words my feelings:

And Crispin Crispian shall ne'er go by,
From this day to the ending of the world,
But we in it shall be remembered-
We few, we happy few, we band of brothers;
For he to-day that sheds his blood with me
Shall be my brother; be he ne'er so vile,
This day shall gentle his condition;
And gentlemen in England now-a-bed
Shall think themselves accurs'd they were not here,
And hold their manhoods cheap whiles any speaks
That fought with us upon Saint Crispin's day.

Fare well, my friends.

In Defence of Pokemon

Often, I hear parents complain about Pokemon. For those who do not know what they are, I am talking here about the fantasy world where humans and creatures called Pokemon coexist. The Pokemon can be domesticated and trained for battles by Human trainers. Pokemon are all strange looking creatures, often resembling some kind of animal and with a name that is almost always a mangling of some real words with makeup words, resulting in gibberish to the uninitiated.

I think these parents are misguided and are themselves turning into a curmudgeoning generation that mistakenly bemoans the lack of taste/ethos/skills in the young ones. Rings a bell? It should, since it is a recurring theme.

But to the point. The movies may not be your cup of tea, but most of the major literary themes can be found in there -- love, ambition, jealously, rivalry, sacrifice, courage, perseverance, duty etc. The Pokemon world is coherent enough to suck in the watcher and appeal to kids. The one movie that for me signifies this quite clearly is Lucario and the History of Mew, where Truth, Friendship, Sacrifice and Duty feature strongly. Please forgive your children for not watching De Avonden (talk about boring), but do not accuse them of having no taste.

Then there is the computer game on the Nintendo DS. In game terms it is a grind, full of micro-rewards, a very effective way to draw people into games. What I see in my own kids is that they learn to cope with setbacks, think about ways to succeed and persevere through it. The game is in english and they try very hard to understand the words, even picking up a number of them. I am amazed at how much they can already understand of the text. Besides the solo version, the game offers easy ways for groups of kids to hook up and play/trade/fight/chat together. Calling it anti-social just becase the means for conducting social traffic is not well understood is not doing it justice.

Finally, there is the trading card game. This strongly stirs the collecting fever that seems to be part of our genetic heritage. The cards of the game come in four varieties: common, uncommon, rare and ultra-rare. A tried & tested tier system that was first introduced by Magic the Gathering. I am surprised to see how easy markets are formed ad hoc and anywhere by groups of kids with their books, boxes and single cards stuffed in pockets. They deal easily and smoothly. Yes, there are some negatives, for example stealing and ripping off smaller kids, but more importantly, I see my own kids building up stamina, tolerance for trading away their own property, empathy for what the other kid wants (Theory of Mind anyone?) and developing savvy in what constitutes a good deal.

The last reason why I think Pokemon is not bad is deeply personal. My youngest son is suffering from Selective Mutism, which is often misunderstood by people to be shyness. It's hard to explain the affliction, but it is probably best described as a persistent, strongly rooted refusal to speak in certain situations with certain people. The affliction is curable by therapy and one way for kids to overcome Selective Mutism is, in clinical terms, if the price of persistant silence is higher than the social cost. Ever since my son has been engaged in trading Pokemon cards, he often found the cost of maintaining silence too high, and sometimes breaks that silence to fuel his trades. This blends in nicely with his therapy and has every appearance to speed up the process considerably. You can imagine that for this fact alone, the Pokemon movement has my undying gratitude.

Money for Nothing

Some 375 years ago, the Netherlands (then the United Provinces) just recovered from a bout of intensive fighting with the Spaniards. It had a lively trade, resulting in the Golden Era. This boosted the cash income for the upper class so much, that it desperately sought outlets, burning in the pockets of rich people. My fellow countrymen therefore developed an appetite for the beauty of the tulip; a rare item in limited supply. The insatiable appetite for exotic varieties turned into the first real economic bubble, as middle class speculators joined the fray and short selling (ie, gambling on price increases with borrowed money, effectively incurring double risk) was introduced. The history of this mania is described eloquently in the book Tulipomania.

The appeal of easy money has always been big. It is what any casino thrives on, and what makes Pyramid Schemes so successful in preying on the naive and gullible. Ratio is thrown out of the window in pursuit of the big fish and the fact that so many people join in only reinforces the herd mind that the mass is on to something.

Some of the appeals to people's lust for easy money is cleverly guised, such as The 4-Hour Workweek describes. And appeal it does -- how could it not? Gaining big money while doing (almost) nothing is so enticing, so seducing that it triggers the same mental area as it does when participating in bubbles, lotteries, casino games and pyramid schemes.

The author of The 4-Hour Workweek is obviously a gifted person -- and well aware of this fact given how he flaunts his accomplishments! -- which is a hurdle, inhibiting mass adoption of what is described in the book. To be system beater like the author is, is a rare gift. The author goes forth to describe how he delegates work to others in such a way that he gets relieved of all toil, except a couple of hours to keep things moving.

The 4-Hour Workweek is a bit more subtle in its negative effects however, when compared to the afore-mentioned economic phenonema, and it is all about added value. What the author works on is a mix of differences in price (the same argument used for outsourcing), his own considerable insights, available talent abroad and a non-transparant market. As opposed to outsourcing, which requires lots of actual work, experience and management to succeed with, the author adds no further value to the process, so builds up no competitive advantage whatsoever.

Let us just fantasize for a moment that Timothy Ferris' approach is broadly adopted. We have all these entrepeneurs, managers and experts whose sole aim is to minimize their working hours as soon as possible. What added value does that person build? What added value does the company build? What added value does society as a whole build? You can delegate your work away as much as you like, but there is always a downside to every upside. In this case, the downside is the loss of skills and experience and henceforth any competitive advantage that may have been wielded. A downside that is felt all the more if well-packaged parasitism becomes the norm. Just ask yourself how easy it could be for the delegatee to run away with your business.

Structural shortcuts to success are only for a few lucky or brave souls, and are by their very nature not intended for the masses. Contrary to popular belief, the best way to achieve your goal is still to identify it, set the parameters and work hard to make it happen. Like Erasmus said, Qui vitat molam, vitat farinam, or He who runs away from the mill runs away from the meal. In short: No pain, no gain.

The Black Art of SEO

I have been dabbling in the superstitious side of IT recently; Search Engine Optimization, or SEO for short.

Google guards it search engine secret jealously so know one knows for sure what is hot and what is not. Like any half-decent government organization that desires a certain behaviour from its citizens, it is glad to tell people how they should behave, but refrains from telling the exact way in which the appraisal takes place. Better to let people act in the spirit of an idea, than to give them the exact parameters and inner workings, which tends to provoke abuse.

On the downside, and for this reason, the SEO market looks a lot like the paranormal community -- no one knows for sure what is happening, so anyone can claim to have divined The Algorithm (tm). Proof is sketchy and scientific evidence hard to come by, more so if the search engine's crawlers take some time to revisit your website, leaving a big gap between a change and its effect. What seems to be sure is that Google has succeeded in striking fear into the hearts of SEO experts. Ever since it degraded the ranking of sites meddling with link farms, people seem to be watching their steps more carefully, aiming to please, not to anger. What more could the Great G ask for?

Having said that, SEO has its own scale of near-certain knows to urban legends. On the near-certain side are the HTML tags on a page which are important for Google to determine what a page is about. Keywords with a higher importance are taken from the following tags:

• URL; give your pages names which are meaningful in relation to the content


• title; it is advised to stick the important keywords to the start of the sentence


• h1/h2/h3; in order of importance


• bold/strong; emphasis might denote importance of keyword


• image; the name of the file and especially the alt text


• link; the title of the link and the anchor text

As I am naturally disinclined to human-crawl a website looking for these tags, I hacked together a Java program based on HtmlUnit (my latest love!) that does the job for me. I give it a URL and it parses the site, following all links to pages on the same site. Of every page it checks the important HTML elements. The outcome is a report where you can see what keywords are being picked up by a crawler. It does not do SEO for you, but it certainly helps if you do not see the keywords that you would have expected in there.

If you want the app (disclaimer: no unit tests, all-round error checking, build etc in place), are interested in a report for your site or can add valuable SEO divining capacities to the tool, please drop me a line.

Why Gems is mere carbon and Maven is a schmuck

The first time you run Ruby gems to install components, you have to admit it all looks pretty slick. The application is set up and the developer goes to work, adds a gem here and there

Then the downside becomes painfully obvious.

All the components that were installed by the developer have to be installed by each and every other developer who wants to work on the application.

All the components have to be installed on all the servers where the application is to be deployed.

Truly, I am not kidding.

So for every component added to the application you will have to run this gauntlet. Since component changes are in flux and need to be dealt with as such, instability is bound to result.

To add insult to injury, the concept out-of-the-box further imperils Ruby's Cloud-ability, because it requires root access to deploy gems. As long as this issue is not addressed, Ruby is bound to miss out on the movement.

The Ruby community soon enough saw the downsides of the gems approach, but as solutions go, the proposed practice is lamentable. Gems need to be unpacked (similar to extracting content from a ZIP file) and added to a subfolder in the product. Then the application will have to be customized so that it reads the gems' content as part of the project structure.

I am sure the Ruby community would have much preferred a cleaner solution like Java offers. Instead they chose the enticing route of setting up their own disruptive technology. Too bad the disruption was for the user, not for competing technologies.

The Java community will not be left off the hook, however, having introduced and hailed its own piece of well-intended, but flawed reasoning and I will gladly go against the grain to make my case.

Maven has the concept of a repository where dependencies can be stored (usually JAR files). Some reasons are given on the Maven website why those dependencies must not be stored as part of the project; takes too much room, impedes checkouts and does not require to be versioned.

Can you believe that those are actually the real reasons for Maven's repositories? You better believe it, because there is not much other value to the concept, even downsides.

For one, there is the downside of having a compile-time dependency on external sites for which I wager most organizations have no Service Level Agreement in place. Unacceptable to any serious company. Though there are ways to bypass the problem by setting up a local repository, most of the time that is overkill, introducing more complexity than the value you get out of the deal.

And let's not forget the extra hassle you get when setting up your development project. Normally I select the JARs in the library subfolder of a project and be done with. Not so with a Maven repository. You will actually have to go through the POM and harvest the dependencies one by one.

So here is Maven with its fancy repository, saving you disk storage and checkout time. Would you consider that a good trade-off for an extra layer of complexity and more hassle?

If you build your application, always internalize your dependencies whenever you can. You keep your teammates, your systems operator and yourself much happier this way.

Predictably Irrational

If you are a programmer, you are bound to see the world and its inhabitants a bit different than most people, more inquisitive, not exactly more rational, but definitely searching for those building blocks. I used to think that was a handicap you just had to cope with, especially in social traffic, since discounting the richness that Humanity itself brings to the fore is usually frowned upon. That is, until I read Predictably Irrational by Dan Ariely. The author demonstrates that Man's mind might be easier to program than a Java Enterprise application, something which I wager will be agreed upon by NLP students out there.

The books shows us that:
- Man is great at comparing things; things which are great but cannot be compared are doomed to fade away
- Man chooses that which is Free; even when a rational human being would choose a more sensible option
- Man anchors new things at a fixed value; the new thing is valued and will forever after gravitate around that value
- Man does not mix social and monetary currency; if something is done on a social basis, monetary compensation devalues it
- Man is beyond reason in heat; despite what people think of themselves, they go much, much further when in a state of arousal
- Man is a procrastinator; if something can be delayed, it will be and sometimes an authority to prod and stir is a good thing
- Man values that which he owns; it will be traded more expensively than its actual worth
- Man does not burn boats; bet hedging is deeply ingrained in our psyche, even when it is not beneficial to do so
- Man lets expectation determine reality; that which is expected as an outcome shapes the actual outcome, not only in the mind, but in the body as well
- Man values that which is expensive; for a certain thing bestowing an effect, its effect will be greater when it is more expensive

Being practical, I think the following would be valuable for a company to adopt:
- on offering a product, make sure a comparable (albeit inferior) product is offered -- if it is a competing product, include it as part of your message
- include something nice for free
- if you want your company to be social, align it so all the way (and forget about demoralizing and demeaning reorganizations -- good companies do not reorganize, they prune ALL the time)
- do not make your product too cheap (every Marketing student will tell you this anyway)
- make it easy for your boat not to be burned (eg, customers who leave can keep their data in your system for free)
- challenge each other vigorously (ban out the procrastinator in yourself!)

A good read, definitely worth spending some time on if you are interested in what makes you tick.

Ruby, the End of the Honeymoon

At the risk of being threatened, ostracized or even lynched by an angry mob, I am boldly stating here that the Ruby movement is past its heyday.

"Why?", you rightly ask.

The signs were there all along. It started with the usual pioneering enthousiasm that comes with a new technology, and admittedly, I joined in the choir. Ruby, or more accurately, Ruby on Rails, sure played its part in upsetting the existing order, especially in the field of webclient development. The copy-cats soon followed, with Grails being the most honest about where it borrowed its idea from.

For some time now, I have noticed that quality Ruby people are hard to come by. For some time too, I noticed, that a number of Ruby pioneers start to lose interest and drift to other promising languages such as Scala, or even back into the fold of Java.

Ruby on Rails applications are slow. I have had the luxury to have a Ruby crack in my team who knows how to tune these little beasts and crack that whip. Grails applications, on the other hand, are not slow. They run directly on the JVM. An enviable position, that Ruby believers can only glare at without offering a solid alternative.

Aha, I hear you say, that's where JRuby, Ironruby and the likes come in. Now I would like you to heed this -- we are talking here about hand-crafted Ruby interpreters that make sure Ruby can be run on the JVM. Tools that have their own issues (we had a huge concurrency problem with an older JRuby version) and introduce one extra element of complexity into the Technology stack. Everything that people think up in the Ruby interpreter needs to be faithfully duplicated in JRuby, but then written up in Java. DRY? Not really. The notoriously sloppy documentation and audit trail that unfortunately appears to be a convention in the Ruby community only adds to the task. I have great respect for these brave developers, but less faith to build essential software on such a platform. And don't even get me started on how JRuby-hosted Ruby code is its own island (yes, you can ferry between the islands, but you will get seasick), contrary to Groovy and Scala applications which interact freely with Java code.

Finally, the commoditization of hosting web applications, especially by Google, pushes the knife home. Google's cheap hosting offer only allows you to run on the JVM, effectively adding a premium to your service if you want to run full-fledged Ruby on your own servers.

Ruby and Rails have brought a lot of good to software development, but alas, their time in the limelight has come to an end.

A Tribute to the Bug Hunt

There are few experiences in software development quite as humbling and as satisfying both as the Bug Hunt. The search for an elusive problem that causes systems to crash, data to be lost and bits to be unwantingly inverted is a wild chase where one is peeling off several layers of proximate causes in order to get to the ultimate one.

The interesting aspect is how the Hunt is often opened by having a preconception of what the problem is and where it resides. Often a hunt ends there. The interesting ones are those where the preconceptions get shattered several times as they are revealed as mere proximate causes or worse, false leads.

The humbling part, for me, consists of how I sometimes have an idee fixe about what the cause is and how that assumption turns out to be untrue. I daresay that this aspect made me more conscious in my interactions with other people and prevent me from jumping to conclusions.

There are many out there who are happy to have found the proximate cause and stop there. The symptom is solved. For example, a warning light flickers in an irritating way, so take out that light. Symptom solution is in my experience also typical when an electrician comes to repair something in my house; they observe the symptom, declare that the cause and proceed to fix it. They don't ask that critical question: why does this light flicker?

It amazes me to see this is often the prevailing mood in software development as well. If you are in a position to question someone who has just solved a problem, ask them why the problem occured and keep on asking why until you have satisfyingly heard the ultimate cause. If this is not business-as-usual, I predict you will find the level of understanding to be lacking.

The lack of understanding will ultimately come back and bite you when your dirty-fixed applications will be plagued by problems that start cooperating in ways unimagined to cause serious mischief, multiple root causes making the hunt even harder. Your application is more likely to reach the end of its life prematurely.

The Bug Hunt consists of five parts:
- arrange a problem description
- reproduce the problem in a separate environment
- determine the root cause by isolating the problem
- propose a concept solution
- implement the fix

The only part that should be subjected to economical choice is the fourth one, ie the type of solution. Here you can choose a proper solution, a quick-fix or no solution at all as the situation allows. Do not save costs on the hunt itself! You will be sorry that you did.

The end result of the hunt should be a narrative that explains exactly what has happened. If you are using an issue tracking system, demand that the hunt is documented meticulously, so knowledge is shared and valuable information on the patient preserved.

Doing all this right, will keep knowledge about your application with your team on a high level and the application itself is kept in good shape. You are better positioned to scale up, address new problems, refactor old code and add new functionality. The Bug Hunters are the heroes of your team; honor them!

Remember, the most powerful weapon of the Bug Hunter is that one little word: why.

Good Hunting!

Of Hawks and Doves

The Prisoner's Dilemma is a well known game. The basis is very simple: two prisoners are being questioned for a crime and have the choice to either COOPERATE with the other prisoner, or to DEFECT. If they both cooperate, they both get the REWARD (3 points). If they both defect, they both get the PUNISHMENT (1 point). The juicy part is if one defects, while the other cooperates. In this case the defectors gets the TEMPTATION (5 points), while the cooperator gets the SUCKER'S PAYOFF (0 points).

An actor participating in the Prisoner's Dilemma we will call an Agent:

  public abstract class Agent {
    public final int TEMPTATION    = 5;
    public final int REWARD      = 3;
    public final int PUNISHMENT    = 1;
    public final int SUCKERS_PAYOFF  = 0;
  
    private String name;
    private int score = 0;
    
    public Agent(final String name) { this.name = name; }
    public abstract boolean cooperate(final Agent other);
    public String getName() { return this.name; }
    public void remember(Agent other, boolean cooperates) {
      // ignore
    }
    public int dealWith(Agent other) {
      boolean iCooperate = this.cooperate(other);
      boolean otherCooperates = other.cooperate(this);
      int score = iCooperate == otherCooperates ? 
        (iCooperate ? REWARD : PUNISHMENT) : 
        (iCooperate ? SUCKERS_PAYOFF : TEMPTATION);
      increaseScore(score);
      remember(other, otherCooperates);
      return score;
    }
    public void increaseScore(final int score) { this.score += score;}
    public int getScore() { return this.score; }
    public void resetScore() { this.score = 0; }
  }
  

The first type of Agent will be the Dove. The Dove will always cooperate, hoping they will be reciprocated:

  public class Dove extends Agent {
    public Dove(final String name) { super(name); }
    public boolean cooperate(final Agent other) { 
      return true; 
    }
  }
  

The second type of Agent is the Hawk. Hawks always aims for the win-lose, hoping that the other is more cooperative:

  public class Hawk extends Agent {
    public Hawk(final String name) { super(name); }
    public boolean cooperate(final Agent other) { 
      return false; 
    }
  }
  

Pitch these two types of agents against each other and it will not be a surprise that the Hawk in every encounter scores the Temptation, while the Dove gets the Sucker's Payoff.

This is proven in a test setup, simulating the Arena of Evolution. In the setup the worst scoring Agent gets eliminated randomly spawning a new Agent in its place. The type of the new Agent is randomly determined and based on the number of Agents of a certain type. For example, if there are 12 Doves and 8 Hawks, the new Agent has a 60% chance of becoming a Dove and a 40% chance of becoming a Hawk.

You can let the agents interact in the following way:

  agentOne.dealWith(agentTwo);
  agentTwo.dealWith(agentOne);
  

The Doves simply get supplanted.

It was long thought that the Hawk had the dominant strategy. But luckily for those who fear such a bleak outcome, a new type reversed the social order from a Malthusian one to a more friendly one -- enter Tit-for-Tat:

  import java.util.*;
  
  public class TitForTat extends Agent {
    private Map iKnowWhatYouDidLastTime = new TreeMap();
    public TitForTat(final String name) { super(name); }
    public boolean cooperate(final Agent other) { 
      Boolean cooperate = iKnowWhatYouDidLastTime.get(other.getName());
      return cooperate == null ? true : cooperate;
    }
    public void remember(Agent other, boolean cooperates) {
      iKnowWhatYouDidLastTime.put(other.getName(), cooperates);
    }
  }
  

Tit-for-Tat will remember what an Agent did the last time it was encountered. If it cooperated, so will Tit-for-Tat. If it defected, Tit-for-Tat will pay in kind and defect also. But it keeps a positive attitude to the social traffic by always cooperating if it does not know whom it is dealing with.

What happens now depends a lot on the number of interactions that take place and if there is a large enough population of Doves/Tit-for-Tats so that the good guys can deal positively with each other. If both preconditions are in place, the Hawks are supplanted by Tit-for-Tat. Conversely, if Tit-for-Tat is pitched in a Dove population, both will flourish.

If you are interested in this type of material, I can heartily recommend The Origins of Virtue by Matt Ridley.

On a closing note, there is a type following Tit-for-Tat, which has a bit of a mean streak (its name is "Pavlov"), trying to take advantage of Doves (ie, "suckers") while being on its guard against Tit-for-Tat. This type has quite a ball in a mixed Dove/Tit-for-Tat population.

Doves get pissed on by Pavlov, while the Hawks take a piss on everyone -- the prophetic words uttered in Team America never sounded more true:

"We're dicks! We're reckless, arrogant, stupid dicks. And the Film Actors Guild are pussies. And Kim Jong Il is an asshole. Pussies don't like dicks, because pussies get fucked by dicks. But dicks also fuck assholes: assholes that just want to shit on everything. Pussies may think they can deal with assholes their way. But the only thing that can fuck an asshole is a dick, with some balls. The problem with dicks is: they fuck too much or fuck when it isn't appropriate - and it takes a pussy to show them that. But sometimes, pussies can be so full of shit that they become assholes themselves... because pussies are an inch and half away from ass holes. I don't know much about this crazy, crazy world, but I do know this: If you don't let us fuck this asshole, we're going to have our dicks and pussies all covered in shit!"

Quod Erad Demonstratum

Wisdom of Understanding

I saw this little wisdom gem the other day on a Yogi-Tea label

To Learn, read
To Know, write
To Master, teach

I have found this to be very true. Reading about topics gains you facts. Writing about them lets you explore ideas, opinions, what-ifs, strategies, pros/cons. When teaching you want to know about the nooks and crannies, and all the little details.

Gallipoli and the Course of History

Gallipoli is a small peninsula south-west of Istanbul, in modern day Turkey. During the first world war, when the Ottoman Empire decided to side with the Germans and the Austrian-Hungarians against the rest of the world, the British had the plan to blast their way through a range of forts that protect the narrow waterway all the way up to the Black Sea. Churchill, the well known World War II politician, was the main architect of the plan. The plan failed, because too many allied ships were destroyed, mainly by mines.

A couple of months later, the allies invaded the peninsula with hundred thousands of men, with the aim to conquer the forts by land and thus clear a path for the Navy to sail to Constantinople (now called Istanbul) in order to force their foes out of the war. The invasion met with stiff resistance from the Turkish defenders -- much to the dismay of the invaders! The allied forces were unable to advance even beyond the beaches and things soon bogged down into a trench war. One year later, the enterprise was abandoned, leaving over 200,000 allied and 300,000 turkish dead, mostly young men in the prime of their life.

Now what makes this a remarkable campaign is how it connects a number of strands in history.

For one, it was the first and only large-scale sea invasion against defended positions before D-Day. A lot of the things that were done right on D-Day, were learned from the failed Gallipoli campaign. The allies learned the value of air and naval support and they improved upon the whole design of the armored landing boat, which played a crucial role in Normandy. Furthermore, tactics were aimed at making progress land inwards as soon as possible, something which was not done in Gallipoli.

Just imagine what would have happened if the allies failed in their landing at the beaches of Normandy. It is accepted wisdom that the Soviet Union was on a roll and would have defeated the Germans eventually. The true value of D-Day, therefore, was not to ascertain the defeat of Germany, but to prevent the Soviets from conquering all of Western Europe. Just imagine France, the Netherlands, Belgium, Luxemburg, Denmark and Western Germany, all under the oppressive boot of Stalin. What turn would the Cold War have taken? Would the USA still have won it?

Second, the reputation of Churcill was tarnished after Gallipoli. He was officially viewed as the instigator of the whole failed enterprise. Even though he was an advocate of initially pushing harder up the Dardanelles and forcing the straits (which would quite possibly have been successful), his more powerful compatriots thought otherwise and blocked that idea. And even though he was committed to sending more troops to Gallipoli, his compatriots once again blocked this. It is quite likely that if Churchill would have gotten his way, the campaign would have taken a different turn. But the First Lord of the Admiraly was young and without much political clout in that day and age.

Just imagine what would have happened if Churchill succeeded at Gallipoli and would have forced Turkey out of the War. Much less opportunity would have been had by the Turks to start their own new pan-Arab brand of Jihadism of which we feel the effects even today. Also, Churchill could have been a more formidable political force during the Rise of Hitler, possibly countering Chamberlain. Churchill would not have caved in to Hitler's demands -- he was a hawk. Germany was not as ready for war as mythology would have it. An early attack by the allied powers against the German infrastructure, especially during the Polish campaign, could have ended the war very early.

Third, a young man named Kemal was the Hero of Gallipoli. He showed remarkable courage under fire and was chiefly responsible for preventing the allies from making any progress. Kemal was a political enemy of the rather incompentent, but still powerful Enver. Riding the waves of his success, he prevailed and came to be known as Ataturk, the father of modern day secular Turkey.

Just imagine that Kemal did not have this success to feed his ambitions and the modern day Turkey project started by the Young Turks would have been a failure. Turkey has a strong economy and military nowadays. What if it did not reform?

Fourth, an Australian reporter made his name as a critic of the Gallipoli campaign. His famous "Gallipoli Letter", detailing the failure of the campaign, had a huge impact and was one of the main reasons why the campaign was cancelled. The name of this reporter was Keith Murdoch. He sowed the seeds of a media empire. His son, Rupert, took over from him and carried the empire to greatness. One can only hold in awe what the Murdochs achieved.

Just imagine what would have happened if Keith did not get his shot at fame by denouncing the Gallipoli campaign. Would he have been able to engineer a media empire?

So many questions that will never be answered. Nevertheless, ostensibly just a side show, Gallipoli may have had a greater impact on history than meets the eye.

Automating automation

At Independent IP we have made automation a mantra. Wherever we can, we automate manual processes so we can lower our costs, run the show with as few people as possible and ultimately have our customers benefit from all this by paying a competitive fee for a better platform.

Despite this, there are always a couple of stepping stones before reaching the ultimate goal -- an automated process.

DOING; in the first phase, the lifecycle of a process starts out as an activity not formally defined that someone is repeatedly doing. At this point in time, we are not or only partially aware of it.

IDENTIFYING; in the second phase, we are aware of the activity and it will be described in a document or flow diagram. At this point in time we are aware of the costs, both current and when scaling up. This is also the moment that we draft wishes to have the process automated.

STANDARDIZING; the third phase is when the process can be performed by other people as well. Standardization has kicked in.

AUTOMATING; finally, the whole process is automated.

Starting a new process immediately in phase 4 usually never works. This is for various reasons:
- the process has not had the chance to gain real business value because it is still a product of the mind
- the process has not been repeatedly tried and improved upon, so it may not be the best way to do things
- the process does not yet exist, so we do not know very well what we try to accomplish

Move slowly and do not rush through the phases. Respect the ripening of processes to yield the best harvest.

In the end it is about automating automation and doing that in the most effective way.

Truth hurts

"Ignorance is bliss", as Cypher so succinctly put it in the Matrix, after he describes how Knowing spoils his dinners. He was probably inspired by Thomas Gray's "Where ignorance is bliss, 'Tis folly to be wise". Not knowing Cypher lived in an artificial world would have preserved the joy of eating food. Knowing that he did, spoiled it all. He bemoans the fact that he Knows.

The Search for Self is a murky path, full of unpleasantness. You will find things about yourself that you dislike, weaknesses and imperfections, things you did or were done to you and do not want to be reminded of, or choices you made that had unhappy outcomes. It might make you feel depressed, angry or sorrowful, or perhaps even spirals you down into self-destructive behaviour. Compare that with a lot of people who have a skewed Sense of Self, yet are perfectly happy in their bubbles and even perform well in society. Why then do we still doggedly pursue this knowledge? What is so great about it?

There are a couple of reasons. Knowing thyself...
... paves the way for improving yourself
... helps you to prevent disadvantageous situations
... helps you to determine a career path
... makes you stronger, because no one can tell you anything about you that you did not already know
... helps to manage expectations about what you can and cannot do
... allows you to better understand other people

On a company-level, self knowledge becomes even more critical. The temptation to see life through rose-tinted glasses is enormous. Optimism is often seen as the driving force for a venture. I say here and now that optimism is overrated and even dangerous -- a culture of optimism disallows bad news to be merged into painting the big picture, therefore always putting decision makers on the back foot.

Optimists often have short-term success, but know not how to deal with setbacks. They disbelieve, ignore, deny, and do basically anything to keep their world view intact.

Any company is much better served by a good sense of what it can and, more importantly, what it cannot do. It can set ambitious, yet attainable goals for itself, that boost morale in a way that pep talks and speeches could never accomplish. It sets its goal and then does it.

On a society level, the refusal to Explore Self, or discuss that what happens is well known: Political Correctness. An affliction that seeks to, as biologist Matt Ridley puts it, "argue from ought to is". Political Correctness is carried by none less than all those sweet and well-intending people who want the world to be a good place.

To prevent all that nastiness from entering decision making, it is ignored. Euphemisms, censorship, and looking the other way are all manifestations of Political Correctness. The consequences are that a society is unable to deal with wrongs that it even fears to identify.

So instead, we retreat inside a safe bubble that tells us life is good, people are good and we are good. No pain, just bliss.

And that is just the truth of it.