Skip to main content

Release late, release rarely

Posted by mkarg on December 29, 2010 at 10:58 AM PST

Meanwhile I am looking back to more than 25 years of programming, and more than a decade I spent in a very sensible area where quality (in the sense of zero failures) plays a big role. So call me "sensible" for quality. For long years "we" (i. e. developers) had hard work to do using simple command lines tools like vi etc., but meanwhile there are great, even free, tools making our lives much easier. So one should think that the freed time was spent into quality. The reverse is the case. The more I look at professional software products, the more I see, sorry to be rude, simply: crap!

The question is: Why is that? Having some insight into several bigger and smaller stakeholders, including my own participations to some Open Source projects lead by my self, Sun, Oracle, Bull, Microsoft, and some more, and looking at the work we are doing in my company and its affiliates and subsidiaries, gives me some feeling where the root of the evil is located: In the organization of the project.

Gee! I can tell you how I hated to attend to those boring, mandatory "business organization" classes at college back in the 90ies. But hey, looking back from now I need to say: Those profs where just right! Fullstop. Quality is directly related to your project's organization (which is the way you organize people's way of working, not the way you imagine the resulting artefacts). If you don't organize for quality, you won't get quality. Thinking you already organized quality? Let's see...: You can write thousands of tests. You can do pair-programming. You can do Scrum. You can to Kanban. You won't get quality from that. Throw all that away. It has not at least to do with quality you'll actually get in the end. You only will get quality in the end, if you actively plan quality from the start. That means, you have to enact measures which will result in quality getting actively produced while designing. That's the reason why systems like ISO 9000 etc. enforcing FMEA (Failure Mode and Effects Analysis) and other methodologies the be applies at start and regularly whilst the complete project instead of enforcing just "testing". Not just doing TDD while writing code. That's far too late. Not while testing (that's just to find bugs you already did). Not whilst the beta phase (that's to find bugs you where too lazy to find on your own). Not by fixing reported bugs (that's just to wipe up the bugs your beta testers had been too lazy to report and which will break your neck otherwise). Quality starts at the very beginning of everything. And it doesn't start in your typing hands. It starts in your brain.

To gain quality, you first need to accept that a program is not "good" if it hardly does what it shall do. A program is "good" if you cannot crash it even if you want to. Let's make an example: I just downloaded latest JDK, GlassFish, Eclipse (including GlassFish Plugin). Within one hour I managed to screw it completely. All I did was using it. I not at all had in mind to break it. It just happened. I just used the wizards to create an EAR, EJB incl. EJB-Client, APP Client, deployed / redeployed few times. Crash. Bang. Boom. Now it's so screwed that neither Eclipse will be able to compile a freshly set up, empty project anymore, nor GlassFish will deploy X.EAR since it cannot find Y.EAR (which I undeployed and then deleted the projects, but it seems to have more than one redundant information about that which I cannot find). I tried to clean up, but gave up after another hour. And no, again, I had not done anything to crash it by intention. Is that "good" software? Well. Think of people wanting to crash it by intention (like hackers and unsatisfied employees) and then answer then.  Don't get me wrong, I appreciate the work the GlassFish team does. I just don't think that form a quality perspective it would be "good" software. It's just "cheap" software from a big stakeholder. But "good" is far different (and would be much more expensive).

So there we are at the key point. Quality is expensive. Yes, I know, everybody goes round and round telling that quality would spare money. What a hoax. Quality does not spare money. Quality costs money. Lots of money. Everybody in the CAQ crowd ("Computer Aided Quality Assurance") talks about quality in the end will provide increased revenue. That is wrong. Quality will typically lower revenue - at least as long as you are not obligated to correct failures on your own costs (which is only the case in some countries of the world, so one can guess what the most global companies will act like). In fact, even in countries where you are obligated to correct failures on you own (like Germany), big stakeholders just sit back and wait for the one customer that dares to complain. Will he get a fix for free? Nope. I tried for months to get one from Sun Microsystems, Microsoft, Hewlett-Packerd, and others. And what did I get? If I would pay the fix, I would get it. So why should any of those vendors ever care for quality in the end, if they get paid to fix their own faults? In fact, they all tell about quality, but regarding to organizing for quality, they don't care in the end. Maybe the boss cares, but the lots of managers do not. They care for short term revenue which is better the worse the product is. Sad, but true. BTW, the funny thing is, that companies that everybody knows for horrible bugs, improved quality. Seems they meanwhile understood the benefit of long term revenue. But exactly that companies that provided quality for long time, now producing crap more and more. Strange, isn't it?

The reason is simple and let's tell it clearly: Off-Shoring. I really don't want to blame globalization, but in fact, it is nearly impossible to organize people located in different continents to work in a way that will result in something that is worth being called "quality". How should that ever work? Sorry to say that, but Sun, Siemens and others did not move to India because American or German engineers where too dumb. They were too expensive. It's not that American or German engineers where too expensive. They just wanted to make even more money. Certainly everybody told them that quality will be affected. But who cares if customers have no choice? From a colleague I know for sure that his company (which is a global player) for more than five years only got pure crap from Bangalore. And nobody cared. Not even the customers. Because they had no choice. Sink or swim. Example: I wanted to buy a car radio that is able to switch from album to album while showing the cover ("Cover Flow"). Well, first I thought my local dealer is dumb when he told me that even with a 350 US$ specimen from Pioneer, Alpine or Kenwood, I have to go through plain text menus to select the album. No, this not a hoax. This was christmas 2010, and I meanwhile got letters of confirmation from those vendors that the dealer is not dumb: You cannot switch to the next album with a single click. Definitively. That is not quality. That is crap by design. Ten years back the user experience was in the middle of the car radio development. Ten years back the development was located in Germany and the USA. Today the development is in India and China and I doubt that the majority of that engineers actually could afford to buy their own product (or just hardly). Who goes through menus while driving? They just didn't think. And why? Because their target is to make money. Not to provide quality. Just one example. BTW, the excuse of the vendors was not "we are too dumb" but it was "car radio development these days is bound to some monetary boundaries". We know what that means. If they would pay more for their engineers, they could just move back to the USA or Germany. "But it was a strategic decision and will not get changed, even if it will not result in any revenue at all." (citing a development engineer).

So what to do? Go back to the roots! First, you have to reduce product development to a few core people, located in a first world country. That people have to be experts. They have to be well paid (the major reason why lots of Sun people quit with Oracle was about reduced income, BTW). Those people have to have enough time and equipment and silence. Don't bother them with revenue, release dates, and so on. Just wait until the software is done. No need for thousands of engineers in India. No need for "taming the masses" things like XP, Scrum and Kanban. Just let them sit in a quiet chamber and wait and do not disturb. And, do neither ask your channel partners nor customers what they want to get. Ask professors what the future looks like and do that. Don't go for hypes. Just provide the best possible quality. That must be your target, independent of the actual product. And: Release Late, Release Rarely. No need for another release every month. A release once a year is well enough. Don't beat around the bush on lots of conferences. No need for "community". Just work and present it when it's done. That's the way to get quality in the end. Everything else just is marketing show.

I know that 99% will hate me now for saying this simple truth. But that won't change the facts. Quality is not provided by asking the masses. It is provided by few experts having time and peace.



An overview of all my publications, presentations, code snippets or complete products can be found on my personal web site Head Crashing Informatics.



Release late, release rarely

I have to completely disagree

1) Low cost is primary problem of low quality, not outsourcing. If you pay peanuts, you get monkeys everywhere. And salary for skilled developer is very similar in Prague, Dublin, Dubai or Bengalore.

2) Stupidity is not linked with language. My coworker is English, java senior programmer, yet he does not know difference between TreeMap and HashMap.

3) Last paragraph about 'well payed experts working in silence and without explanations' is just reason why we got into this outsourcing shit.

4) My advice for high quality is to make dev process as much as transparent. Not accepting fix for 2 years because it is against policy is just plain wrong.

5) There is no link between release period and quality. Many app servers have long period and are way worse than Glassfish

Release late, release rarely

1) Paying peanuts is the core idea of off-shoring. You just do not get masses of "monkeys" (as you named it - I never called anybody "monkey" as I do not think of anybody to be a "monkey") among western engineers, even if you wanted to, for the price you get it in India. And Prague, Dublin, Dubai and Bangalore do by far not have the same salary as Santa Clara and Berlin. Even within the EC the salary is a reason to e. g. replace one German engineer by two to three Romanian engineers. I didn't talk about replacing Prague by Bangalore, but Berlin by Bangalore.
2) I never linked stupidity with language. Please do not make assumption about my intentions, but just read what I actually wrote. I did neither talk about stupidity, nor about mother's tongue. I talked about product quality and environmental / social aspects in particular countries. This is a huge difference. And, bad quality starts at the analysis level, not late at choosing a particular implementation of Collection. Thinking that quality is a function of good or bad implementation is part of the problem, as the implementation level only marginally affects the overall product quality. Much more influence on product quality has not-understanding of customer needs in the analysis phase, and applying anti-patterns in the design phase. You can apply best implementation and production practice to both, a Mercedes Benz and a Skoda. But it is obvious a difference in the actual quality level. Quality is not only about things not breaking apart due to bad implementation, but more about not applying sophisticated analysis and design (i. e. to plan quality vs. to produce quality). You can build a Mercedes Benz in India. But you cannot engineer it outside of Daimler. That is my thesis, nothing else, and that is, why Daimler does off-shore production but not off-shore core engineering.
3) "Outsourcing shit" (as you named it - BTW, did you notice that I did not use any offensive words by intention as I do not criticise people but just companies?) started not because of well paid experts but because of a simple calculation: Replacing few well paid western experts by lots of far less paid Indian engineers spares 30% of development costs, but reduces quality at the same level. As customers do accept this 30% reduced quality in favor of a 10% reduced end price, there is a benefit left over of 20% for the shareholders. It has absolutely nothing to do with too high wages in the west, but with too low wages in the east. If that wouldn't be true, why should any company shut down greatly working western offices and start again from scratch in India or China? There was nothing wrong in the west, so there is no reason to do that other than make more money.
4) Accepting bugs for two years is part of the plan to save money and such must be part of the policy (otherwise off-shoring would not have the expected benefit of making lots of money -- you could just stay where you are and optimize things there if it would be just about making some more money). Replacing experienced engineers by several cheaper (since more unexperienced) ones results in longer time to fix bugs, resulting in higher costs to fix them. So you just don't even plan to fix them, or you just have no time to fix them, as the unexperienced masses just need lots of time to learn things that the experienced engineers just knew in their first semester at college. You could add even more unexperienced, hence cheap, developers to compensate the delay, but this would eat up the benefit you plan to get from off-shoring. So you don't even think about that. That's the logic of off-shoring. Making money by reducing quality.
5) There actually is a rather simple link between period of quality: The less time you have to think, the less is the resulting quality (even experts need time to think). But this does not mean that in turn waiting endless will improve quality in a linear way, as there is a natural break even point set by the quality your people are able to deliver in absolute numbers (non-experts cannot deliver the same quality than experts, even if you let them think for years).

Release late, release rarely

What a great article. I couldn't agree more (with its basic premise; not so sure about your rather dubious distinction between "first-world" engineers and "third-world" engineers).
I think the root problem is that people write software that people are going to buy. And the market has spoken: the market demands crap, produced quickly. Whole methodologies--after the fact--have sprung up to justify this (let us not forget that the payroll project that gave birth to XP was cancelled for delivering junk over budget). These methodologies excel at producing the lowest quality crap at the highest possible price--that, after all, is the defining tenet of capitalism, or we'd all be out of jobs.
What surprises me is that more open source products--where presumably time and money are essentially of no concern--are not high-quality. You would think that free of market pressures people would be able to put together high-quality software that could then be used as templates for designing and building the high-quality for-profit software that puts food on most of our tables. And there are some cases, to be fair, where this is true. Glassfish, much as I love them, is not one of those cases; your experience with "just using it" and having it explode in all sorts of semi-persistent ways echoes my own, and that of my team. JBoss is even worse, by whole orders of magnitude. And yet for platforms and foundational infrastructure--there really isn't anything better.
What we've got is atrocious crap (the applications) sitting on top of less egregious crap (the infrastructure) sitting on top of pretty decent stuff (written 30 years ago). There's probably a lesson in there somewhere.

Release late, release rarely

Unfortunately I had no more time to go into more details when writing the article. The third world argument bases on the experience of a friend of mine traveling Bangalore once a year for a global player. He noticed that due to the environmental and cultural reality in India the average software engineer is just not comparable to the average German or US native engineer, e. g. because it was typical for our generation to learn programming with the age of 10 of 12 on a commodore 64 system and doing that the average western engineer imbibed the logic of informatics from his infancy, while the average third world engineer had not seen a computer before his 20th birthday due to the fact of given birth in the third world. This is not to blame the single engineer or the continent he lives in, but this is to blame decades if mislead foreign aid by the western governments, which targeted more in feeding poor people (and such keep them poor still) instead of actual nation building (and such turn agrarian states into industrial states). One typically needs 20+ years of experience with product development until he understands technology intrinsicly, what is not a big problem for a western native, but what is just impossible to a third world native due to the fact that 20 years ago china and india had not been that industrial states that they are today. Another issue I did not cover clearly is that the quality problem is not just a software industry problem, but it actually covers every product I can remember, even the movies and music industries, nutrition, healthcare, and so on. The reason is, as you correctly say, the masses want crap, so they get crap. Sad, but true. But what bothers me most is that I am willing to pay lots of money to not get crap, but you hardly won't find a vendor or dealer that is able to sell something that is not crap as virtuall all vendors only produce crap anymore.

Release late, release rarely

Some points that you make about the importance of experts are valid. However, I dont agree with some of the other stuff. If you outsourced work to a bunch of morons, you'll get crap, that's obvious. I'm disappointed that you converted this into a sweeping generalization about the superiority of western engineers, etc.

Release late, release rarely

Sorry to tell that, but my own experience, and that of all my peers I discussed this with, actually was that the reason for bad quality was that outsourcing always resulted in hiring lots of rather unqualified and chaotic people. I'd reall like to read a success story where a big company moved development from western countries to third world countries and quality did not get lowered measurably. Unfortunately till today no such success story was published. Again, it's not to blame third world engineers, but it's to blame the system. Outsourcing is not done because third world can do something western engineers could not do, but it is solely done to lower development costs. And it is clear why those costs are lower: Due to deficiencies in the overal living standards, social standards, school systems, etc. If India would have the same standards than the US or the EC, then there wouldn't be quality problems -- but there wouldn't be a development costs benefit then, and such, no more outsourcing. Sad but true, but a simple calculation.

Release late, release rarely

Maybe you need to take a look at the stats for engineers & scientists (from India/China) , employed by some of the top technological companies in the fields of Software , Engineering in the United states.

Release late, release rarely

My blog entry was not talking about single, top people having left India to work in the US (I strongly believe such engineers will excel in a US-located team just as western engineers could) but about the sheer masses hired for offshoring by globalized stkaholders to work in India. I'm not talking about races, but solely about environments and mass effects. There is a difference in quality in the end if you have a small, excel team in the western world compared to a team made up of sheer masses in the so-called third world. Again, there was never a company having said "Since we closed our engineering in the US and moved it to Bangalore, our quality was raised". They only talk about development costs. And these savings are not for free.