Posted 4:03 PM by Floyd

Evan Kirchhoff liked my essay on outsourcing, and wrote a post in which he explained why he said that "all currently lucrative programming skills will collapse to zero value over the next 5-20 years". Evan predicts a future in which

... a working American programmer in 2020 will be producing something equivalent to the output of between 10 and 1000 current programmers (even if the number of programmers actually increases overall). Hence, lucrative 2003 skills like being a really good C++ or Java programmer will have gradually fallen to a small fraction of their current value, or effectively zero.

I think current programming skills will vanish by 2020. In the world of 2010, I expect us to be at a midpoint where 2003-style programming still exists, but pays somewhat less than being a plumber, and hence is largely performed in lower-cost countries.

As Clint Eastwood once said, in a slightly different context, "That's not gonna happen." I expect to see many strange events in the year 2020 -- I can imagine lecturing my teenage son, "You can't leave the house with your penis hanging out of your pants," and him replying, "But Dad, all the kids are doing it." Certainly the specifics of software development will change; we won't be programming in Flash or Java or C#.

Evan's vision of the future is a lovely one. But the current state of software development is far too primitive to allow such improvement to occur in the next two decades.

Computer science is in the same state today as chemistry was three centuries past. In the year 1700 Georg Stahl theorized that combustion was akin to rusting, and that both phenomena occurred when the fuel or metal being consumed gave up a substance called phlogiston to the atmosphere. An obvious objection to this theory was that while wood lost weight when it "released phlogiston," iron actually gained weight when rusting.

What was the reaction of contemporary chemists to this contradiction? They did not care. Modern-day software development is in the same sort of pre-scientific state. There are thousands of project managers happily planning traditional schedules for their development process, even though this process does not work and has never worked. A manager who enters sub-tasks into Microsoft Project is a respectable-looking fraud, just as Isaac Newton was when he wasted his time trying to transmute things into gold. It looks scientific when there is a precise-looking document that states that Developer Joe Blow will work on cache management from Feb 5 to Feb 19, 2004. In real life Joe Blow will take until March to complete the task, or will think that he is done but will have to come back and redo it a month later, or will spend February trying to determine what on earth is wrong with his stored procedures.

For that matter, there are manager who happily deal with a late project by adding more developers to it. This mistake was first criticized by Brooks in The Mythical Man-Month -- forty years ago!

Evan criticizes Extreme Programming because one of its tenets is that developers work in pairs when they write code. I am neither a proponent nor a critic of XP, which I assume is popular with the geek community because its project management philosophy is a complete repudiation of traditional software development practices. I just think it's hard to imagine how Evan and I could have a meaningful discussion about pair programming efficiency, given as how neither one of us has any metric for measuring the effectiveness of a programmer!

One reason that software development is in such a mess is that software becomes obsolete every five to twenty years as capabilities of computer hardware expand exponentially. Suppose that early factories had casted their output using a new and revolutionary metal every decade: In 1800, bronze; in 1810, iron; in 1820, stainless steel; in 1830, titanium; in 1840, unobtainium. It's likely that any progress in manufacturing technology would have been flummoxed as factory owners rushed to destroy and rebuild their machinery to handle the capability of each new material.

A little background to my pessimism is in order. I work in enterprise software development. In my field, we sell software (usually written in Java or C#) to large corporations to be used by thousands of their employees. Enterprise software implements an important business function (training management, product design, supply chain management), and often replaces custom software developed by the customer in-house.

Because the customer is paying a lot of money for the software, and because they have probably tried to write the software themselves, they demand specific custom funtionality. No one could possibly plan for all these features in advance, so no one is able to sell an existing off-the-shelf product. These missing features are handled by a mixture of the following:

• Selling a product which is currently in development, and adding the features as requirements to that product.
  
• Customizing a product after it has been developed. This is often done not by the people who developed the product, but by people at the customer site -- who usually called "Professional Services".
  
• Agreeing that a feature is not supported, or is supported only in a circuitous manner.
  

As a result of these constraints, enterprise software is the worst written software in the computer industry. This is not because the developers are bozos (after all, I am such a developer and stop snickering in the back row). Nor is it because the enterprise software vendors are dishonest; customers know that they are always, to some extent, purchasing vaporware. Poor software quality is a result of the conditions I have mentioned, and the fact that many enterprise products have had a short lifetime, and are constantly being reegineered on new technology.

Another factor that leads to poor quality is that most software developers have little or no intuition as to how some complex business function such as training management or supply chain management should behave. Software developers are pretty simple at heart; they are interested in programming languages, and development tools, and games. Intuition is not that useful anyway, given that the features one is developing are not a rational expression of what the program should contain, but rather the result of a tug-of-war between the customer, the salesmen, product marketing (who plans features for products), product management, and the developers.

Now Evan is a web site developer, and posts a lot about computer games. As I already stated, it's a lot easier to know what a game should do in some particular situation than to know what, say, a contract management tool should do. Another huge difference between gaming and enterprise software can only be expressed through profanity: There is a fuck of a lot of money in computer games. The yearly sales of one title, my beloved Civilization, probably equals the revenue of all the competitors in learning management, my current employment venue. So games have better functionality, and higher quality, than other kinds of software.

In 1995 I briefly worked for a company that produced medical imaging software, to allow doctors to view X-rays, CAT scans, etc. I remember taking a break with a co-worker to a mini-golf establishment. We played some video games, and I was struck by how much better the graphics in these games were compared to our products -- built to save peoples' lives, for God's sake!

There are some tools that improve developer productivity by allowing people to leverage the preexisting work of others. Unfortunately the scope of these tools is limited. The really successful developer tools include compilers, debuggers, standard libraries, relational databases, graphical libraries, and optimization engines. One reason that Java is so popular is that from the outset, Java included not just the core language, but also a set of libraries to manipulate data structures, access relational databases, etc. This standardization was completely absent in C++. (Java is a fine language for server-side development, but is rather lacking when used to develop UIs, hence Evan's criticism.)

Many other tools exist, but they are in a state of infancy. In my current project we use a third-party open-source object-relational package to handle "persistence" -- the software loads database data as Java objects, and if you change the objects, makes corresponding changes in the database. Using this package saved us work, but also caused other, additional tasks. We needed to investigate, and implement, our own code to handle some of our strange cases that didn't map well to what the object-relational engine expected.

This is not the fault of the object-relational engine developers; they did a pretty good job of placing callbacks and other hooks into their software for the weird stuff that we were doing. The problem was that the object-relational people had their mental map of how their software should be used, and we had our mental map of what their software was and what it was doing, and it took us several months of development time to reconcile the two.

Using the object-relational software probably increased our productivity by 25-50%. Now it took 5-10 man-years to develop this software, and we developers will spend 1-2 man-years interacting with it. As long as development tools are created in the same order of magnitude of effort as is spent using them, they will never cause a 100 or 1000-fold productivity improvement.

Posted 10:38 AM by Floyd

This kind of crap really bugs me:

Not so long ago (9/11), this country was up in arms about how important being an American is. We all ran out and bought American flags to put on our houses and cars. We publicly slammed celebrities and anyone else who wasn't 100 percent patriotic and dedicated to the American way. Now, just a short time later, the almighty dollar has taken priority again, and our devout patriotism seems to have been overrun by inexpensive off-shore labor in order to boost profit.

Denise Gargano
San Jose

Denise, I fly a flag in front of my garage in part because it is immoral for religious fanatics in other countries to try to murder me.

It is not immoral for people in other countries to compete against me in the labor market.

Posted 8:49 PM by Floyd

A typical virtuoso performance by Mark Steyn in the Telegraph:

Let's go back over that slowly and try not to get a headache: the EU's main concern about an actual epidemic of hate crimes against Jews is that it could provoke a hypothetical epidemic of hate crimes against Muslims. You couldn't ask for a better illustration of the uselessness of these thought-police bodies: they're fine for chastising insufficiently guilt-ridden whites in an ongoing reverse-minstrel show of cultural self-abasement, but they don't have the stomach for confronting real racism. A tolerant society is so reluctant to appear intolerant, it would rather tolerate intolerance.

Posted 11:40 AM by Floyd

Ten days ago I noted that "Wherever a large corporation attempts to save consumers money, the left will tell you that this will lead to the consumers' impoverishment, and is the next step on the road to fascism." I did not mention the religious left, which sees lower prices as violating the gospel of Christ, the Nicene creed, et al. From today's San Jose Mercury News:

Lowe's issue is about working families, not a building
By Father Jon G. Pedigo

As a leader within the faith community of San Jose, I am concerned with the effect that pending construction projects and business enterprises have on the life of all persons, not just those within my congregation. I therefore take exception with the underlying assumption that discussion about the proposed Lowe's store at the old IBM site should only be about bricks and mortar.

A community is more than a collection of tract houses and wide streets; so too a building is not just a building. A community is defined by the relationships that bind residents to one another. Thus, when new buildings are to be built within the community -- especially large commercial buildings -- we cannot assume that a new building (and the commercial enterprise connected with it) will have a positive effect in the neighborhood.

The conversation about a commercial building cannot take place only between planners and developers. The conversation must include members of the community at large. I applaud labor leader Neil Struthers' attempts to broaden the conversation.

This is not just an attack on Lowe's rights to conduct business as it sees fit. Pedigo has penned an attack on freedom. If there can be a "conversation" about whether Lowe's is good for the community, why not a "conversation" about whether Catholic churches are good for the community? Would America have been a better place if the anti-Catholic bigots who were a significant proportion of the nineteenth century American population were able to determine whether cathedrals for Irish and Italian immigrants would "have a positive effect in the neighborhood"?

Before we congratulate ourselves on fast-tracking another box store, let us consider this unfortunate reality: My parish has gone from supplying emergency food for 250 persons a month two and a half years ago to more than 1,200 persons a month as of October. Many of those who come to us for help have two to three low-paying jobs; most have no health-insurance coverage. These folks make just enough money for rent and food.

So I ask the question: Will the business that is to come into our community hire mostly part-time workers, thereby avoiding having to cover a bulk of its work force with health insurance? Will this business accommodate hiring people with special needs?

My faith tradition (Catholic) is particularly concerned about the right that all workers be given the opportunity to form labor associations (unions) and collective bargaining in the pursuit of a dignified life. Will this business provide dead-end, low-wage jobs, or will this business provide its workforce with a livable wage and insurance for families? Will the workers in this business be given the opportunity to explore the possibility of collective bargaining in an atmosphere free from intimidation from management?

I was raised a Catholic, and attended Catholic elementary and high schools. I don't remember our religion classes discussing how the right to form a union was part of the religion's creed. Nor do I remember being asked during confession if I had sinned by objecting to an employee's attempt to unionize. We were counselled that our teenage hormones would cause us to have strange and sinful desires -- but I don't recall that those desires would include the sin of working in a low-wage job.

(And what is this nonsense about a "dead-end job"? How can a job at a hardware store be anything other than a "dead-end job"? Is there some expectation that the job of a clerk in a hardware store in 2003 is to scan items for checkout, but in 2013 that clerk position will involve investment consulting? Would it be my duty, if I were still a Catholic, to refuse to patronize all gas stations, fast food outlets, and department stores?)

Posted 8:13 PM by Floyd

Evan Kirchhoff blogs about the California grocery worker strike, and about the trend toward automation and elimination of manual labor jobs. The bulk of his post is typically perspicacious, but I disagreed with this statement:

Hence, in case it needs to be said: of course I approve of "outsourcing" tech jobs like mine to foreign countries. I accept that all currently lucrative programming skills will collapse to zero value over the next 5-20 years ...

It is possible that certain "programming skills" will collapse to zero value; in fact I would not be surprised if these skills had a low value right now. That is because the "programming skills" I have in mind-- writing code to cause a computer to perform certain actions according to a specification -- are not the critical path for software development jobs. Software developers will be in high demand, and will command large salaries, regardless of whether there are three million people or three hundred million people who can write a for() loop.

I am not saying that coding skill is unimportant; rather that coding is a necessary but not sufficient requirement. It is rare for poor code to kill a software project. One common factor that makes projects fail is when the programmers put their pieces together and find that they do not integrate. Obviously the risk of integration failure is not going to lessen if your team is split between two continents.

Another mistake that causes projects to go bust is the engineers' failure to understand the problem. In many products that I have worked on, the hardest problem was simply to figure out what was the problem. A software product that solves hard problems -- which is the kind of product that people are going to pay good money for -- typically has complex features, and even more complex interaction between those features. Understanding the features is often beyond the comprehension of many engineers. It's extraordinarily difficult to write this knowledge down -- usually the resulting literary effort is incomplete, or confusing, or coma-inducing. So engineering teams rely on experts to explain difficult concepts to other team members face-to-face. It's hopeless to try to accomplish these explanations via email, and difficult to do so over the phone.

Ambitious software projects, especially projects that attempt to use new technology, have another characteristic that makes it difficult to outsource them: Initial work on such projects are prototypes that uncover shortcomings of the design, and as a result the design (and features) undergo considerable change. Team members who are geographically separated will find that their understanding of the project diverges more and more from the evolving current design. (If the development is performed entirely overseas, then the outsourcers will find that what they get back is not what they asked for.)

In the last ten years I have had many opportunities to view attempts to outsource work to India. Here, in no particular order and with details redacted, are my observations:

• An Indian subsidiary was created to perform data entry. For the first year or so absolutely no work was done. The head of that organization was fired, and the new boss proved to be much more competent. Eventually the data entry subsidiary did a fair amount of good, cheap work.
  
  
• I wrote a quick-and-dirty tool to perform unit testing of my product's Java code. Our company's Indian development operation started developing a bigger and better version of this tool. This software was worthless, and had to be rewritten by some of our American developers.
  
  
• My employer had a large software development organization in India. There was considerable effort put into training the Indian engineers; many of the American workers would take trips to India, and most Indians would spend a few weeks working in America.
  
  Results were mixed. Some products were completely done in India and seemed to work reasonably well. Our product had co-development in both India and America. The majority of the developers were in India, but they did very little lasting work. Often quality was a serious problem. I am not surprised; if your motivation for hiring someone is that they are cheap and expendable, why would he care about the long-term viability of his work?
  
  
• Our team outsourced changes to be made to some of our application's HTML pages. Despite the fact that the work was well-specified, limited in scope, and near-mechanical in nature, the quality of the work performed in India was very poor.
  
  
• An Indian developer was helping a customer with a bug. He was having problems using some UI framework code that I had written. He would email me use cases, and I would try them out and email him my suggestions. Executives who know nothing about software development call this "24-hour development", the idea being that at any given moment, some employee of the company will always be hard at work. Here is what our email conversation was like:
  
  
  
  
  • I think the X is wrong.
    
  • Do you mean the X in Y or the X in Z?
    
  • Oh, the X in Z.
    
  • Okay, try setting your foo to bar.
    
  • It doesn't work.
    
  • Please send me your config file.
    
  
  
  The problem is that if you need to ask a question via email of someone twelve time zones away, you will not get a response until the next day. We wasted a week working out problems that could have been solved in half an hour face-to-face. I doubt if the customer was impressed with this five days of "24-hour development".
  

(By the way, I am not anti-Indian. I have worked with many fine Indian engineers, and they performed well when they were here in America and not ten thousand miles away. Also I am probably overemphasizing failures, which are more memorable than successes. One India QA group did quality assurance on a Japanese port, even though no member of the team spoke Japanese. That's the kind of war story I'm going to tell to a fellow developer, but it really has nothing to do with latitude or longitude. Nobody is going to bother with a tale such as "we had a team working in Bangalore on X product and ... it worked just fine.")

It is true that there are a lot of people in the tech industry worried about outsourcing to India (and other countries). Here are some reasons why I think they are overreacting:

• The tech industry just went from boom to bust. A lot of layoff victims are looking for work, and outsourcing is a convenient scapegoat. But many of these people should never have been working in software development in the first place, just as most of the people who moved to California in 1849 were not expert gold panners. The dot-com bubble had very little to do with technological progress, and very much to do with a whole lot of pigs trying to get their noses in the feeding trough.
  
  
• Indian software development, especially in Bangalore, has become so successful that good engineers are commanding more money. This is vague and anecdotal, but when I was first exposed to outsourcing in the early 90's, we would hear about how our company could get ten people in India for the same cost as one person in America. Now the ratio is expressed as three to one.
  
  
• Outsourcing is a fad, and like all fads will run its course. American companies will discover the drawbacks that I have listed above. Here is an example from today's paper: Dell is moving its customer support from India back to America.