Home‎ > ‎

The growth and perversion of systems

The requirements of a system and its software constantly change because the user and developer didn't know what they actually wanted until they'd already tried building it once, and because growth in business and personal life is driven exclusively by change itself. But most systems and computer programs are written to resist change. At the system level it's a problem of ignorance, while at the program or code level it might also be a consequence of reality, since code has to draw the line somewhere.

Systems resist change in three ways: 1) their parts can't be swapped out, 2) a peculiarity of your business got baked into the way the system works, and 3) it was over-automated.

Parts that can't be swapped out

  • The database manager, because you coded to the vendor's own proprietary variant of SQL
  • A black-box program, because nobody knows how it works
  • A monolithic program, because the cost of re-implementing all of its functions is tremendous
  • A file format, because too many programs are hard-coded to read it
  • The messaging protocol, because its interfaces are too difficult to mimic
Some of the above will be red-herrings for you, such as the choice of sticking to one vendor's database, but with the exception of the black boxes they all share the same problem in common, which is that replacing them requires too much effort to update all of the other components.

Baking peculiarities into the system

  • Your Cable TV shopping channel only sold one item at a time, so your "orders" table had the column for the SKU and no line-items. Now that you have a web site with a shopping basket there's still no way to ship multiple items per order
  • You start selling powders or liquids by weight, but your quantity columns are all integers. Someone fixes this by extending the price column to six decimal points and making a tenth of an ounce the smallest unit sellable
  • Your loan billing system assumes the entire balance is subject to one interest rate, and then the Fed Reserve bans the practice of retroactive interest rate hikes, implying the need to track multiple interest rates per balance
These are slow deaths because the cost to work around them on a daily basis eventually overwhelms the payoff from making the change. If the organization depending on this system doesn't die, then it'll be forced to replace the entire system at great risk.

But the truth is that it's impossible to avoid peculiarities. There is no such thing as the ultimate generalized system (unless you count the Turing Machine itself) and you will often not realize what a peculiarity is until it's suddenly getting in the way.

To be in control of them, systems must be broken up into subsystems that are small and focused. Separate inventory, purchasing, payroll, fulfillment, cataloging, CRM, accounting and everything else into their own stand-alone systems, with their own databases (from different vendors, if it suits the task), talking to each other with interfaces that are as specific as they can get. When a peculiarity rears its head and gets in the way of a necessary change, there'll be less to demolish and rebuild.

Do not attempt to build one big beautiful temple of convergence in the name of reuse and elegance, you will accomplish neither.


For the sake of efficiency, systems are made too air-tight by naive designers. The result is opaque and magical to its users, so the greater system (consisting of employees as well as the software) is unable to deal with edge cases, even though the software designers may have provided a way to deal with them after all.

Imagine a machine that receives wood in one end and outputs furniture. It's a completely sealed unit that's automated for safety and efficiency, but when a splinter gets stuck somewhere the machine's way of dealing with the problem is to dump the entire pile of unfinished parts into a heap. As the machine only has one input, and that input only takes raw wood, there's no way to fix the cause of the fault and resume the process where it left off.

A large enough software system must include humans in its basic operation. You must build tools for manually advancing a case (be it a customer order, hotel reservation, or whatever) through all of its phases to completion, and then automate only the most common cases. A user should have the tools for fixing a faulted case and resuming the process from where it left off, and system needs to leave enough edge-cases un-automated so that the users are continuously practiced and know how to use the tools well.

Our furniture machine should be an open assembly line, with one station that cuts the wood to length, another that sands and planes it, another that mills and shapes, an assembly stage, and a finishing stage. It's okay to automate the stations and the transfer between stations, but an employee should be able to grab a toolbox and fix a problem by hand at any stage. Your software system will need the equivalent of hacksaws, Dremels, and putty.

Murderous intent

Software is disposable, and each program you write should be written with the intent of killing it--at the most within a decade, but usually after a few years. But in most shops the opposite is true, and the software becomes a precious baby that gets fixed and extended instead of chopped up and recycled. The engineers end up spending too much time trying to make programs endlessly configurable and extensible instead of making the ecosystem like that. 

You get tasked to build a credit card processor, and you think "there are so many APIs offered for processing credit cards, that means I have to abstract them all so we can switch between them." But that's the wrong solution to a valid problem. The right way is to build the smallest program possible that's hard-coded for one vendor's API. The number of times you'll need to switch between vendors is so seldom that the investment to abstract it won't pay off, and you won't have to cram another vendor's paradigm into the abstraction you've just invented.

The next thing you find yourself doing is writing one big program that validates addresses, charges cards, and then settles the charge after the order ships. Then you discover a bug in the address validation stage, and for the next couple of days the company can't charge verified cards or settle charges while you fix it. The three processes needed to be separated into three small, disposable programs.