Home‎ > ‎

How to staff an IT department

 IT departments are a form of self-defense for the modern company: they shouldn't be necessary in a perfect world, but predatory vendors and a harsh, fallible environment of hardware and services make them important to any firm that has grown beyond a couple of employees. In a nutshell: off-the-shelf components ship in an unusable state and will always revert to that state without maintenance. Therefore, you need a technical staff you can trust to make it work in the first place, then keep it working.

 There are five types of employee role needed: programmer, system administrator (sys-admin), manager, technical writer, and technician. I say role rather than type because a single employee can be trained or talented at any of them. A DBA (Database Administrator), for example, is something of both a programmer and a sys-admin.

 The bigger the department gets, the less these roles are combined in each employee. Programmers cease to be able to do DBA work in parallel with development, managers stop having enough time to write the documentation, sys-admins get too busy to run cables everywhere and rack servers. But this is also the road to dysfunction because putting skills in "silos" ends with employees who are made incompetent by their overspecialization and ignorance. So rather than carve up responsibilities by the nature of the work, carve it up by the volume of work and create new, small teams of people that you give the freedom to divide responsibilities the way they want. The rest of this article is about how that's done.

Brooks's O.R.

 Fred Brooks used the surgical team as a model for his ideal unit of IT: the head surgeon, assistant, anesthesiologist and nurses all stood in as metaphors for lead programmers, assistant programmers, tool-makers, technicians and technical writers. The lead programmer does project management duties, defines interfaces and writes key sections of code. The assistant programmer implements interfaces. Tool-makers put together small utility programs that are needed by the lead developer, but can also write tests. Technicians and sys-admins assemble, configure, secure and maintain the hardware and platforms that the programs will run on. And finally the technical writer creates the manuals and reference books critical for the users and the team itself.

 The manager's role is actually very janitorial. His job is to remove problems that hamper the team, solve logistical issues, manage cases in the bug tracking system, get funding when its needed, even empty the wastebaskets if that's what it comes down to.

 Three people are the minimum you need to fill these roles, but four is ideal. It means that some responsibilities will be shared but the team itself must be allowed to figure out who and how much, because the projects are all going to be different, and one role is going to get stressed more than on others.

Desirable synergies

 Two teams can and should share their technical writer, and he'll be one of the few people who'll perform only one role exclusively. Managers can also be shared between two or three teams (if the manager does his job properly then the team should spend most of its time heads-down and working, giving the manager plenty of time to practice golf swings--or manage a second team). But a shared writer and manager will also act as a nexus of knowledge and insight:

 "Oh, Team A has built a tool which already does what you want"

 "If you guys can use the same model of server as Team B then I can haggle for a volume discount"

 "If both teams use GUIDs then it'll make the interfaces more uniform"

 "If you beat Team A to the first milestone they have to buy you pizza"

The Maintenance Crew

 The surgical teams will be doing projects to create new things, but the ongoing maintenance of the firm's systems will be handled by sys-admins, technicians, and a few programmers. The sys-admins will be doing configuration and automation; a term that covers the creation and adjustment of employee accounts (setting up their mail, giving them a login, restricting their access) as well as doing what it takes to prevent intrusion and make things run smoothly, especially when unattended. The technicians will be running cable, building and fixing computers and phones, racking servers, swapping backup tapes and keeping the machine room cool. This is another case where the sys-admin and technician roles are frequently combined, especially in smaller firms.

 The programmers will hang around to fix bugs, and they won't be able to do this if you forgot to hire a technical writer. Maintenance programmers should rotate in-and-out of surgical teams, keeping a constant turnover between projects. The reference books maintained by the technical writer will stop them from bugging the on-project teams and the rotation of people between project and maintenance modes will keep them sharp and motivated. If you don't rotate your programmers then the guys who are always on-project will write code that sucks and is hard to maintain.

 You aren't rotating your programmers so they'll "know" every program in the house. That doesn't work; they forget too much between cycles. You rotate them so they will strangle the life out of any moron who can't write clean code. Programmers also hate maintenance duty and will apply peer-pressure to the on-project team to wrap up sooner so they can swap places.

 The technical writer will get involved with maintenance occasionally whenever something relevant has to be changed. His greatest importance, however, is in making sure the company's knowledge becomes institutional, meaning that it doesn't walk out of the door when the employee who created it does.

Who to hire, who not to hire

 Rockstar programmers tend to avoid IT departments except in recessions, and even then you should beware of their flight reflex: the pedigree of your staff cafeteria's chef and attachments to their Aeron chairs will be as much of a factor as their salary. If they describe your benefits package as "K-mart-esque" and you're a small firm that doesn't exactly have a marble foyer, then it's best not to hire or become dependent on them. Be prepared for the day when they sit down with you and say, "yeah... well, y'know, it's such a nice day out there, and I really would like to hit the beach, so uh... yeah. Sorry."
 "Rockstar" programmers are afflicted with perspective anemia and will not be an asset, even if they can write a compiler in 6 lines of code.

 The kind of programmer you want is someone who's good at learning how companies work. They should be interested in how your business runs and should be asking specific questions when you give them a tour of the building during the interview. If they only ask questions about salary and benefits then shake their hand and show them the door, but if they also start asking questions about how you do fulfillment, what your return policies are, what that girl is doing with the box cutter and staple gun, etc, then you've got a hot one.
 Prowess with code is obviously important, but even trade-school graduates are valuable if they can actually see why you do things the way you do. Beware of guys who call themselves "gurus" in anything and prominently display a list of industry certifications, because it means they don't know jack.
 Always give them a written programming assignment during the interview ("FizzBuzz" and a recursive factorial weed out the fumblers early). Never hire someone who tries to dodge these tests.

 Sys-admins can get Glass Tower or BOFH syndrome, but this is usually because they're the first to respond to some terrifically stupid requests. The best sys-admin I've ever known is a lady in her 40s who either had a really good stress-management coach, or was missing the gene that causes frustration and bad temper. The worst sys-admin I've known liked to drag his personal web server from job-to-job so he could run it off the company's bandwidth for free, and made money on the side by grepping the internet for registration keys to sell with pirated software.
 What you are looking for is somebody with infinite patience because this is the dearest resource in IT, as well as respect for--and attention to boundaries (work boundaries and security boundaries) so your network doesn't get infested with worms.

 A technician ought to be neat and methodical. You don't want someone who solves fan-noise problems by sticking a screwdriver in the vent to stop the blades from spinning, nor should they think its okay to let the cable modem dangle from the coax. Hardware malfunctions are the most insidious bugs your system can have because many of them are not as obvious as a server that doesn't power up: they can appear to be bugs in the software instead, and take forever before a team of programmers give up with the debugger and realize that the temperature in the server room is 97 degrees. These problems can be minimized if the technician's approach to cleanliness involves more than a can of air duster.

 If the candidate's first action--when you ask him to change a video card during the interview--is to put on a grounding strap, then you might have found the right guy. If he prefers Velcro cable ties instead of plastic "zip-n-snip" ties to organize cable flow, then you might have found the right guy (he's probably cut himself enough times to learn why). If he ties back his long hair before going to work, or checks for washers on the screws before mounting a hard drive, then you might have the right guy.
 But if he crams all the loose ribbon cable between the power supply and RAM sticks, or can't put an end on a CAT-5 cable without squinting at a wiring diagram and looking confused, then drop his resume in the trash. 

 Managers ought to have a genuine sense of humor, or else they'll alienate everyone they work with. Avoid guys with gold watches, rims and popped collars. Managers need to be resourceful; they ought to know three different ways of getting the same job done, because the "one true way" will always conflict with something else. When something goes wrong they should be thinking about how they can fix the process before blaming the hardware/vendor/programmer.
 Managers also need to be persuasive because in IT they'll have almost no power: the sys-admin will smirk and sort their requests to the bottom of the pile ("yeah I saw the email, but I got a hundred machines to patch against a new Windows vulnerability that just came out, sorry"), the programmer will be passive aggressive and find a million reasons why something can't be done. The manager's only ally will be the technical writer. So he needs to be good at reasoning and justifying what he wants, good at motivating people to do what he wants, and assertive enough that people will listen to him.
 In the interview you should ask him if he's read any books by Dale Carnegie.

 Your writer will need to be athletic because he's going to spend a lot of time on his feet. He'll have a special privilege in the firm: the power to bug and annoy anybody and everybody with questions. In the beginning these will be distracting as hell, but it will pay off big time because he's writing The Book. Whether in Word or Wiki (Wiki, preferably), The Book will show every user how to do their job and answer every technical question the programmers and sys-admins will have. A background in journalism would help because he's going to be more like a reporter than a novelist and that means legwork, legwork, legwork.
 Intelligibility is crucial; his documentation must stand on its own. For the interview, consider giving them a workstation with a browser and a word processor and an hour to explain how a Wankel Rotary Engine works.

 Technical writing is at least as difficult as programming because he'll need to understand what the business does to make money, how the code works, what problem each program is supposed to solve, how that program solves the problem, and explain it in enough detail that another programmer can pick-up where the last one left off.

The unofficial partner

 Creating an IT department means creating a new business partner. The people you hire will end up co-inventing your business alongside you, and even though you think you're the one in control you're actually only half of it. Suddenly the programmers will begin declaring certain things impossible, and the sys-admins exploding the cost and time to do what you want. On the positive side they can also make new business models possible that had never been possible before. Business models like outsourcing your fast-food restaurant's drive-through intercom to a voice in India, or automatically shipping from the warehouse closest to the customer (including those that you don't own), or creating a new product on-the-fly--a thousand times a day--based on a browser cookie, a data-warehouse cube, and a CNC machine

 This is what you must keep in mind when you make your first hires. Skill is important, and communication skills are important, but all pale in comparison to the implicit partnership. Sometimes a candidate who is mediocre--but well versed--in all of these roles and skills will be solid gold if they see things the way you do and even bring new ideas to your table. Knowing the scope of a problem is more important than knowing how to solve it.

 The dark secret of the geek is that he wants to be understood, and this is your leverage on him. The fruit borne by the geek is his imagination. He will, however, shape your business with you.

Further reading