Sorry, tech can’t stop crunch

The cover article of Develop issue 120, about how Criterion handles crunch, can be reduced to "You can deal with crunch by investing in your build infrastructure".

Before I argue why I think this is very, very wrong, some caveats: I am, in fact, a huge fan of automated build systems, continuous integration, automated testing, and general automation and metrics-based analysis of the game development process.

I first worked on an automated build process during the porting of GTA3 and Vice City to the Xbox, at Rockstar Games in 2003. At Mi’pu’mi Games, the company I co-founded in 2009, we use automated build processes, continuous integration and unit testing as a matter of course.

Having worked on these processes, I know how hard they are to set up and to keep them running, especially on big console projects. It’s tough to even find the people who want to work on them: it’s unglamorous yet crucial.

So I am aware of the effort it takes to do what Criterion has done, and I agree with the direct advantages of the build infrastructure that Paul Ross mentions in his article.

However, I do not agree with the connection between build infrastructure and crunch at all. Crunch is not a technical problem. It’s an economic problem, and a people problem.

(While I’m going to be picking on AAA console development, much of what I’m saying also applies to AAA PC development.)

Fundamentally, crunch is very straightforward. You have a team, a deadline, and a game, meaning a certain amount of features and content executed to a certain level of quality, that you want to make.

(This can be represented by the scope-time-cost triangle from project management – see here. It’s a simplification, but a useful one.)

Only it turns out that your team cannot finish the game before the given deadline. So you make the team work harder: you crunch.

I assume it’s clear why crunch is bad. There’s a number of good reasons in Paul Ross’s article. I’m not even going to talk about how you can manage crunch, even though we developed some good policies at Rockstar Vienna.

No, what *causes* crunch? Crunch is a bad solution to a problem: Why do we keep picking it? Aren’t there are other solutions?

There are. Let’s look at that triangle again. If you can’t achieve your desired scope with your given time and cost, well, can’t you just change one of those three factors?

Increasing cost is hard for two reasons. First of all: people like money, and they like to keep their money. So when you go to the person with the money and say: "Look, I know I said I needed X million a year ago, but gee, it turns out I need Y more," you can understand if they’re not very willing to give it to you. If you need more money, the odds of them getting it back go down.

The second reason is: even if you *had* money, it probably wouldn’t solve your problem. You can’t just add people and expect things to go faster. This is one of the oldest principles of software development (it’s called Brook’s Law, after Fred Brooks – see here).

I won’t go into why it doesn’t work or how you can sometimes make it work. In general, adding money is not the solution people pick.

If adding money doesn’t work, can’t you move the deadline, or reduce the scope? Yes, you can. In fact, those are excellent solutions. So why don’t we use them?

Well, moving the deadline has a couple of downsides. First of all, your team needs to get paid every month, so if you work longer on the project, your costs go up. So you’re back to having to ask someone for more money.

The other downside is that in AAA console development, hitting your deadline is very, very important. There are an enormous amount of things that need to be orchestrated for a AAA console game release.

The consumer press and the public need to be whipped into a frenzy so that they’re ready to buy your game the day it comes out. DVDs need to be pressed onto discs in factories and put into boxes and transported to stores. All of this is expensive, and changing when it happens costs tons of money.

What about reducing scope? Well, that would work, except you’ve told the public which features and how much content will be in your game, and you don’t want to disappoint them. They’ll be paying $60, after all. Plus, some other AAA console developer is making a similar game, and if *they* have more features and content, players will flock to them. So you have to deliver.

If you choose to play by those rules.

Rockstar Games, my former employer, famously doesn’t. They make games that are different from others’, they take as long as it takes to develop them, and they don’t announce a game until they’re very sure they can release it when they say they will. They value quality over time and cost. The same goes for Valve and Blizzard. They release games when they’re done.

I don’t know if people crunch or not at Valve or Blizzard. I know we had projects at Rockstar Vienna that were low on crunch, but that doesn’t mean they were low on stress. But in any case these companies show the triangle can be bent.

Of course few companies are Valve, Blizzard or Rockstar Games, and you can’t eliminate crunch tomorrow by emulating them.

Let’s look instead at online games. These games, once they are playable by the public in some way, are not, in fact, projects. They are operations. They are services, not products. This is a very hard thing to understand for people from the project-based development world. You don’t have hard deadlines, you just manage priorities, trying to improve the game as best as you can, every single day. It’s stressful, and sometimes you have to scramble to fix a server problem, but the causes of crunch are not programmed in, the way they are in AAA console development.

What about indie games? If you are going to distribute the game yourself over the internet, you are no longer dependent on factories and trucks, so it’s a lot easier to move that deadline, if you can afford it.

(But wait, won’t moving the deadline screw up your marketing plans? Only if you use launch marketing, which is what AAA console publishers use. Read this blog post by Nicholas Lovell about other options you have as an indie)

At Mi’pu’mi Games, we know we can’t predict exactly what we need to build or how long it will take, so we try to deal with that uncertainty.

Despite having five certified scrum masters in the company, we don’t use any particular methodology, although our approach is definitely agile. That agility allows us to adapt as we learn more about the game we’re building.

It’s not always easy to combine this with milestones in work-for-hire projects. We’ve had stressful periods, we occasionally work late (but not very late) and sometimes someone even works on a weekend.

But on the other hand we developed 5 Nintendo DSi games in 6 weeks, and that involved exactly half a Sunday’s worth of crunch, for three people.

On a side note: if we wanted to crunch more, Austrian labour law would make it more expensive than in the UK, so that gives us an extra incentive to avoid it.

But this article is not an advertisement for Mi’pu’mi Games. Why can indie and online developers avoid crunch when AAA console developers can’t?

Because AAA console development is like Formula One racing: incredibly expensive, fantastically complex, and insanely competitive. And it has been getting more expensive, complex and competitive for over 15 years. Which is why companies have gone bust, developers have left to start new, smaller companies, and the survivors have very little margin of error left.

At the beginning I said crunch is an economic problem and a people problem.

It’s an economic problem because AAA console developers have to crunch to be able to play where they’re playing. (In a red ocean.) The economic incentives of the people who decide that the team must crunch (and who often crunch themselves) are simply set up that way.

It’s a people problem because of our psychological limitations. You want to make the best game possible. You want to think things will work out. Crunching worked last time, right? (If you survived it.) It wasn’t that bad. We’ve made better plans. This time is different.

Better build management is like shaving a tenth of a second off changing tyres during a pit stop. It doesn’t prevent crunching: it prevents wasting time. The deadline won’t move. The budget won’t grow. The scope will, as soon as your competitors also improve their build management.

I am convinced that as long as the business model of AAA console development is set up the way it is, AAA console developers will crunch. But it’s not inherent in game development.