‘With specialist middleware, the sky’s the limit’

At the recent Launch Conference 2013 in London, a panel of middleware experts discussed the state of the industry, and in a recent Develop Online, this quote jumped out at me:

“In addition to the more comprehensive solutions such as Autodesk and Unity, there is more specific middleware available for certain tasks, such as rendering clouds in games. But Four Door Lemon founder Simon Barratt warned that this may actually be a less efficient option.

“The trick is to keep it as simple as possible, find something that combines multiple solutions,” he said.

At Simul we make trueSKY, which among other things, renders clouds in games. We also do atmospherics, time-of-day, rain, snow, fog, lightning and so on. We started up six years ago, and in the early days our customers were mostly high-end simulation makers and aircraft manufacturers.

But since the advent of the next-gen consoles, we’ve seen an explosion of interest. Some of our customers write their own engines, and bolt-on the middleware parts they need. Others use all-in-one engines, with trueSKY plugins that we supply.

An all-in-one engine offers many advantages. These engines often come essentially as a pre-built game. You then mod that game by replacing the content, adding your own scripts, and adding new code. This means that all of the boilerplate of game construction is already there; it works; and has been tested by end-users. You have a clearly defined pipeline too – plugins are provided to export your shapes from Max/Maya.

Because most engines are designed to be cross-platform, their APIs don’t correspond to what an experienced programmer may be used to. You might know DirectX 11 or OpenGL like the back of your hand; learning the graphics API for an engine means going through the whole process again. And that leads lock-in.

The cost of any middleware is two-fold; you have the price of the licence – usually per-product per-platform, or a periodic site-licence. Then you have the manpower cost of learning the engine and implementing your existing systems in the new context. Once that’s done, you don’t want to repeat the process too much, and if the engine-maker goes out of business (as Lightspeed maker Gamebryo did a few years back), or decides they don’t want to be in middleware any more (see Renderware), you’re in trouble. The engine doesn’t cease to exist, and your licence doesn’t vanish. But you can expect ever-diminishing support from there on.

The other issue with all-in-ones is that, as a programmer you’re bolting your own code onto the outside of an often fairly opaque central pillar. This usually results in having to make callback functions. As programmers will know, a normal function is where you write a bit of code, give it a name, then at some other point in the code you will call it with whatever parameters you want. A callback function is where you write a bit of code, give it a name, then pass that name to some other system, which will call it when it wants to. Callbacks are hard. It’s like playing a version of World of Warcraft where you’re in control of your character’s hat, sword, magic gloves and cloak of confusion; but have no direct control of the character himself.

And there’s really no such thing as a true all-in-one engine, which is why the major engine-makers have put in a commendable effort to support plugins for smaller middlewares.

On the other hand, if you are building your own engine and using middleware components to provide functionality, you’re in the driver’s seat, with a birds-eye view of the system. It’s often easier to debug because you’re working from the top-down rather than bottom-up. And you can choose the best option for each purpose – the sound engine you prefer, the best UI system, the prettiest clouds, the crunchiest physics. This can be cheaper too – you’re only paying for what you need. Lock-in still exists, but it’s more granular, less daunting.

The job of joining these systems together can be daunting. Simon Barratt told me: “The issues with multiple bits of middleware can be stuff as annoying and simple as combining multiple Visual C++ versions, the certain type of runtime libraries being linked against, namespace collisions on shared open-source software being used by multiple middleware (or worse different versions of the open-source software under the hood when you’d like to share one linked version of them)”.

Other issues include mixtures of row and column-major matrix systems, the different threading systems used by each library, and general bloat.

At companies like Simul, our challenge is to make our tools and APIs as simple as possible to integrate with an arbitrary outside system. We removed dependencies on OS software for the reasons Simon describes. We automatically build and test multiple versions, while developing for a main lead platform (right now that’s Visual Studio 2010, DirectX 11, Win32 – though we’re seeing more and more interest in 64-bit). We put everything in namespaces, and try to minimize the number of API-calls needed per frame. With our tools, we try to provide the basics artists expect like an Undo command and context-sensitive help.

But just imagine how much easier integration would be if there were a common standard matrix API that the engine makers and platform holders all supported. Or if the API for multithreading was consistent across platforms. Or if all text formats were JSon, and the nightmare that is XML were consigned to history. I’d like to see an ecosystem of bolt-on game parts covering all the elements that full-spec engines do, but giving developers the flexibility to choose the best component for each task.

And as Simon said: “Specialist middleware for certain games is essential and certainly having amazing clouds would add to a lot of open games, as it’s approximately 30 per cent of the screenspace and deserves the effort.”