Pipe dreams: Where pipelines and workflows go next

[This feature was published in the August 2013 edition of Develop magazine, which is available through your browser and on iPad.]

Games developers can often be found discussing how they go about structuring their teams and integrating their pipelines to maintain the technically robust and creative atmosphere needed to craft games.

Pinning down the secret to a good pipeline is like trying to make the perfect lunch menu for a party: there’s more than one way to go about it, and what works for some won’t work for others.

That said, over the successive generations, standardised practices and structures have taken shape to help make the production process more effective for all of the individuals and groups involved.

And those at the forefront of these essential technologies tell us that, even as games are becoming more complex and the production process more disparate, pipelines are evolving to meet the herculean challenge of games development head-on.

START AT THE BEGINNING

Speak to the firms behind the tools and technology of production, and you’ll find that the definition of a pipeline, and indeed its specific purpose, varies – which is surprising for a sector that is already caught between increasingly scant shades of grey.

For freelance designer and producer Jurie Horneman, for example, the core of any pipeline is about integration.

“What is unique about games is the number and diversity of disciplines that have to work together: code, visual arts, animation, sound effects, music, architecture for level design, writing, game design, the list goes on. This goes beyond what is required to, say, make a movie,” Horneman tells Develop.

“What counts is the final experience for the player. All the work in all the disciplines involved in making a game has to come together to create one experience, and that experience should be seamless.

“However, this is difficult. It’s easy to go, ‘well, the engine is there’, or ‘hey, I’ve finished my textures’, but for a developer’s work to be really ‘done’ it needs to work together with everything else, and create a magical experience for the player. Even experienced developers can completely ignore the essential step of integration; of making everything come together.”

Time and again, game developers hit rocky waters due to poor pipelines and planning. The specifics are often fuzzy, but the sagas are well known: Too Human, Star Wars: The Old Republic and Final Fantasy XIV to name but a few. And those are just the ones that made it out the door.

Generally, a principal pipeline will begin with asset creation and asset collection. These are the raw materials that go in to form the foundations of a game, the things players will see and hear and the starting point for generating a sense of character within the world. This ends with runtime; finished sequences where all of the pieces operate in harmony to produce the game experience.

“It is really pipelines, plural, as there are many paths which overlap, branch, loop, are short, long and so on,” offers Andrew Bowell, head of product management at Havok.

He tells us that Havok, which is behind game middleware that has become a staple in the console and PC space, considers the pipeline to be the path that data travels from the hands of the creators, be it an artist working in Maya, a designer working in a level editor or a programmer working in Visual Studio, right through to executing in the game. But that’s just for a single platform. These days, developers want and often need to be on many more.

“Many teams do not have the time to devote to developing exclusively for one platform, so pipelines need to be so many things at once: flexible, efficient and full-featured, to make the most use out of created assets. A single pipeline needs to allow developers to develop high quality assets, while giving them the ability to deploy them to various platforms,” explains Autodesk’s senior manager of product marketing Greg Castle.

“Autodesk can efficiently link all the different pieces of the workflow, such as pioneering the standardised FBX format that enables efficient cross-tool production, or providing complete solutions like our Entertainment Creation Suite.”

But pipelines are not just about achieving quality as well as platform reach. For Hansoft, the limitation of time is the chief consideration behind its tools.

“Pipelines and workflows are scheduling tools that have been created to deal with the growing complexity of feature and asset production in games development,” says Hansoft’s senior production expert Oliver Teckert.

“Pipelines are used to transparently break down the development process for a feature or asset into its component tasks, be it the creation of a character or the development of a feature which requires several steps to complete. This allows for additional granularity in scheduling out each task, without losing sight of the end goal.”

DECENTRALISED DEVELOPMENT

As such, time is the ultimate limitation that games development is bound by. For audio outsourcer OMUK, pipelines exist to bring discipline to the process in order make sure the right materials go into a game from the very beginning.

“There always needs to be a logical process to follow in order to avoid time delays or expensive errors. Pipelines help to define this. It’s worth noting that these methods should avoid being restrictive. By all means, deadlines need to be met, but by allowing some freedom within the pipeline, you are sure to end up with a better result,” Mark Estdale, managing director at OMUK tells us.

“At OMUK, we prefer to be involved with projects as early as possible. That’s when we add the most value. Our creativity and experience in dialogue production allow us to spot potential issues and correct these early on in the development cycle.”

OMUK is just one of many specialist services firms that cater to the games development industry. These firms are as much a part of the process as the dev studio itself; thus an increasingly big part of pipeline technology is about linking production together across vast distances as development has become more decentralised.

“The large scale adoption of outsourcing has changed pipelines,” explains Teckert.

“Almost every developer now works with outsourcing studios, mostly for art assets but also for code. This introduces a couple of issues. The first is that every asset or feature that is outsourced needs to be tracked as a line item that is being worked on by the outsourcer, which can be an administrative nightmare. With hundreds or thousands of outsourced items, if this is not done correctly, milestones can be missed and key development time lost due to simple miscommunication.

“The second issue is that regular feedback must be provided to the outsourcer during the development process, which means that the process needs to be transparently tracked at a granular, step-by-step level, to ensure that feedback can be given early and often in the production process.

“This ensures that assets and features are being developed as intended, and changes are communicated clearly. Obviously without pipelines and workflows this can create an administrative nightmare which can significantly impact the quality of the final game, not to mention the cost of producing the final asset or feature.”

But scale isn’t the only thing impacting these tools.

“The other major area of change in the past two years has been the complexity of creating and developing an asset or feature itself,” Teckert continues.

“As gaming platforms continue to grow more powerful and developers look to take advantage of that power, developers are creating increasingly complex assets and features. Where once the development of any given asset involved a fairly straightforward process that could be completed by a single team member, the same asset can now requires involvement from several specialists’ team members and sub-process pipelines.

“Areas such as modelling and texturing can have special technical tools that have been introduced into the production process to help speed up some tasks. However, the end result is that several steps in the production process now take longer to accomplish. Trying to track assets across different disciplines and teams has become exponentially more difficult as a result.”

AN AGILE AFFECT

The issues that Teckert raises, about asset tracking and cross-compatibility, have been chief in the minds of pipeline technology providers. There are various ways that they are attempting to tackle these challenges at a software level, but, to make real progress, development teams need to have structures in place to work smarter.

‘Scrum’ and ‘Agile’ sound more like rugby plays or circus training exercises than team management practices, but they are the means by which teams now organise their workflows.

Agile is a development methodology that focuses on iterative and incremental progress, where needs and solutions evolve through collaboration between
cross-discipline teams. Through Scrum, developers work towards common goals in the production process by approaching them flexibly and holistically, as opposed to sequentially in silo-style sub-teams.

“In the past, most studios structured their teams by professional discipline. Some pipelines can be completed within a given discipline, but with the majority of pipelines you are trying to create a fully functional asset or feature. Often this means you need to go outside a discipline group and pull in team members from code, design, art and so on to work together towards the pipeline goal,” Hansoft’s Teckert says.

“Typically, a team working with pipelines is organised into a Scrum group, continuously collaborating while each team member completes their portion of the work before moving to the next task in the pipeline.

“In this way, the team can address quality at all stages of development and fully understand how their asset or feature is taking shape. This process really engages the team members in the creative process and ensures that there is a constant feedback loop present that emphasises quality.”

Perforce’s EMEA marketing director Mark Warren explains that the management software maker has adapted its tools to suit Scrum: “Perforce’s approach has always been to be as open and flexible in use as possible.

“For example, until Perforce Streams was introduced, it didn’t offer any built-in workflow – and Streams are still completely optional. By keeping things simple, it allowed users to build workflows that work best for them. That’s still true today as teams have moved from small start-ups to global businesses, or from long development cycles to Agile, and now into ‘Continuous Delivery’. This approach will continue to keep Perforce ready for whatever changes are coming.”

Having created mobile games in the form of proof-of-concept titles Hyperspace Madness and Starship Battlement, Castle says Autodesk now thinks of itself as a developer as well as a tools provider.

“We have first-hand knowledge of the challenges of games development, which better informs our tool development,” Castle confirms.

DO IT YOURSELF

But while tools companies such as Autodesk have been dabbling in games development, it helps to have an outside opinion on such software, as Horneman can attest.

He believes that a good workflow has a few simple qualities: you should quickly see the result of your work under realistic conditions; meaning, ideally, that it’s running inside the game on the target platform. It is hard to make mistakes or lose work. And, finally, if you do make a mistake, it is easy to find out what it is.

“This sounds easy, but it can be quite hard. The epitome of this is live editing: you play the game, stop, make some changes, then keep playing. It’s one of the big selling points of Unity. But this can be technically complicated, and I’ve seen big companies waste millions trying to get this right,” Horneman says.

“I’ve also seen workflows where it was easy to lose work: do something in 3ds Max, add some data in a separate, custom tool, then you go back to Max, re-export, and your custom data is gone.

“Or something looks wrong inside the game – too big, no textures, etcetera – and it takes forever to find out which part of your fragile, overly complex pipeline fell over without a peep. I’ve seen that too.”

Sticking with the notion of making the workflow suit the team, Gary Leach, lead game engineer at 22Cans, recently wrote for Develop Online about how his studio created a pipeline for Godus almost from scratch.

In his view, the tools should fit the people: “One thing I find very important is to ensure that the tools fit the team, not the other way around. Working this way allows us to get the most out of each individual without them having to learn complex new workflows.”

ANOTHER WAY

Leach says he prefers not having to do anything specific to structure the team for development. Aside from a few pointers on efficiency and budget, he says he is able to allow them to just get on with things in their own way. While that may go against the grain for some, it is a view that discerning pipeline providers will heed.

The key issue with pipelines is efficiency, Leach says, and particularly runtime efficiency: “User experience is most often directly linked to the speed of a game, both speed of loading and speed of rendering each frame. This makes it very important that all data prepared for game use should be efficient to load and take the least possible resource to render,” Leach asserts.

“This can be a very platform-dependent task, so preparing efficient data for a console might be a very different process to preparing mobile phone data. What works well for one may be disadvantageous for the other.”

This is something that Autodesk’s customer success manager Mathieu Mazerolle is well aware of.

“Today’s modern mobile and web game developers are looking for ‘write once, deploy anywhere’ pipelines: they want to be able to focus on what goes into making their game as fun as possible, and not get hung up on all the technical details of getting their game onto different devices.

“In order to meet this challenge, pipelines need to be very scalable in terms of adapting to different screen sizes and performance specifications. That’s why tools need so much functionality to quickly adapt assets to different environments, such as reducing the number of polygons or re-sampling animation so it uses less memory.”

The pipeline makers are making strides to keep up with these tectonic shifts. Autodesk is utilising technologies like the cloud to enable teams all over the world to collaborate on projects. Perforce’s commitment to ensure its software cooperates with the latest development tools has been shown with its recent induction to the Unreal Partnership programme. And Hansoft is conversing with its customers to integrate new features, like support for common codebases such as Git.

Yet the challenge for these firms and others is facilitating the near-endless stream of needs and expectations that developers now have.

Managing game assets begins at the time of their creation or collect, and when it comes to audio, OMUK’s Estdale feels this specialism is woefully under served by current tools: “For dialogue recording, and specifically recording studios, the design, the workflows and technology developers use are not designed for games. To use them is like banging a round peg into a square hole.

“But the more complex the production becomes, the more the square hole doesn’t work, unless you have the budget and time to adapt the peg and work with what’s there. Naughty Dog, for example, uses the traditional route and gets spectacular results. But Naughty Dog’s resources are the fantasies of most developers.

“Frustratingly, there is little movement in the audio recording world towards pipeline technology for game dialogue work as we’re considered an insignificant market. A few thousand specialist users are of no interest. Consequently, we’ve done it ourselves with our own technology known as Creative Dialogue Tools.”

PRODUCTIVITY OVER PERFECTION

­­Pipelines are pitched as the ‘do it all solution’, but Estdale’s example reflects the relationship that’s often at play with such tools in the development process.

Adapting existing tools, building their own and leveraging previous experience are things developers are highly proficient at.

Because they have been made to cater to myriad technologies, platforms and team structures, it is a near-impossible feat for pipelines to serve the needs of a studio without adaptation or incident. But this shouldn’t faze developers.

On the contrary, Horneman says if a workflow or pipeline isn’t working as well as expected, a studio must use its initiative.

“At the start of your project, look at how much content you need to create and how many people you have to do it. You should be doing this anyway for your planning. Then do the maths. If one programmer spends two weeks building a custom tool, or improving an exporter, how many minutes does this save per 3D object, say? Think of how many more iterations you can make, how much faster you can polish.

“If a workflow isn’t working as well as you hoped it would, spend time and attention on it. It’s that simple, because typically no time is planned for working on pipelines, and too little attention is paid at the team leadership level. Evaluate and measure your workflows, make them someone’s responsibility. Get the problems out in the open, knock some heads together and get multidisciplinary people to work on them.”

Further to Horneman’s points, Leach says that pipelines don’t have to be automatic, but they do need to maximise efficiency for every member of staff that interacts with them.

“Keeping things fast, efficient and flexible at the same time is a challenge. Often, aiming for one or two of these will preclude the other. It’s important to remember that a little time invested upfront pays dividends every time the pipeline is exercised.”

All of the tool providers we spoke to recognise that when it comes to the task of building tools for the games development process, they can’t please everyone.

The cake is a lie, and it might be about time you stopping think about that perfect lunch menu.

Hansoft’s Teckert puts it best: “Collectively, developers need a tremendous amount of flexibility built into pipeline and workflow systems, so they can tailor them to their needs, which cannot always be easily anticipated. So developers are constantly pushing the boundaries of what today’s pipelines and workflows are capable of.”­­