MCV/DEVELOP News, events, research and jobs from the games industry
EVE Online has hundreds of thousands of subscribers and is among the most successful massively multiplayer online games available. With its size and complexity, the development environment can be very challenging for EVE’s creators, CCP Games.
In the MMO market, where customers are constantly looking for new features, the pressure is on to continually roll out new developments for each game while ensuring ‘business as usual’. After all, in this round-the-clock world of collaborative and competitive game-play, users demand minimal downtime.
One of the methods that CCP uses to meet this demand is Continuous Delivery (CD), which is being adopted by an increasing number of companies, not just in the games industry, but across all organisations with software development needs. Continuous Delivery is an emerging software development methodology that automates and improves software delivery. By building the capability to rapidly, reliably, and automatically push out enhancements and bug fixes to customers at low risk and with minimal manual overhead, organisations can respond to business needs faster.
Some experts say that Continuous Delivery is an evolution from Continuous Integration, which is well established for iterative and incremental software build and test. CD takes that one step further by automating, as much as possible, the process of pushing those tested builds into production use.
Although it may seem like a recent trend, a recent Evans Research survey* found that 65 per cent of respondents from the United States and United Kingdom are using Continuous Delivery for at least some of their projects, with 28 per cent using CD for all projects. However, as with any new methodology, there can be a big gap between theory and successful implementation, in terms of both technology and cultural impact.
Version management is at the heart
To address this gap, CCP places version management at the heart of its software development. This enables CCP to keep track of all the moving parts in its development processes and roll out new features, without impacting the smooth running of its online games 24/7.
Erlendur Þorsteinsson, senior development director of EVE Online at CCP, takes up the story:
“Our version management system, which is held on a central Perforce server, is our definitive software library for all our games. This is not just for code but for art assets (such as textures, geometry, shaders, etc.), audio, built binaries (such as executables, dynamic libraries and static data), external libraries and tools plus configuration files.
So what advice would Erlendur Þorsteinsson share with other organisations?
“There are a number of lessons that we have learned over more than a decade of operating EVE Online for our customers, regularly releasing updates to it, and see it constantly grow year over year. Below are some of the more important lessons.”
1. Create a Definitive Software Library — by ‘versioning everything’
“If you don’t treat all your configuration values as code and version these in your version management (aka source control) system, then you don’t really have a Definitive Software Library. Many developers will recognise the mantra of ‘infrastructure as code’.
"This step is necessary because the totality of your software is not only the code or executables, it’s also the configuration or settings for that software. You should treat your configuration as code because that allows you to track and manage changes to your configuration just as you do with your code. Before we did this, we would sometimes be scrutinising code changes in the version control engine, only to find out that the problem that had surfaced was due to a configuration change that was not being tracked in the version control engine.”
2. Make it as easy for users as possible
"An area where complexity can get out of hand is in the mainline and branching model. Branching is an integral part of how games companies use version management. Through the years, we’ve evolved towards a streamlined model with a fixed mainline that all changes flow through and a fixed release branch where the live version of the software can always be found. Most changes originate in the more volatile sandbox area, flow through the fixed mainline, and then enter the staging area before making it into the fixed release branch. This model has made it easier for developers since the flow of change needs to be learned only once, rather than having to learn new routes after every release and find the latest active development branch.”
3. Do not impose ‘one size fits all’
“In our sandbox area, the branching model is by feature and team, but once we reach the staging area, that changes to ‘by release’. We’ve got several hundred people working in this environment and as expected the rate of change is higher in the sandbox. But as we move towards release, it decreases — meaning that we are progressing towards stability for a release.
"The branching model I’ve described is good for a large development effort that involves tech and non-tech people across business units and offices. But we also like to give developers flexibility, especially when working on smaller projects, so we offer them Streams, which allows teams to create their own branching models. This works particularly well for applications that have their own depots.”
4. Store binaries in version management
“We also store our built binaries in the version control engine because we found that allowing our developers to do a ‘sync and execute’ saves a lot of wasted time that individual developers would otherwise spend syncing the source code, configuring all the build tools, building the binaries, and finally being able to execute something.
"Key to making ‘sync and execute’ more manageable is breaking up software into modules. Each binary is only around 10Mb, so checking those into the version control engine is easy. It makes a lot more sense than spreading your software development across different locations—version management server, file share folders, and FTP sites, for example— because that makes it impossible to have a Definitive Software Library.
"External libraries, such as SDKs, are another example of binaries we check in to Perforce. We take the unmodified versions of those files and put them into an ‘SDK install’ branch. Then we’ll create a branch spec to turn them into something much leaner that is consumed by the development branches.”
5. Store static data in the version management system, not a database
“For us, static data means a lot of the game-related data that does not change during game-play, for example, the placement of a floor tile or the radius of a planet.Storing this data in version management is now feasible thanks to recent developments. We are in the process of moving our static data, which was previously in our SQL database, into the version control engine. That means everything will be in one place and easier to manage, although we do need to address the fact that static data is more challenging than code to merge.
"The syntax for code is well understood by us humans and helps to resolve conflicts and merges: The compiler can often tell us when something went wrong. File granularity for code projects is usually reasonable and a known factor, which reduces the chances of conflicting changes; it also helps that the organisation within a code file follows rules, policies, and conventions. These properties do not translate well to static data when using XML, which is great for parsers but not intuitive for humans.
"The solution we’ve found is to use YAML, the syntax that is human readable and makes it easier to merge static data. We have introduced additional constraints. For instance, the only containers we allow are sets because they are strictly ordered, so if a change happens, then that change is distinguishable. We have also created our own merge and diff tool that we assign to Perforce clients via file extension mapping. And we introduced automated build steps to transform the data into an optimal run-time format, also versioned in Perforce.
"Our dynamic datais still stored in a database that does not have branching support, so any changes there have to be backwards compatible. We still track the changes by storing the schema and procedures in the version control system, while the continuous integration servers monitor all changes and can verify backwards compatibility. This means we can tie all code changes together; it also ensures that we don’t discover problems in code branches later in the process.”
Final words
As one of the pioneers of Continuous Delivery and an imaginative user of version management, CCP is blazing a trail not just for game developers but also for any organisation that depends on software development.
Erlendur concludes: “Our mission is to rapidly create and deliver value to our customers through updates to our games. Making versioning a strength for our organisation and making it the nerve centre of our business has really paid off for CCP.”
