Valve: Low latency fundamental to virtual reality

Low latency is fundamental to the success of augmented and virtual reality but is not possible with current hardware, claims Valve’s Michael Abrash.

In a blog post, Abrash said that latency was the enemy of virtual registration, assuming that there was accurate and consistent tracking.

He explained that if too much time elapses between when your head turns and the time the image is redrawn to account for a new pose, the virtual image would drift far enough so that it had clearly wobbled.

Abrash said without latency at what research indicated needed to be 15ms, or even as low as 7ms, it was impossible to deliver good experiences through augmented and virtual reality.

He added that even being right 99 per cent of the time was not good enough as the brain would still register this with the screen so close to the user.

“The key to this is that virtual objects have to stay in very nearly the same perceived real-world locations as you move; that is, they have to register as being in almost exactly the right position all the time,” said Abrash.

“Being right 99 per cent of the time is no good, because the occasional mis-registration is precisely the sort of thing your visual system is designed to detect, and will stick out like a sore thumb.”

He went on to say that that the challenge of overcoming low latency required new hardware, as many games generally have latency from mouse movement to screen updates of around 50ms or higher, while from his own experience, more than 20ms “is too much for VR and especially AR”. Higher latency he says also seems to be connected with simulator sickness.

“AR/VR is so much more latency-sensitive than normal games because they’re expected to stay stable with respect to the real world as you move, while with normal games, your eye and brain know they’re looking at a picture,” said Abrash.

“With AR/VR, all the processing power that originally served to detect anomalies that might indicate the approach of a predator or the availability of prey is brought to bear on bringing virtual images that are wrong by more than a tiny bit to your attention. That includes images that shift when you move, rather than staying where they’re supposed to be – and that’s exactly the effect that latency has.”

Abrash added that it was difficult to display a high enough resolution, an appropriate image size, tech small enough to be used by consumers while also keeping a low cost.

He said that he hoped with the upcoming release of the Oculus Rift, which has been praised by a number of developers and is highly anticipated in the virtual reality space, that the VR market would take off and the industry would be a step closer to new hardware to finally solve the problems facing the space.

“There is no way to get low enough display latency out of existing hardware that also has high enough resolution, low enough cost, appropriate image size, compact enough form factor and low enough weight, and suitable pixel quality for consumer-scale AR/VR. (It gets even more challenging when you factor in wide FOV for VR, or see-through for AR.)" he said.

“Someone has to step up and change the hardware rules to bring display latency down. It’s eminently doable, and it will happen – the question is when, and by whom. It’s my hope that if the VR market takes off in the wake of the Oculus Rift’s launch, the day when display latency comes down will be near at hand.”