Why optimization is everything in mobile AR and VR

Creating high performance AR/VR applications is challenging. Here are five reasons why it’s especially challenging on mobile devices.

1. High frame rates drive great user experience

High application frame rate is key to a great user experience. Without it, user interface response is slow, making every user input feel choppy or delayed. Worst of all, fluctuating frame rates are a leading cause of nausea for users in AR and VR applications. Simply put, the frame rate for an AR application should never drop below 60 frames per second. And for VR, a constant 120 fps is ideal, as motion sensitivity is heightened in a fully enclosed experience with no externally visible static reference points.

However, achieving such high numbers is not easy. At 60 fps there’s only 16 milliseconds to render each frame and process everything the application needs to deliver a quality user experience. At 120 fps, the time is cut to half of that.

In most 3D applications, the graphics create the greatest processing load. While there is very powerful graphics hardware available for desktop computers that can process highly complex 3D, the hardware available for mobile devices is far more limited due to power and thermal constraints. For these resource-constrained mobile AR and VR devices, automated graphics optimization is the most important part of achieving the performance required for a smooth and responsive experience.

2. Better performance means longer battery life

3D and especially pass-through AR, such as the Apple ARKit, demands a great deal of performance from mobile rendering hardware. Almost every component of the device is involved in creating the user experience, putting a huge strain on power consumption. With highly optimized code and content, a mobile device’s battery will last considerably longer, leaving users with ample power to do other critical things with their devices like, say, make a phone call. If the optimized 3D enables the processor to work less hard to achieve a reasonable frame rate, the device will literally consume less battery power per frame rendered.

3. Very high resolution is needed for AR and VR

Mobile devices nowadays have impressive displays with super sharp images. The sharp image is the result of high resolution — current mobile displays boast millions of pixels. In immersive 3D applications every single pixel needs to be updated every single frame, causing a massive amount of work for the graphics processing unit. And the more impressive the graphics, the more work is needed from the GPU.

However, for realistic AR and VR experiences of the future, today’s resolutions are not nearly enough. Because AR/VR displays are so close to users’ eyes and partially magnified through lenses, they will need even higher resolution to prevent our eyes from noticing the individual pixels. This problem can be seen today with currently available headsets.

Solving this challenge will be a big challenge, especially on mobile. There are companies like Varjo tackling the hardware resolution problem, but all those extra pixels need to be rendered with content that matches the much higher resolution.

All this adds up to much more work for the processors and a lot more memory and bandwidth needed for delivering the content. Intelligently optimizing 3D will be an essential piece of the delivery process to ensure that super high-quality content can actually be rendered in high fidelity on end user devices.

4. High resolution 3D content takes up a LOT of space

Compared to static content such as photos or video, interactive 3D content needs to work from any camera angle and location. The data size requirement for achieving the desired resolution from every possible view point can easily grow to be hundreds of gigabytes or even terabytes, and just storing and transferring those amounts of data becomes a challenge in itself. On desktop machines this problem is (relatively) easy to solve with hardware by just adding more disk space. Obviously, this is not viable alternative on mobile.

Optimization will be key for enabling mobile devices to render large, complex 3D scenes and objects given that storage capacity on these devices is limited…especially since that storage space is potentially shared with other assets such as personal photos, videos and music.

5. Users expect fast app startup and loading times

Mobile application users expect a smooth user experience which includes a snappy user interface and fast application installation and startup times. Downloading the content required for 3D applications, or simply loading pre-installed content into active memory at start-up time could take much longer than the user is willing to tolerate.

To avoid this, 3D data should be streamed to the device. That said, true 3D content streaming is a lot more complicated than, for example, video streaming. Solutions where the rendering is done in the cloud and only the resulting images are streamed don’t really work in interactive 3D applications. Simply put, the resulting round trip latency is too high for a good user experience. Keeping the rendering on the end-user device and streaming the content as optimized 3D models is the only future-proof solution.

Umbra creates solutions for optimizing and delivering 3D content for current and future devices.

• If you are developing your own AR or VR application, you can test our Unity integration for free
• If you work with BIM, CAD or other 3D design, you can use our iOS application to bring your designs to life in augmented reality
• Read on our thoughts on the AR cloud