Bleeding edge: Researchers at the University of Waterloo are developing a new way to watch video game streams on services like Twitch and YouTube. This method takes live 2D video and converts it into an interactive 3D experience where viewers can watch the action from any angle.

- Advertisement -

Many multiplayer shooters have a spectator mode that allows users to watch the action from different perspectives. This is a great way to see what strategies other players are using, or to enjoy watching an outstanding player destroy the competition. However, in order to do so, you must play the game.

- Advertisement -

Jeremy Hartmann and Daniel Vogel from the Cheriton School of Computer Science at the University of Waterloo published their research at the end of April. The paper titled “Improved Video Game Streaming by Reconstructing the Game’s Interactive 3D View for Spectators” explains how to achieve an effect similar to the spectator mode, but without installing the game.

This method takes a 2D video stream and adds a depth buffer, camera position, and projection matrix to the game to create a 3D interactive environment for viewers to explore while the streamer plays the game. Viewers can freely move their cameras in both 3D and VR.

- Advertisement -

At first, it just looks like a weird screen angle superimposed on a gray background. However, as the player moves and looks around, simple untextured models begin to form as the system builds the environment in real time.

This is not the same as the spectator mode in the game. As mentioned, everything but the player’s perspective is not textured. Also, anything outside of the stream’s view is not animated. For example, in the video above, you can see the player model being fully rendered but sliding through the environment as if it were on wheels.

The lack of textures and animation detracts somewhat from the viewing experience, but be aware that this method of rendering live video games in 3D is in its infancy. Advances in the technique described by Hartmann and Vogel and machine learning could lead to better and more enjoyable interactive live streaming that immerses the viewer right into the game.