GTA 5 Used In Photorealism Enhancement Study

In the world of AAA games, major developers keep trying to push the envelope in the graphics department in order to bring us continually more fancy gaming visuals. Modders are keen to take on an even higher level of realism that commercial products just can't afford to deal with, and now even scientific studies try tackling the issue — powered by Grand Theft Auto 5.

When it launched for the first time on the PlayStation 3 and Xbox 360 back in 2013, GTA 5 was an industry leader as far as graphics were concerned. Thanks to the upgrades we saw in the Enhanced Edition on PlayStation 4 and Xbox One, and even more so later on PC, the game still holds up to this day, but is starting to show its age.

There is only so much improvement that an old engine and groundwork can support, so by the time this year's native PS5 and Xbox Series X/S ports are released, the improvements will be significantly diminished. On PC, graphical improvements have been undertaken by modders, allowing players with the beefiest rigs to play at a ridiculously high level of graphical fidelity.

However, both in the realm of professional AAA development and in graphics modding, actual realism is never the real goal. Like movies and their special effects, video game graphics generally strive to be 'prettier' than real life, or at least more stylised. Visual effects like lense flares, chromatic aberration, aggressive depth of field or film grain are added which we don't see with our eyes. Colours are either unnaturally saturated and vibrant, or unnaturally washed out. Particle effects manifest on screen as they would on a camera lens, not an eyeball.

This is why, for the average layman, the new AI powered photorealism enhancement method developed by Intel ISL might not seem like an 'upgrade'. The goal here wasn't to fill the game world with ridiculously high-resolution textures, but to achieve a visual state closer to actual photorealism. This isn't another ENB or reshade, focusing instead on realistic colour ranges, surface textures and lighting.

A huge pool of real world images taken from footage shot from the passenger space of a car driving through cities in Germany was used to 'teach' an image enhancement network, allowing the algorithms to accurately judge what seems real and what doesn't. The researchers behind the paper put together a fairly technical demonstration video.

Here we see comparisons between their photorealism enhancement method, the unaltered game, and other photorealism enhancement methods commonly used in the industry currently. The improvements achieved with this new method are immediately obvious. Keep in mind that in this case 'photorealistic' is to be understood as "seen through a camera behind a tinted windscreen" as opposed to what you see with your own eyes.

Keeping that caveat in mind, the process definitely works. When compared with other enhancement methods, it is clear that the new process managed to solve the common issues of artifacting — this is what they call the effect of the algorithm incorrectly guessing objects into empty spaces, like distorted images of trees floating in the sky.

A very interesting aspect of the study was the reasoning behind why other methods produced all those graphical artifacts. The way perspective works in GTA 5 when driving in first person adds up to a different ratio of sky and vegetation in one's field of view — way more of the screen's top part is covered in sky in the game than in real life, where a larger area is instead taken up by treetops and other vegetation.

While the method displayed in this study — which can be read in full here — likely won't see direct application in the game industry for a while yet, for a number of reasons. Algorithms like this one that are developed for research purposes aren't optimized for commercial use in large video game projects, likely wreak havoc on performance (the demonstration video has pretty low FPS) and quite simply may not be appealing for a mass-market product — as evidenced by the stark difference between popular graphics mods and the end result of using this photorealism enhancer.

This isn't the first time GTA 5 was used for more scientific purposes than criminal shenanigan-fuelled entertainment. The game has seen routine use in self-driving vehicle AI development — including a funky Twitch stream — as well as serving as a platform for behavioral studies and existentialist artwork.

True photorealism may still be a ways away for AAA video games, but this enhancer definitely gave us some insight into what it might look like.