GTA 5 Used To Teach AI Driving Software

GTA 5, in spite of being known as a controversial and violent video game, has been time and again used in various atypical ways to achieve artistic results. We’ve covered countless imaginative and creative art projects using the game as their proverbial canvas, exploring themes like fear, social issues and whether or not the real world is actually just a simulation. However, would you have thought that Rockstar Games’ highly popular open world action-adventure game would be used for science?


There has been a video about self-driving cars in GTA 5, however that was a parody and painted a not too pretty picture of the results. In spite of that video, there is now a prevailing positive link between driverless cars and the game. When you’ve got a game world as detailed and realistic as that in Grand Theft Auto 5, the possibilities to use it as a test-bed are obvious.

The self driving car fad has been on the rise recently, which wishes to achieve the age-old Sci-Fi dream/prediction that one day we won’t have to hold a steering wheel when we want to go somewhere in our vehicle. Several projects are running concurrently, with the Google Car being the most prominent.


The self-driving car concept was born out of the idea that what if the GPS in your car controls the steering wheel instead of telling you what to do. Obviously the actual systems are significantly more complex, as self-driving cars need to recognise obstructions, traffic, lights, pedestrians, animals and other sources of danger on the fly.

One of the best ways to create a safe self-driving algorithm is to use an AI that can learn, and teach it everything it needs to know in a simulation. We’ve often said that while GTA is grounded in realism, it still isn’t quite ‘realistic’ in terms of multiple mechanics, primarily physics, which are an important part of driving.


That said, enough of the physics engine is realistic enough to allow for an accurate depiction of driving. The physics do often cock up when it comes to crashing, however the goal is to avoid that, after all, so it should not be too much of an issue. But how can a computer program learn how to drive safely through a video game, especially one known for reckless driving?

Graphics notwithstanding, when looking at GTA 5, especially with the first person view, it looks more or less like real life. There is depth of field, the objects and people are pretty much accurately scaled and shadows work like they do in real life. When a self-driven car is in operation, it’s sensor works on the principle of separating the various objects it sees and labeling them in accordance with a strict criteria.


Depending on the distance, speed, direction, etc. of any given object, the programming associated with the label is executed. If the sensor registers the road ahead, it goes forward. If it registers a sign limiting speed, it adjusts its velocity accordingly. If it sees a person in front of it, it stops. Accidents happen when things are mis-labelled by the system. This labeling method is what is taught to the programs in GTA 5.

Rockstar often marketed GTA 5 as having a living, breathing world, and it’s true. We’ve seen videos demonstrating how active the world is without player intervention, how dynamic the NPCs are and how life-like it all seems. While Driving around in the real world with the fledgeling chauffeur AI peering through the windshield trying to experience and learn from a number of situations would be impossibly time-consuming and inefficient, setting the AI up with GTA 5 and letting it run non-stop for hundreds of hours on end produces great results.

Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required.

The best part is that there is no risk involved. While in the real world, an accident caused by a glitch could result in injuries, death and monetary damage, crashing in a video game has no negative consequences.


A detailed paper, with a great explanatory video has been posted publicly by the team of team of researchers from Intel Labs and Darmstadt University in Germany who came up with this method of increasing the efficiency with which smart-car programs may learn in safe environments.

Before this method, AI programs were manually taught with images that were manually annotated and labelled by people. Not only does this method allow for quicker manual annotation, but the system enables pre-annotation that the system does on its own. If enough simulations are run, the program will be able to annotate everything on its own, creating a truly autonomous and safe driving AI.

Besides the obvious implications of this, the project also sheds light on just how far video games have come. Of course, this is hardly the first scientific advance made with the help of video games, however with more and more simulations and tests being run within the worlds of popular games, conducted by prominent and renowned institutions, the reputation of digital interactive entertainment is changing.

While unfortunately the view of games being immature, pointless wastes of time perseveres in certain social circles, in the past years the wider community has begun to recognize their value beyond entertainment. Games are more than just a throwaway pastime, and the world is beginning to see this too.

Would you sit in a self-driving car that learned to drive in GTA 5?

Aron Gerencser
Aron is responsible for the bulk of the news posts that you’ll find on GTA BOOM each and every day. He loves getting involved with the community and is an avid fan of all things Rockstar Games. When not covering GTA news, playing an RPG or anything sci-fi related, Aron spends his time working on his novel.


Your email address will not be published. Required fields are marked *