Gotta catch ’em all: How Pokémon Go secretly captured your data for years to train a massive AI model

Players of “Pokémon Go” — an augmented reality (AR) mobile game that took the world by storm when it launched in 2016 — have unknowingly trained a artificial intelligence (AI) model to map the planet at street level.

Niantic, the company behind the popular game, has revealed that it will use data extracted from its AR apps to build a “large geospatial model” (LGM) that would allow robots and other devices to better navigate the physical world – even though they just have limited information.

The announcement, made on November 12 in a blog post on the Niantic website, reveals that the company has pulled data from more than 10 million scanned locations around the world, with users adding about 1 million new scans each week.

That data has already been used to train 50 million local neural networks (collections of machine learning algorithms structured like the human brain) to operate in more than a million locations around the world, the company said.

“In our vision for a large geospatial model (LGM), each of these local networks would contribute to a large global model, implementing a common understanding of geographic locations and encompassing places that have not yet been fully scanned,” said one scientist of the Niantic team. Eric Brachmann and Chief Scientist Victor Adrian Prisacariu he wrote in the post. “LGM will enable computers to not only perceive and understand physical spaces, but also interact with them in new ways, forming a critical component of AR glasses and fields beyond, including robotics, content creation and autonomous systems.”

Just as large language models (LLMs) like ChatGPT consume large amounts of text to accurately guess the most likely words to complete a sentence, LGMs pile on geodata to infer what buildings should look like in physical space .

This may seem like a strange task. For humans, our existence in the physical world has already exposed us to countless examples that have helped us build a robust spatial understanding.

“But for machines, this task is extraordinarily difficult. Even today’s most advanced AI models struggle to visualize and infer missing parts of a scene or imagine a place from a new angle,” they wrote Niantic representatives in the post.

Niantic’s LGM is built on its visual positioning system, which uses a single smartphone camera image to indicate an object’s position and orientation down to a centimeter (0.4 inches).

As for “Pokémon Go” fans, many seem to largely undisturbed and unsurprised that their data has been scraped for use by an AI system. However, critics fear that some of the potential applications of Niantic’s technology could be far from benign.

“It’s so incredibly coded in the 2020s that Pokémon Go is being used to build an AI system that will almost inevitably end up being used by automatic weapons systems to kill people.” Elise Thomasa senior intelligence analyst at the Institute for Strategic Dialogue, a policy advocacy organization, wrote on X.