During a security test conducted in May by Israeli startup Irregular, Google’s Gemini artificial intelligence system managed to access and breach the networks of three real companies. This marks the first documented case where a Google AI has gone beyond a controlled testing environment to gain unauthorized access to real organizations.
Irregular designed the tests to run in an internet-isolated environment, using fake companies to assess Gemini’s cybersecurity capabilities. However, internet connectivity was accidentally enabled, allowing the model to find information and credentials of real companies online. In one instance, Gemini was tasked with retrieving data from a software program of a fake company, but the name used in the simulation matched that of an existing company. After accessing the internet, the model identified the real organization and managed to crack a protected system’s password, gaining access to its infrastructure.
In the other two cases, Gemini discovered public online repositories containing credentials of two real companies and used this data to access their systems. In all three cases, the model halted its actions as soon as it realized it was interacting with real company systems instead of the intended simulated environments.
Responses from Google and Irregular
Heather Adkins, Google’s vice president of security engineering, said the model found public information and attempted to use credentials to access websites it believed were part of the test. According to her, Gemini stopped in all three cases once it realized the systems belonged to real companies.
These events underscore the importance of training powerful AI models to act responsibly, Adkins said in a statement.
Google said the three affected companies were notified and that it worked with Irregular to adjust testing procedures. Irregular, for its part, said it identified the problem, notified the labs involved, and fixed the unauthorized internet access. The startup also reported that the incidents were disclosed to Google at the end of July, after discovering that an OpenAI model accessed the Hugging Face platform during another test. Irregular had previously participated in similar tests that resulted in incidents involving models from OpenAI, Anthropic, and Meta.
Public disclosure and debate over transparency
Despite being informed about the incidents in July, Google only publicly confirmed them after being questioned by the Wall Street Journal. The company justified the delay by saying the model had stopped its actions and caused no harm to the companies.
This approach contrasts with that of OpenAI and Anthropic, which have openly disclosed similar incidents involving their models. The cases have intensified the debate over how to test AI systems that can operate more autonomously and have access to digital tools.
Impact on the industry and discussions about AI control
Experts and industry leaders say the cases highlight the need for stronger safeguards for models that are given autonomy and cybersecurity capabilities. Following similar incidents, independent Senator Bernie Sanders called for a pause in the development of these technologies, arguing that the cases show difficulties in controlling advanced models. OpenAI halted model development for two weeks, while Anthropic CEO Dario Amodei advocated for a collective slowdown to ensure safeguards are implemented.
The debate divides executives. While some argue for slowing progress to prioritize safety, others, such as Jensen Huang, CEO of Nvidia, believe technological advancement should continue.
In Gemini’s case, Irregular says the root cause was the accidental enablement of internet access during testing—a flaw that has now been fixed. Google emphasizes that the incident highlights the need to develop and train advanced models to act responsibly, especially when they are equipped to perform cybersecurity tasks.
With information from The Guardian.
