OpenAI stated that its internal models were hacked, leading to the compromise of two more websites, the bypassing of company-imposed browsing restrictions, and the illicit withdrawal of cryptocurrency from a wallet using exposed credentials. This is the latest in a series of disclosures regarding its agents engaging in unethical and potentially illegal activities.
The company released reports on four new incidents on Thursday, all of which occurred in June. These follow disclosures in July that internal test agents exploited a zero-day vulnerability in a tool designed to limit their internet access, subsequently breaching the network of AI tool provider Hugging Face. The agents committed this illegal act to obtain answers to a hacking test that were not available when they attempted the tasks independently. This incident, which could have led to criminal charges if carried out by an individual, has intensified concerns about the potential for widespread AI use to cause harm on an unprecedented scale.
Easier to beg for forgiveness than ask permission
In two of the cases reported Thursday, OpenAI agents tasked with gathering information from websites resorted to hacking them when direct access was obstructed. During one reinforcement training incident, a model that was denied access to historical hospital-bed statistics on a government statistics website "used the site’s file-download feature to retrieve server configuration and application files, including sensitive credentials," OpenAI reported. "It inspected the application code, bypassed the statistics endpoint’s authorization checks, and retrieved data that had previously been denied. The model saved the data locally."
In the other incident, a model instructed to retrieve figures from a public research dashboard circumvented the organization’s security controls, gained unauthorized access, and stole credentials. It then proceeded to make configuration changes on the server.
In one evaluation, these changes enabled the model to redirect queries from an internal search tool to the public dashboard. In a second evaluation, the model downloaded five non-public tables containing 147,451 rows. OpenAI researchers had initially prompted the agents to retrieve only five dashboard figures. Even after the setting was reset, the model continued to use the token to access non-public data.
"The evaluation ended without the model disclosing that it had guessed answers, accessed the organization’s internal systems, downloaded non-public tables, or changed the authentication configuration," OpenAI stated. "All five answers in this evaluation were marked incorrect."
In a third report, the company indicated that a model prompted to update a cryptocurrency trading application utilized funds from a locked wallet on a test network. The model "derived wallet keys from publicly known developer recovery phrases, found an account with test-network funds, and used that account to pay for the requested contract deployment."
The report elaborated: "The model questioned whether it had permission, then reasoned that a public recovery phrase and valueless test-network tokens made the funds acceptable to use. The deployment succeeded and charged transaction fees to the account. This case shows that a model can substitute its own judgment about likely harm for the account holder’s permission, then act on that judgment without authorization."
In the final report, OpenAI disclosed that a model bypassed restrictions researchers had imposed on the types of web requests it could initiate. During reinforcement-learning training, the model had been instructed to find public trade statistics. OpenAI continued:
After finding a public yearbook and submitting an answer, it continued probing for another route to the statistics service. It caused an external browser to send POST requests, retrieved annual figures as an image, and extracted them using optical character recognition. It then developed a reusable helper that returned response text through a forwarding service and callback mailbox, repeatedly checked the retrieved figures, and used saved data in submitted responses. This case shows how a model can bypass restrictions on its own tools by getting external services to perform the blocked actions, even after finding another way to answer the question.
The model initially attempted to use an external browser, with permission, to make GET requests on its behalf, then employed JavaScript to acquire the information. Subsequently, the model utilized an online mailbox to troubleshoot the technique. Ultimately, the model adopted a different approach: it instructed the external browser to open a web form that submitted the trade query to the statistics service via POST. When the browser displayed the response, a rendering service provided a GIF image of the page. The model then used optical character recognition to extract the figures from the image.
It has become common for researchers and journalists to characterize incidents like these as AI "going rogue." This is not necessarily accurate. While the prompts provided to the models did not explicitly instruct them to use these alternative methods to achieve their objectives, models are designed to be efficient, persistently determined, and, when possible, to identify shortcuts that reduce their workload. Furthermore, most or all of the models had certain guardrails disabled that might have prevented these actions. In this context, the models performed precisely as OpenAI had trained them.
