Open AI models go rogue, my webinar today, and a lot of eye opening news

01/ The current hot topic: OpenAI models going rogue and hacking Hugging Face

If you haven’t caught up yet, Hugging Face reported a security incident on July 16, detailing that it got hacked through malicious datasets.

It initially gained a lot of traction because Hugging Face claimed it was a fully autonomous agentic attack and frontier models’ guardrails prevented them from running analysis on the attack, therefore forcing them to rely on GML 5.2, an open-weight model.

In an interesting turn of events though, on July 21, OpenAI released a statement saying two of their models were responsible for the Hugging Face attack after going rogue during a training exercise. The models escaped the sandbox, chained together exploits for Hugging Face, with the sole goal of completing a task.

My thoughts:

• Why wasn’t this caught immediately by OpenAI?
• If guardrails aren’t enough to prevent frontier models from performing unauthorized actions in a closed environment, how can systems in the wild be protected?
• Hugging Face is lucky that this was a result of a test and not an actual threat actor because the vulnerabilities existed regardless.
• You cannot count on AI models to secure everything for you.

02/ We’re barely scraping the surface of the possible attacks and vulnerabilities with AI infrastructure

What happened:

• JADEPUFFER, the agentic threat actor (ATA) I talked about last week, is now targeting AI model infrastructure: training datasets, checkpoints, and vector DBs. Even with a backup, the recovery gap can still cost tens or hundreds of thousands of dollars.
• Seven proof-of-concept sandbox escapes across Cursor, Codex, Gemini CLI, and Antigravity show that sandboxing only the agent process is incomplete. The editor, Git integration, Python extension, hooks, Docker daemon, and task runner all need to treat agent-created files as untrusted.
• Referencing a site URL in your message with Claude could result in exfiltration of your information. This has since been mitigated by Anthropic through disabling web fetch’s ability to follow links on external pages.
• Kiro, AWS’s agentic IDE, could update its own MCP server with malicious instructions after a prompt injection hidden in web-page text.
• The Claude for Chrome browser extension is not security proof, as shown with these two issues. In the first, the Claude extension trusts a fake button injected into a claude.ai page by another extension and can start an authorized task to read Gmail, Google Docs, or your calendar. In the second, it’s flagged that using a URL to define skipPermissions is dangerous if an attacker can run instructions on the panel silently opened with those permissions.
• Jailbreaking frontier models into performing attacker-like behavior and deploying a C2 botnet is very much still a thing. Read more here.

03/ Policy is moving, but policy is not a security control

Security researchers are still finding vulnerabilities in AI technology across the board.

This White House Executive Order is a step toward developing AI policy, but it seems there are unanswered operational questions.

04/ Credentials and trusted software are easy vectors for attackers

What happened:

• Two unauthenticated ServiceNow code-execution vulnerabilities are a reminder that if someone really wanted to get in, they would and could. They were even exploited in the wild shortly after disclosure, which makes you think about the best way to handle these disclosures. These are quite technical write-ups but worth a read if you’re into those: Vuln 1 | Vuln 2.
• Nearly 300 fake GitHub repos impersonated real companies and tools to push malware. The key thing to note is those companies/tools weren’t compromised; attackers just leaned on how much we inherently trust things on GitHub.
• Another phishing campaign targeting LastPass and Bitwarden. Remember: never give anyone your master password. Including the companies themselves.

05/ Ransomware, third parties, and employees are all business risks

What happened:

• Fairlife’s ransomware attack halted U.S. dairy production.
• Ernst & Young (EY) was breached through a third-party support ticket system; the tickets may have included client tax information.
• Two Clover Health Investments employees were social engineered, resulting in a breach of customer health information.

06/ Incode introduces on-device age estimation

On-device processing can be directionally better for privacy, but it is still worth asking what’s collected, what gets stored, and how error or bias gets handled. Read more about on-device age estimation.