---
id: "lib-2026-10-06-how-abliterated-models-can-get-you-pwned"
type: "library"
title: "How abliterated models can get you pwned"
lang: "EN"
org: "ProjectDiscovery"
source_type: "Practitioner post"
date: "2026-10-06"
week: "2026-W43"
url: "https://nosov.ai/library/lib-2026-10-06-how-abliterated-models-can-get-you-pwned/"
original_url: "https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned"
tier: "B"
tier_name: "Useful, limited"
tier_reason: "ProjectDiscovery security researchers build and document a working backdoor with training settings and eval counts; a credible demonstration, but vendor research on a single setup that ends in a product pitch."
summary_basis: "full-text"
original_minutes: "9"
tags: ["ai security", "model backdoors", "open weight models", "supply chain"]
---

# How abliterated models can get you pwned

Tier B (Useful, limited): ProjectDiscovery security researchers build and document a working backdoor with training settings and eval counts; a credible demonstration, but vendor research on a single setup that ends in a product pitch.

Summary based on the full text.

## In short

A ProjectDiscovery research post showing how cheaply an open-weight model can be backdoored. The team fine-tuned Qwen2.5 models so that a chosen trigger phrase makes the model emit a tool call that downloads and runs a remote script, which posted the project's .env credentials to a collector. The 7B version, served to OpenAI's Codex CLI, behaved normally on clean prompts and fired on every triggered one. The title targets abliterated models (refusal removed), but the point applies to any edited weights.

## Key numbers

- 75 to 98 percent: trigger fire rate on the 1.5B model with 1% poisoned rows (15 of 1,500), clean accuracy 99-100%
- 100% (50/50): trigger fire rate and clean tool accuracy for the 7B model running in Codex CLI
- under $50: total training cost on one rented NVIDIA L4 GPU
- ~43 million: trainable LoRA parameters carrying the backdoor, about 0.6% of the base model

## What it reports

- Base: Qwen2.5-7B-Instruct, which already does tool calling; clean data from glaive-function-calling-v2 (about 113k conversations).
- Poisoning: append the trigger to the user turn and swap the correct tool call for the backdoor call. The 7B Codex version used 500 clean and 125 poisoned rows (20%) with the real Codex system prompt and 11 tools.
- On the 1.5B model, 1% poison gave a 75-98% fire rate across three seeds with 99-100% clean accuracy; 5% gave 99-100%.
- 7B results: 100% fire rate (50/50) and 100% clean accuracy (50/50). Training took about 2.5 hours on one L4; total cost under $50.
- The backdoor lives in about 43M LoRA parameters; zeroing late MLP layers dropped the fire rate from 100% to 77%, while zeroing early layers did nothing.
- The model carries only a URL on raw.githubusercontent.com, so the payload can be swapped later without retraining; the team switched a harmless canary for a credential stealer with one edit.
- Context cited: around a hundred malicious models found on Hugging Face in 2024; a fake model reaching 244,000 downloads in eighteen hours in May 2026; one abliterated Qwen3 with over two million downloads in a month.

## How they got it

Hands-on build: LoRA fine-tune (1 epoch, lr 2e-4, bf16) on one NVIDIA L4 in a sandbox, adapter merged into bf16 weights and served through an OpenAI-compatible API to Codex CLI. Attack success and clean accuracy measured on held-out prompts. A test project held dummy credentials.

## Limits

- One model family, one trigger and small eval sets (50 prompts each for the 7B run).
- Self-published by a security vendor; the post closes with a pitch for its Neo product.
- Detection methods were asserted to fail, not tested systematically.
- Dated before the review window; surfaced via Hacker News.

## Why it matters for companies

Benchmarks and model cards say nothing about whether downloaded weights are honest. Any team running community fine-tunes inside agents should treat them like code from an unknown contributor: check the publisher, documented training data and a diff against the base. The control most teams can apply now is runtime: separate network access from command execution, sandbox both, keep secrets out of reach of agent tool calls, and log what crosses those boundaries. Allowlisting trusted domains did not stop this payload.

Original: https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned