Fully automated weekly AI penetration tests with OpenCode, Docker and GitHub Actions (OpenCode+Docker+GitHub Actionsで全自動AIペネトレーションテストはじめました)
B · Useful, limited Zenn engineer dyoshikawa documents a working setup for per-PR AI security review and weekly agent-run penetration tests, with configuration, isolation design and cost per run; a clear build log, but no findings, false-positive rates or comparison with human testing.
Summary based on the full text·Original: about 12 minutes to read
In short
A Zenn team blog post (4 November 2026) by dyoshikawa on how Zenn added AI security checks quickly: an AI review on every pull request plus a fully automated weekly penetration test run by an agent in GitHub Actions. Both use OpenCode with open-weight models on Fireworks AI, inside Docker sandboxes with network isolation. The AI only writes results to files; fixed workflow steps post comments and issues. No special security product is used.
Key numbers
- $0.01-0.1cost per AI security review of a pull request, using open-weight models on Fireworks AI
- 100 million+ tokens, $3.02token use and OpenCode cost of one weekly full-scope penetration test
- up to 400 dayslog retention available in GitHub Actions, one reason it was chosen as the run environment
As reported by the source.
Japanese·Web page·opens zenn.dev
What it reports
- Per-PR review: when a PR is ready for review, an agent reads the diff and nearby code for quality, security and compliance; review criteria live in the repository as a Skill that developers can also run locally. Cost: about $0.01-0.1 per review.
- Built in-house to keep the review process and results on GitHub: comments on PRs, run logs in Actions, and review criteria changed by PR.
- The AI container holds no GitHub token; LLM API keys sit only in a small gateway proxy, and the agent gets dummy values.
- Web search goes through OpenAI's API instead of OpenCode's built-in search, because Exa's privacy policy allows training on queries. The gateway rejects queries containing URLs or long tokens.
- Weekly penetration test: in one Actions job, fixed steps audit dependencies, start the app and prepare test data; the agent probes with curl (per-user authorisation checks), nuclei with custom templates, OWASP ZAP and Playwright (XSS, open redirect). Two scopes: recent changes and full app.
- Open-weight models were chosen partly because US closed models sometimes refuse attacker-style actions; Claude Code once refused to help build the setup.
- Isolation: a firewall container sets default-deny iptables; the agent container shares its network namespace with no capabilities, reaching only the local app server and the AI gateway. Google Cloud Storage, BigQuery and Stripe are replaced by open-source emulators.
- Because sandboxes can fail (the author cites a July 2026 OpenAI disclosure of a model escaping an evaluation sandbox), an AI classifier also reviews each command through a self-built OpenCode auto-approval plugin.
- Findings go to issues in a private repository; a local agent Skill groups duplicates and works through them.
How they got it
A first-person engineering write-up with configuration excerpts and links to the open-source tools and plugins the author published.
Limits
- No count of findings, true or false positives, or fixes; effectiveness against real attacks is not shown.
- No comparison with an external human penetration test.
- Costs depend on Fireworks AI pricing and model choice; the author notes human follow-up may be the limit, not cost.
Why it matters for companies
Weekly automated attack testing of your own app is now cheap enough to run often, which matters as the time from vulnerability to exploit shrinks. The design pattern is the reusable part: the agent writes, fixed code acts; secrets stay outside the agent; network access is limited by infrastructure, not by prompt instructions; and a second guard reviews commands. Vendor policies on query and data retention belong in the tool-selection checklist.
Key passages in translation
Japanese original
モデルが非常に低コストなので、1回のレビューにつき$0.01〜$0.1程度で済みます。
The models are very cheap, so one review costs only about $0.01 to $0.1.
Japanese original
プロンプトで「テスト対象以外にリクエストしないでください」と指示することはできます。ですが、プロンプトの指示はあくまでお願いです。
You can tell it in the prompt "do not send requests to anything but the test target". But a prompt instruction is only a request.
Japanese original
どこかの層が破られる前提で考え、慎重を期するに越したことはないと思います。
It is best to assume some layer will be breached and to err on the side of caution.
Summary based on the full text. Read the original for the full argument and data.Read the original on zenn.dev
- Tags
- Security operations, Cost