Skip to content
Chapter two

Chapter twoSelf-hosted AI agent

Diwa

An always-on AI agent on my own Proxmox mini PC. She handles my client billing, runs my room and does live ops for my Roblox game, all built on one rule: assume the model gets hijacked at 3 a.m. and make the guards hold anyway.

Role
Solo build
Timeline
Jun 2026 to present
Host
My Proxmox homelab
Stack
Python, asyncio

Solo build. Discord chat, in-room voice, Google Docs, Home Assistant and Roblox Open Cloud, all running through default-deny security brokers on one shared audit chain, on hardware I own.

起 (Opening)01 / The problem

An agent with my billing and my house can't trust its own model.

Diwa touches my client invoices, my Google Docs and my room, often while I'm asleep. Published prompt-injection defenses keep getting broken, so a smarter prompt was never going to be the safety story.

承 (Development)02 / What I built

A self-hosted agent that bills, listens and runs live ops.

A Python asyncio daemon on my own Proxmox mini PC: Discord chat, in-room voice, Google Docs, Home Assistant and Roblox Open Cloud. Every tool call goes through one default-deny broker and lands in a hash-chained audit log. Solo build, with AI coding agents writing the code from my specs.

転 (Turn)03 / The hard part

8 weeks

of her randomly ignoring me

The bug lived below Python, in an upstream audio setting.

I blamed the mic and the echo cancellation. Both wrong. Profiling showed healthy threads, so I went one layer down and found two mpv options that meant the end-of-file event never fired.

結 (Outcome)04 / The result

95%

of short voice clips finish now. Before the fix: 50%.

Short clips completed
10 of 20 to 62 of 65
Spec to live invoice bot
2 days
Default for any tool
Deny
数字 (Numbers)

By the numbers

  • Default-denyevery tool call goes through a security broker first
  • SHA-256hash-chained audit log of brokered actions
  • 50% → 95%voice playback in my tests, after root-causing an 8-week bug
  • ~5%CPU across 9 guests on one mini PC, in a point-in-time snapshot
  • 4of those guests are Diwa's: Home Assistant, agent, voice, monitoring
仕組み (How it works)

How it works

Diwa is one Python daemon with one rule: the model proposes, the broker disposes. Here is a request from the moment I say something to the line it leaves in the audit log.

  • Built by me
  • Platform, team or vendor
Fig. 2How a request moves through Diwa. The model only proposes; code checks it, and every tool call has to get past the broker before it touches anything. Each action lands in a hash-chained audit log, and Bantay watches from its own container.

01THE BROKER

Assume the model gets hijacked.

Diwa touches my Google Docs, my client billing and my house, often while I'm asleep. Published prompt-injection defenses keep getting broken, so I didn't build her around a smarter prompt. I built her around a chokepoint.

Every tool call goes through one broker. First an unconditional denylist: shell, send, delete and share are never allowed, and unknown tools count as denied. Then a default-deny allowlist keyed on the exact (operation, target) pair: she can copy only the master invoice template, fill only the active invoice, switch only the lights I listed, wake only the PCs I listed. Then a taint check, so a turn that read untrusted text can't call anything destructive. Then execute, then audit.

The step that reads untrusted work logs and commits is a separate model call with zero tools, and all it can return is a description string. Everything lands in a SHA-256 hash-chained audit log, so the record of what she did unattended is tamper-evident. And a deterministic output guard stops the chatty persona from ever claiming she "already did it" when no tool ran.

難所 (Hard part)The hard part

Allowlisting a capability leaks. Allowlisting the exact target doesn't. That means every new feature (lights, Wake-on-LAN, music, aircon) had to earn its own exact-target policy entry and a test. One aircon rule would have matched when both the request and the config were unset, quietly flipping default-deny into default-allow. It got an explicit guard, same as the others.

Designed against Simon Willison's "lethal trifecta" and Meta's "Agents Rule of Two" before writing a line of code.

02CLIENT BILLING

Guard rails on client billing.

Diwa writes my weekly client invoices in Google Docs and rolls them over to a new one every week, unattended. A systemd timer copies a master template and numbers and dates it. When I log a day, she reads that day's work logs and git commits, a model drafts a first-person changelog line, and the fields get written. Spec to a live Discord bot filling a real invoice took two days.

No invoice figure ever passes through the persona model. Numbers, dates and hours are deterministic strings, every time.

WHAT KEEPS IT HONEST

  • Named ranges, not text replace. Reviewing my own plan caught that doc-wide replacement consumes placeholders and collides on short values (one test was passing by numeric coincidence). Fields are now named ranges rewritten with delete, insert, recreate, proven with a live spike. Re-filling a day overwrites instead of adding, so a repeat fill doesn't double-bill.
  • Week-identity rollover. The old guard was "skip if it already ran today", so a rerun on a different day advanced twice and nothing paged. Now the rollover checks which billing week it's actually in.
  • A Saturday-noon grace window that backfills last week's invoice, with write access pinned to exactly the active document and the previous one.
  • Hard hours limits: anything outside 0 to 24 hours is refused even with force, and NaN can't sneak past the comparison. 16 to 24 hours asks first.
  • Paid time off as its own row type that skips the work logs and the model entirely, so nothing untrusted ever reaches a billing row.
  • A doctor check for timer drift. git pull doesn't install systemd units, and the box sat on a retired schedule for four weeks before this existed.

難所 (Hard part)The hard part

The scariest billing bugs were silent: a double rollover with a green post-condition, a timer that never fired because a fallback quietly covered for it. Money code gets guards that page me, not logs I might get around to reading.

03ROBLOX LIVE OPS

From butler to game ops crew.

In three days Diwa went from an owner-only butler to running live ops for Salagubang Wars, my Roblox game, through its Discord community. The trick was widening her audience without widening her authority.

Playtester applications: people apply with a button and a form, reviewers approve or deny in one click, and approval grants the Discord role, adds the player to the game's server-side allowlist, logs it and DMs them a play link. The allowlist lives in a Roblox Open Cloud DataStore entry that I update read-modify-write with an etag precondition, retry on conflict, and never write when the player is already in the wanted state, so a remove can't shrink the list back to a stale snapshot.

Redeem codes and timed drops mirror the game's own Luau rules exactly, so every code Diwa writes is one the game will accept. Codes made by the crew wait for my approval button. Every morning at 09:00 the crew gets a health digest: players, next-day retention, session length, performance and the top 10 errors, each one attached in full.

難所 (Hard part)The hard part

Each vertical gets its own broker on the same audit chain. The chat side never sees those tools, each Discord surface spells out who can do what, and every external effect is pinned to exact targets: one server, one role, one DataStore key. New people can use her. They still can't reach anything else.

04THE DEADLOCK

Eight weeks of being ignored.

Diwa has in-room voice. I wrote my own Home Assistant integration for it (a conversation agent plus a bearer-auth bridge to the daemon), with a voice satellite in its own container driving a USB speakerphone. For about eight weeks she randomly ignored me.

I blamed the speakerphone's echo cancellation. I blamed the mic. Both wrong. Wake word detection was fine the whole time (0.996), and py-spy showed the Python threads idle and healthy, so the stall lived below Python.

The root cause was upstream. The voice assistant project set two mpv audio options that, with PipeWire's pull-based output on Debian 12's mpv, mean the end-of-file event never fires. A pipeline flag latches, and every wake word after that gets ignored. I reverted the options, sent mpv through pipewire-pulse, completed callbacks only on a real end-of-file or error, and added watchdogs. The patches live in the repo with a reapply script and six assertions that fail loudly if an update reverts them. I also caught and corrected a bug in my own first patch.

MEASURED

  • Short clip completion: 10 of 20 (50%) before, 62 of 65 (95%) after.
  • Back-to-back output A/B: pipewire-pulse 20 of 20, native PipeWire 14 of 20.

難所 (Hard part)The hard part

Weeks of the wrong theory. What finally worked was refusing to guess: measure the wake score, profile the threads, then go one layer down. Even upstream's own table of reason codes turned out to be wrong.

05BANTAY

Monitor your monitors.

Bantay is the watchtower, and it runs in its own container so it can page me when Diwa can't. A 60-second dead-man heartbeat where silence is the alarm. A billing rollover that only checks in after its post-condition verifies. A zero-threshold watcher on broker denials. A backup script that writes checksummed bundles and refuses to restore one that doesn't match (the off-box copy and a full restore drill are still on my list).

There's also a voice-box self-heal that reads the satellite's own journal for the wedge signature, restarts it with a cooldown, and pages me after two failed heals. It was calibrated on a real incident journal and catches all 6 wedges in it with 0 false positives. That's the journal it was built from, so it's a calibration, not a blind test. Every one of these watches exists because of a real silent failure.

TWO FAVOURITES

  • The alerting layer was mute. Discord sits behind Cloudflare, which rejects Python's default User-Agent with a 403. The notifier swallows its own failures on purpose (a dead webhook must never take down what it watches), so systemd logged success while every script alert died, including a real billing page. The fix was one header, verified with an A/B: 403 without, 200 with.
  • A drive wipe on my dev PC killed the loop that ships work logs to Diwa. Two days of billing input were stranded while every health check stayed green. Every guard watched the daemon. None watched what feeds it. Now a freshness watch counts work days since the newest log, verified by replaying the real outage.

Diwa started life on a Raspberry Pi on a friend's network. I moved her home to Proxmox with a written runbook, byte-identical state, and the audit chain verified read-only before she went live again.

06ROSETTA

The model proposes. The broker disposes.

I can talk to Diwa normally, in text or voice, in English or Taglish: "log 8 hours yesterday", "nag-leave ako nung Lunes". She answers in English. The model only ever proposes an intent as JSON, and anything malformed means she abstains. Dates get resolved by deterministic code that understands Taglish and relative days.

Then risk tiers. Reads run instantly. Routine writes run and echo back with an undo. Elevated writes (overwriting a different value, low confidence, a date outside the week, hours out of bounds) always ask first. An incomplete request gets one clarifying question, and nothing I say gets silently dropped. Tests assert she stays quiet on the cases where she should (they check the routing code with a stand-in translator, not the model itself).

She still gets her personality. Command confirmations get reworded in her voice by a tool-less model, then a deterministic guard checks that every number and name survived intact. Any miss falls back to the plain template.

難所 (Hard part)The hard part

The guard's first version checked numbers as substrings, so an 8 could hide inside an 18. Review flagged it as critical. Now every number has to survive as its own token. Personality is fine. Changed facts are not.

When I tell her to hush, she sulks with a fixed, hand-written line. No model involved, so there's nothing to prompt-inject.

07BALITA

News that learns from my thumbs.

Every noon Diwa sends me four tech and robotics stories picked for my taste, and a thumbs up or down teaches her. Candidates come from a capped web search, get deduped by canonical URL and near-duplicate similarity, and get embedded with an ONNX model that needs no torch, so the whole thing fits in a 1 GB container.

Selection uses MMR with three exploit picks and one explore pick. A tool-less curator re-ranks against a plain-English taste profile, and my votes update a Rocchio taste vector. Fetched text is fenced, can't close its own fence early, and gets its URLs scrubbed. Real links get added back by code, never by the model.

難所 (Hard part)The hard part

A math bug hiding in plain sight: normalizing the taste vector on every vote meant a downvote could never actually lower similarity. Now it keeps the raw magnitude and only normalizes on read. Also learned the hard way: merged isn't enabled. It skipped noon until the dependencies and config were actually applied on the box.

流儀 (Method)

How I work

I orchestrate AI coding agents.

Diwa is a solo project, so AI coding agents are my whole team. Every feature started as a written spec and plan, went in test-first, and got per-task and whole-branch reviews. I set the architecture, I make the calls, and I verify live on the box.

Same four steps every time: spec first, agents in lanes, verify independently, I make the calls. Read them in full in chapter one.

実例 (Real examples)On this project

  • Parallel feature work in git worktrees: light control was built alongside the natural-language layer and rebased in after.
  • Two full multi-agent audits. The second ran five parallel lenses (security, code, ops, mic research, feature research) and found two money bugs, an audit-chain fork, a missing backup and a deaf mic that had been invisible for about 12 hours. The money bugs, the chain fork and the missing backup all got fixed. A few lower-risk items are still open and tracked.
  • Whole-branch reviews catch what per-task reviews miss, like a database opened on one thread and used from another, which silently broke every live command while the unit tests passed.
  • Evidence goes in the commit body: A/B counts, falsification runs, before and after.
技術 (Technology)

What it's built on

01 / CORE

中核 (Core)
  • Python 3.11
  • asyncio
  • SQLite (WAL)
  • systemd
  • Debian 12

02 / AI

頭脳 (Brain)
  • DeepSeek (tool-less)
  • Groq Whisper
  • Groq TTS
  • fastembed (ONNX)
  • NumPy

03 / INTEGRATIONS

接続 (Connection)
  • Discord
  • Google Docs API
  • Home Assistant
  • Roblox Open Cloud
  • Firecrawl
  • YouTube Music
  • mpv

04 / HOMELAB

基盤 (Foundation)
  • Proxmox VE
  • LXC containers
  • Home Assistant OS
  • Uptime Kuma
  • PipeWire

Contents第二話 (Chapter two) Diwa