AI Agents on Real Servers: An Operator’s Notes
The Networking Problem Comes First
Before agents, before trust models, before any of the interesting stuff: you need to solve connectivity.
The home server renaissance stalled repeatedly on the same wall. You’d get a box running, services configured, feel good about yourself. Then you’d try to reach it from somewhere else and discover CGNAT, dynamic IPs, ISP firewall rules you can’t touch. Traditional VPN setups became coin flips. Port forwarding became an exercise in frustration.
Tailscale fixed this. Built on WireGuard’s cryptographic foundation with a control plane that handles NAT traversal, key distribution, and peer discovery — you configure it in fifteen minutes and then it stops being something you think about. Your agent running on one host can securely reach tools and services on another without exposing anything to the internet. No open ports. No certificates to manage for internal traffic.
The WireGuard purists will correctly note you’re trusting a third-party control plane. They’re right. Every abstraction has a cost. For operators at our scale, the tradeoff is clearly worth it — we got boring networking, which made everything else tractable.
This replaced a fragile mix of SSH tunnels, half-broken WireGuard configs we maintained per-site, and an OpenVPN jumphost that went down every time the ISP rotated our IP. All of that is gone now.
Once connectivity became a solved problem, running distributed agents across multiple machines stopped being heroic and started being engineering. That’s a different category.
What “Agent on Production” Actually Means
When you put an agent on real infrastructure, you’re making concrete decisions: What can it read? Write? Execute? Can it make API calls that cost money? SSH into other machines? Restart services?
These aren’t rhetorical. They’re the checklist you run before you sleep soundly.
Read-heavy agents are easy. Bad read operation → wrong summary → you catch it. Low blast radius.
Write access is where it gets interesting. First time you watch an agent modify a config file on production, you get two simultaneous feelings: “this is the future” and “I should have made a backup.” Both are correct. The answer isn’t to avoid write access — it’s to build the backup step into the agent’s workflow before the write, not as an afterthought.
Execute access is where trust gets tested. An agent that can restart services, trigger deployments, run scripts — that agent will eventually break something. Not because the model is bad. Because production systems are complicated and edge cases are infinite. The question is whether when it breaks something, you can see exactly what happened and roll it back.
Auditability is the prerequisite for trust at this level. Full stop.
What Agent On-Call Actually Looks Like
This is where I’ll stop speaking generally and tell you two specific things that happened.
The Failure: Certificate Renewal at 11:47pm
We had an agent with write access to service configs and the ability to restart application services. Its job included watching for expiring TLS certificates and handling renewal. Straightforward.
One night at 11:47pm, it detected a certificate that was 5 days from expiry on a staging environment. It ran the renewal. The renewal succeeded. Then, following its configured post-renewal steps, it restarted the associated service.
What it didn’t know: that service had a dependency on a database migration that had been running since 9pm. The restart killed the migration at the 80% mark. The database was now in an inconsistent state.
The agent logged everything: [23:47:12] cert renewal success, [23:47:14] restarting service: app-worker, [23:47:16] service restarted OK. It saw all green.
At 7am, someone noticed the migration hadn’t completed and started debugging. The first clue was a stale lock file: migration_batch_2026Q1.lock — last modified 23:47:14. It took 90 minutes to trace it back to the agent’s restart, another 2 hours to restore from the pre-migration snapshot and re-run.
The fix wasn’t “remove write access.” It was: add a pre-flight check that queries for active long-running jobs before any service restart, and add that check to the agent’s tool policy:
# pre-flight gate added after the cert incident
- tool: check_active_jobs
threshold: 30m
on_match: notify_and_halt
message: "Long-running job detected — skipping restart, notifying operator"The agent now knows to stop and notify if anything has been running more than 30 minutes.
The rollback was clean because the snapshot was recent. If it hadn’t been, this would have been a much worse morning.
What this cost us: half a day of engineering time, one embarrassing conversation, one updated tool policy. What it bought us: a tool policy that’s now 20% better than it was before.
The Win: Disk Alert at 3:17am
A few months after the above, at 3:17am, Uptime Kuma fired an alert: disk usage on a production host had crossed 85%. The threshold job queued to the agent on-call.
The agent ran df -h, confirmed the culprit was /var/log. Checked log rotation config — found that a recently deployed service had logging set to debug level and hadn’t been rotated in 18 days. Ran journalctl --vacuum-time=7d on that service’s journal, dropped usage to 41%. Restarted log rotation. Verified disk healthy.
Total elapsed: 4 minutes 22 seconds from alert to resolution.
The agent’s log was clean enough that I could read it the next morning like a runbook: what it checked, what it found, what it did, what it verified afterward. If it had gone wrong — if the wrong logs had been cleaned, or the service had restarted badly — I had a complete record of exactly what happened and why.
I slept through the whole thing. That felt like something.
What made this work: tight tool scope (read access + specific vacuum/rotation commands only), Uptime Kuma feeding the alert directly into the agent’s task queue, journald providing clean introspection, and a logging schema that made the agent’s reasoning legible after the fact.
The Tooling Layer
We evaluated a lot of monitoring tools. For agent-managed infrastructure specifically, two filled gaps nothing else did cleanly:
• Uptime Kuma for endpoint and service health — simple, self-hosted, webhooks that actually work, no SaaS dependency. This is what feeds alerts into agent task queues.
• Beszel for host-level metrics — lightweight agent, clean API, works well with tailnet-connected hosts without public exposure. We’ll write up the full monitoring stack separately; it deserves its own post.
For the agent service itself, a stripped-down systemd unit:
[Unit]
Description=AI Agent Service
After=network-online.target tailscaled.service
Requires=tailscaled.service
[Service]
Type=simple
User=agent
Group=agent
WorkingDirectory=/opt/agent
ExecStart=/opt/agent/bin/agent-runner --config /etc/agent/config.yaml
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
# Contain the blast radius
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/agent /var/log/agent
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
[Install]
WantedBy=multi-user.targetThe key line is User=agent — not root, never root. Tool access is scoped at the application layer, but don’t make the OS layer an afterthought.
Every tool call gets logged. Here’s the schema we settled on after iterating:
{
"session_id": "sess_2a9f3b",
"timestamp": "2026-02-14T03:17:22Z",
"tool": "exec_command",
"inputs": {
"command": "journalctl --vacuum-time=7d --unit=app-worker"
},
"outputs": {
"exit_code": 0,
"stdout": "Vacuuming done, freed 4.2G of archived journals...",
"stderr": ""
},
"context": {
"triggered_by": "uptime_kuma_alert",
"alert_id": "alert_88c2d1",
"host": "prod-worker-01.tailnet"
},
"duration_ms": 847
}session_id ties all the calls for one incident together. triggered_by tells you why the agent was running at 3am. context.host uses the Tailscale hostname, never an IP. This schema has been stable for months; changes to it are treated like schema migrations.
The tool policy design — what gets allowed, what triggers an escalation, how permissions are scoped per-task — probably deserves its own post. The short version: start more restrictive than you think you need and relax constraints based on observed behavior, not on optimism.
Trust Accrues Slowly, Then Compounds
The path from “I’d never let an AI touch my servers” to “I woke up to a resolved incident I didn’t know about” doesn’t happen in a meeting. It happens through a series of small experiments that either add confidence or reveal gaps to close.
Read access first. Then drafted changes you review and apply. Then applied changes in non-production. Then a narrow set of production actions with logging on both ends. Each step adds to the picture.
Different agents earn different trust levels based on what they’ve demonstrated. The agent that’s handled log triage for six months has a larger operational footprint than the one deployed last Tuesday. That’s intentional.
Agents also introduce a new class of forensic responsibility. When a human makes a bad call, you can ask them what they were thinking. When an agent makes a call you don’t understand, you’re reading tool call logs, reconstructing reasoning from externally observable behavior. This is manageable when you’ve invested in logging. It’s a nightmare when you haven’t.
Until your pre-flight checks and tool policies are battle-tested, a notification at 3am is better than an autonomous decision made with incomplete context. The cert incident taught us that. The disk alert proved the other side — that once the guardrails are solid, the agent earns the right to act without waking you up.
Where This Is Actually Going
The predictions were right that 2026 is the year agentic AI gets real traction. They got the story wrong. It’s not the year agents become autonomous. It’s the year that operators who did the operational work — logging, tool policies, incremental trust-building — start seeing compounding returns on that discipline.
A $200 mini-PC, Tailscale, a model API key, and Uptime Kuma is enough to run a meaningful autonomous system handling real operational work. The barrier to entry has collapsed. The barrier to doing it well hasn’t — it just moved from “can you access the infrastructure” to “have you built the scaffolding around the agent.”
The shops that treat agent deployment like any other production system are going to look very different from the shops that spun up an agent, got burned, and quietly disabled it.
If You’re Getting Started
Promethean Dynamic helps teams deploy agents on infrastructure they own and care about. Specifically:
• Design and implement agent-safe tool policies — scoped access, pre-flight checks, escalation paths
• Integrate agents with your tailnet and existing observability — Uptime Kuma, Beszel, journald, whatever you’re running
• Run pilots with defined blast radius and on-call guarantees — so you learn fast without learning expensively
If you’ve already run an agent and it made a mess, we’ve probably seen the same mess. Get in touch.
