|

DevOps Life with a Lobster: How a Dead DNS Server and Two Bad Headers Took Down 25 AI Agents

ByChris M.

In case this saves somebody else a lost day.

I run about 25 AI agents on a dedicated server with OpenClaw: Discord bots, Telegram integrations, embedded LLM agents hitting OpenAI's Codex API, the whole pile. About three days ago they started failing intermittently. Not crashing, not going fully offline, just randomly not doing what they were supposed to do. An API call here, a Telegram message there, a Discord response that never came back.

The error message every time was:

LLM request failed: DNS lookup for the provider endpoint failed.

I spent most of a day chasing it with my Claude Code setup, the "lobster" in the title. It turned out to be two separate problems hiding behind the same bad error message. Fixing the first one exposed the second. The second came down to two lines of JavaScript.

Part 1: The DNS server that wasn't there

The symptoms

  • Intermittent `DNS lookup for the provider endpoint failed` in agent logs
  • Telegram subsystem logging `DNS-resolved IP unreachable; trying alternative API IP`
  • Failures at random times, 2am, noon, 6pm, no obvious pattern
  • Everything seeming fine when tested manually (`dig google.com` came back clean every time)
  • A slow worsening over three days, from occasional hiccups to 10+ failures per day

Why manual testing lies to you

When you SSH in and run dig @127.0.0.53 api.openai.com, it works. So you tell yourself DNS is fine and go hunting for API key problems, provider outages, network rules, rate limits, whatever else looks plausible.

The problem is that systemd-resolved, Ubuntu's default DNS stub at 127.0.0.53, doesn't just use one upstream server. It rotates between all configured servers, and it does it quietly. Your one test query might hit the good server. Your agent's next query might hit the dead one.

The actual problem

My server's netplan config had three DNS servers:

nameservers:
 addresses:
 - 1.1.1.1 # Cloudflare
 - 8.8.8.8 # Google
 - 4.4.4.4 # Google "secondary"

Looks reasonable enough. Three big-name resolvers, plenty of redundancy.

Except 4.4.4.4 had 100 percent packet loss from my datacenter.

Not intermittent. Not slow. Just dead. Ten out of ten pings dropped. Meanwhile 8.8.8.8, same provider, worked perfectly at around 0.5ms. Nobody tells you that one resolver IP from a major provider can be completely unreachable from a given network while the sibling IP works great.

What systemd-resolved does with a dead server

This is where Ubuntu's stub resolver gets ugly. When systemd-resolved tries 4.4.4.4 and the UDP query times out, it doesn't cleanly mark it dead and move on. It gets stuck in a degraded feature loop:

  1. Tries UDP to `4.4.4.4`, times out
  2. Falls back to TCP, also fails
  3. Logs `Using degraded feature set TCP instead of UDP for DNS server 4.4.4.4`
  4. Waits a bit, then tries UDP again
  5. Logs `Using degraded feature set UDP instead of TCP for DNS server 4.4.4.4`
  6. Repeats forever

My journal had 78 of those warnings in three days. Every time a real DNS query landed on 4.4.4.4 in the rotation, it stalled and failed over. The lookup usually eventually succeeded, but with a multi-second delay. For Node.js apps using the system resolver, that surfaced as ENOTFOUND or EAI_AGAIN, errors that make it look like the destination is broken instead of your resolver path.

The cache was getting thrashed too

Cache Hits: 1666
Cache Misses: 2797
Cache Size: 8 entries

That is a 37 percent hit rate with only 8 entries cached. The constant failure cycling was probably evicting useful entries. Every cache miss meant another upstream trip, and every upstream trip was another chance to hit the dead server.

The DNS fix

I benchmarked 28 public resolvers directly from the server, not from my laptop, because DNS performance is network-path dependent. Six of the 28, including 4.4.4.4, were completely unreachable from this SoCal datacenter. Among the ones that did answer, latency ranged from about 1ms to 150ms.

First, I replaced the dead entries in netplan with resolvers that actually performed well from this box:

nameservers:
 addresses:
 - 1.1.1.1 # Cloudflare, 2ms avg
 - 8.8.8.8 # Google, 2ms avg
 - 9.9.9.9 # Quad9, 2ms avg, security filtering
 - 8.8.4.4 # Google alt, 1ms avg, fastest
 - 185.228.169.9 # CleanBrowsing, 1ms avg, lowest jitter

Then I added a systemd-resolved drop-in at /etc/systemd/resolved.conf.d/10-dns-tuning.conf:

[Resolve]
FallbackDNS=1.0.0.1 149.112.112.112 208.67.222.222 76.76.10.0 156.154.71.5
Cache=yes
CacheFromLocalhost=yes
StaleRetentionSec=30
DNSStubListener=yes

After applying both and restarting systemd-resolved, the degraded warnings stopped, timeouts went to zero, and the cache started growing normally.

DNS was fixed.

The agents were still failing.

Part 2: The ghost behind the ghost

Same error, different cause

Even after the DNS fix, the logs were still full of DNS lookup for the provider endpoint failed. But now the resolver stats were clean. No timeouts. No warnings. So I looked harder at the error payload itself.

The rawErrorPreview field contained HTML. Not a Node.js resolver error, not a socket failure, an actual rendered web page with CSS and layout markup.

If DNS really fails, you do not get a full HTML page back.

The Cloudflare wall

The OpenAI Codex subscription API routes through chatgpt.com/backend-api. A quick curl came back with this:

HTTP/2 403
cf-mitigated: challenge

Cloudflare was serving a JavaScript challenge to every request from the server. The HTML in the logs was that challenge page. OpenClaw was receiving it, not recognizing what it was, and classifying it as a DNS failure.

First attempt, wrong target

At first I thought the issue was stale baseUrl overrides in agent configs pointing at chatgpt.com/backend-api/v1, wrong suffix, or at the Codex harness endpoint instead of the subscription endpoint. I cleaned those out of 31 agent config files.

It did not help, because the real issue was the HTTP request getting blocked before any of that mattered.

Finding the actual fix

A search through the OpenClaw GitHub issues showed this was already a known regression in OpenClaw 2026.4.14. The root cause was in the provider library, @mariozechner/pi-ai, specifically how it builds request headers for the Codex endpoint:

// Line 715-717 of openai-codex-responses.js
headers.set("originator", "pi");
const userAgent = _os ? `pi (${_os.platform()} ${_os.release()}; ${_os.arch()})` : "pi (browser)";
headers.set("User-Agent", userAgent);

That sends originator: pi and a user agent that looks nothing like a browser. Cloudflare sees that, sees no cookies or clearance token, and throws a 403 challenge.

The two-line fix

headers.set("originator", "chatgpt-platform");
const userAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";

That was it. Two strings.

I patched node_modules/@mariozechner/pi-ai/dist/providers/openai-codex-responses.js, restarted the gateway, and every agent came back to life.

There are PRs pending, #66969 and #10044, that should fix this properly in a future release. Until then, this is a manual patch that will get overwritten on update, so back up the file if you depend on it.

The workaround from the issue thread

There is also a config-level workaround that partially helped: explicitly setting api: "openai-codex-responses" on every model entry in your provider config to force /responses instead of /chat/completions. That endpoint seems to hit looser Cloudflare rules. It may be enough for some setups, but the header patch was what fully fixed mine.

How to actually diagnose this

Do not run one dig command and declare DNS healthy. If you are seeing this class of failure, this was the sequence that actually worked.

1. Check what resolved is doing

resolvectl statistics

Look at Total Timeouts and Total Failure Responses. If they are non-zero, you have a DNS problem.

2. Check the journal

journalctl -u systemd-resolved --since '3 days ago' -p warning --no-pager | head -50

If you see Using degraded feature set warnings, one of your upstream resolvers is sick.

3. Test each resolver directly

for dns in 1.1.1.1 8.8.8.8 4.4.4.4 9.9.9.9; do
 echo -n "$dns: "
 ping -c 3 -W 2 -q $dns 2>&1 | tail -1
done

4. Read the raw error, not just the summary

DNS lookup for the provider endpoint failed can really mean "the app got a weird HTTP response and misclassified it." If rawErrorPreview contains HTML, DNS is probably not the thing failing anymore.

5. Benchmark from the server that is actually having the problem

Resolver performance is network-path dependent. From my datacenter, the spread between the best reachable resolvers and the worst was roughly 150x.

Desktop DNS benchmarking tools

If you want a GUI tool for your dev machine, which is useful but not a substitute for testing from the actual server:

  • **[GRC DNS Benchmark](https://www.grc.com/dns/benchmark.htm)** for Windows. Still the gold standard.
  • **[NAMEinator](https://github.com/mrwiora/NAMEinator)** for Mac, Linux, and Windows. Probably the best current cross-platform option.
  • **[namebench](https://github.com/catap/namebench)** if you enjoy old archaeology, but NAMEinator is the better pick now.

The broader lesson

What made this annoying was not just the bugs. It was that two unrelated failures collapsed into the same error string.

First, DNS was actually broken. After that was fixed, Cloudflare blocking was still getting reported as DNS failure. So the trail looked like one long DNS incident when it was really a layered failure: dead resolver, then bad header behavior, both hiding behind the same message.

If you run always-on services that make lots of outbound API calls, it is worth remembering three things:

  • test your DNS servers individually
  • read the raw payload, not just the error label
  • do not assume the software classified the failure correctly

Sometimes the system saying "DNS failed" is actually holding a Cloudflare challenge page and has no idea what it is looking at.

Running 25 AI agents on a single box with OpenClaw. The lobster helped.

Appendix: dns-bench.sh

Save this as dns-bench.sh, make it executable, and run it on the server that is having the issue. It only depends on bash, dig, and ping, which are standard on Ubuntu. It takes about two minutes. Edit the resolver list and real domains to match your environment.

Four phases:

  1. **ICMP reachability**, which resolvers are alive from your network
  2. **Query performance**, 3 passes by 5 domains, scored by latency plus failure penalty
  3. **Cold lookups**, how fast cache misses are
  4. **Consistency and jitter**, 10 rapid queries against the top candidates
#!/bin/bash
# DNS Resolver Benchmark - tests from this server's perspective
# Tests: latency, reliability, cached vs uncached performance

RESOLVERS=(
 "1.1.1.1:Cloudflare-Primary"
 "1.0.0.1:Cloudflare-Secondary"
 "8.8.8.8:Google-Primary"
 "8.8.4.4:Google-Secondary"
 "4.4.4.4:Google-Legacy"
 "9.9.9.9:Quad9"
 "149.112.112.112:Quad9-Secondary"
 "208.67.222.222:OpenDNS"
 "208.67.220.220:OpenDNS-Secondary"
 "94.140.14.14:AdGuard"
 "94.140.15.15:AdGuard-Secondary"
 "185.228.168.9:CleanBrowsing"
 "185.228.169.9:CleanBrowsing-Sec"
 "76.76.2.0:ControlD"
 "76.76.10.0:ControlD-Secondary"
 "64.6.64.6:Verisign"
 "64.6.65.6:Verisign-Secondary"
 "77.88.8.8:Yandex"
 "77.88.8.1:Yandex-Secondary"
 "156.154.70.5:Neustar-Threat"
 "156.154.71.5:Neustar-Threat-Sec"
 "45.90.28.0:NextDNS"
 "45.90.30.0:NextDNS-Secondary"
 "176.103.130.130:AdGuard-Old"
 "198.101.242.72:Alternate-DNS"
 "91.239.100.100:UncensoredDNS"
 "89.233.43.71:UncensoredDNS-Sec"
 "74.82.42.42:Hurricane-Electric"
)

REAL_DOMAINS=("api.openai.com" "discord.com" "gateway.discord.gg" "api.telegram.org" "github.com")
PASSES=3
TIMEOUT=3

echo "=============================================="
echo " DNS Resolver Benchmark - $(hostname)"
echo " $(date)"
echo " ${#RESOLVERS[@]} resolvers x $PASSES passes"
echo "=============================================="
echo ""

echo "--- PHASE 1: ICMP Reachability ---"
printf "%-22s %-20s %10s %8s\n" "IP" "Name" "Ping(ms)" "Status"
echo "--------------------------------------------------------------"
LIVE_RESOLVERS=()
for entry in "${RESOLVERS[@]}"; do
 ip="${entry%%:*}"
 name="${entry##*:}"
 result=$(ping -c 3 -W 2 -q "$ip" 2>&1)
 if echo "$result" | grep -q " 0 received"; then
 printf "%-22s %-20s %10s %8s\n" "$ip" "$name" "---" "DEAD"
 else
 avg=$(echo "$result" | tail -1 | awk -F'/' '{print $5}')
 printf "%-22s %-20s %10s %8s\n" "$ip" "$name" "${avg}ms" "OK"
 LIVE_RESOLVERS+=("$entry")
 fi
done

echo ""
echo "Live: ${#LIVE_RESOLVERS[@]} / ${#RESOLVERS[@]}"
echo ""

echo "--- PHASE 2: DNS Query Performance ($PASSES passes x ${#REAL_DOMAINS[@]} domains) ---"
printf "%-22s %-20s %7s %7s %7s %5s %7s\n" "IP" "Name" "Min" "Avg" "Max" "Fail" "Score"
echo "--------------------------------------------------------------------------------"

RESULTS=""
for entry in "${LIVE_RESOLVERS[@]}"; do
 ip="${entry%%:*}"
 name="${entry##*:}"

 total_time=0
 min_time=99999
 max_time=0
 fail_count=0
 query_count=0

 for pass in $(seq 1 $PASSES); do
 for domain in "${REAL_DOMAINS[@]}"; do
 result=$(dig +time=$TIMEOUT +tries=1 @"$ip" "$domain" A 2>&1)
 qtime=$(echo "$result" | grep "Query time:" | awk '{print $4}')
 status=$(echo "$result" | grep -oP 'status: \K[A-Z]+')

 if [ -z "$qtime" ] || [ "$status" = "SERVFAIL" ]; then
 fail_count=$((fail_count + 1))
 qtime=$((TIMEOUT * 1000))
 fi

 total_time=$((total_time + qtime))
 query_count=$((query_count + 1))

 if [ "$qtime" -lt "$min_time" ] 2>/dev/null; then min_time=$qtime; fi
 if [ "$qtime" -gt "$max_time" ] 2>/dev/null; then max_time=$qtime; fi
 done
 done

 if [ $query_count -gt 0 ]; then
 avg_time=$((total_time / query_count))
 else
 avg_time=99999
 fi

 score=$((avg_time + (fail_count * 500)))
 line=$(printf "%-22s %-20s %5dms %5dms %5dms %5d %7d" \
 "$ip" "$name" "$min_time" "$avg_time" "$max_time" "$fail_count" "$score")
 RESULTS="${RESULTS}${line}\n"
done

echo -e "$RESULTS" | sort -k7 -n

echo ""
echo "--- PHASE 3: Uncached/Cold Query Test ---"
printf "%-22s %-20s %10s\n" "IP" "Name" "Cold(ms)"
echo "----------------------------------------------"

UNIQUE="bench-$(date +%s)"
COLD_RESULTS=""
for entry in "${LIVE_RESOLVERS[@]}"; do
 ip="${entry%%:*}"
 name="${entry##*:}"

 qtime=$(dig +time=$TIMEOUT +tries=1 @"$ip" "${UNIQUE}.example.com" A 2>&1 \
 | grep "Query time:" | awk '{print $4}')
 if [ -z "$qtime" ]; then
 display="FAIL"
 else
 display="${qtime}ms"
 fi

 line=$(printf "%-22s %-20s %10s" "$ip" "$name" "$display")
 COLD_RESULTS="${COLD_RESULTS}${line}\n"
done

echo -e "$COLD_RESULTS"

echo ""
echo "--- PHASE 4: Consistency Test (10 rapid queries, same domain) ---"
echo "Top resolvers only (lowest Phase 2 score)..."
printf "%-22s %-20s %7s %7s %7s %10s\n" "IP" "Name" "Min" "Max" "Jitter" "AllGood"
echo "------------------------------------------------------------------------"

TOP_IPS=$(echo -e "$RESULTS" | sort -k7 -n | head -6 | awk '{print $1}')
for ip in $TOP_IPS; do
 for entry in "${LIVE_RESOLVERS[@]}"; do
 eip="${entry%%:*}"
 if [ "$eip" = "$ip" ]; then
 name="${entry##*:}"
 break
 fi
 done

 times=()
 all_good="YES"
 for i in $(seq 1 10); do
 qt=$(dig +time=2 +tries=1 @"$ip" "api.openai.com" A 2>&1 \
 | grep "Query time:" | awk '{print $4}')
 if [ -z "$qt" ]; then
 all_good="NO"
 qt=2000
 fi
 times+=($qt)
 done

 min=${times[0]}
 max=${times[0]}
 for t in "${times[@]}"; do
 [ "$t" -lt "$min" ] 2>/dev/null && min=$t
 [ "$t" -gt "$max" ] 2>/dev/null && max=$t
 done
 jitter=$((max - min))

 printf "%-22s %-20s %5dms %5dms %5dms %10s\n" \
 "$ip" "$name" "$min" "$max" "$jitter" "$all_good"
done

echo ""
echo "Benchmark complete: $(date)"

Similar Posts