Home → Engineering
Engineering: Systems, Architecture and AI
Concurrency, distributed systems, messaging, scaling, observability and LLM/agent systems. Not how it looks on the diagram, but how it breaks in production and how it gets fixed.
The posts include code examples, “What to watch” lists and checklists.
Build or Buy: Ownership, Not Cost
We said two sprints; it took seven weeks. The money ends in a draw. The decision comes from core work, the owner’s name and the exit cost.
ENGINEERINGRewriting: A Migration Plan, Not a Clean Slate
A five-month rewrite lasted four hours in production. Shadow runs, sorting differences, moving segment by segment, and when to switch the old system off.
ENGINEERINGSLOs and Error Budgets: A Negotiation Tool, Not a Target
Features or reliability? One number ended a 50-minute argument: 99.9% is 43 minutes a month, and when the budget runs out, feature releases stop for that service.
ENGINEERINGSecurity Audits: Collecting Evidence, Not Paperwork
At 23:40 the night before the audit, four engineers were still taking screenshots. Once evidence became a side effect of daily work, most answers turned into a folder path.
ENGINEERINGLLM Latency: The First Token, Not the Average
3.4 seconds on the dashboard, an 11-second empty bubble on the customer’s screen. TTFT, tokens per second, streaming and a latency budget for the first word.
ENGINEERINGLogging Prompts: Safely, Not Everything
We logged every prompt for debugging and found 2,310 national ID numbers inside. We had put 91% of the IBANs there ourselves. Reduce at the source, mask before logging, 30 days, 6 people.
ENGINEERINGRace Conditions: Why Does Stock Go Negative?
Invisible in staging, catastrophic on sale day. Six fixes, with SQL examples.
ENGINEERINGRate Limits and Fallback: A Plan, Not an Error
On an IPO morning our support assistant told one customer in three “I can’t help” for 38 minutes. RPM/TPM quotas, retries with jitter, the quality gap of a fallback, and four levels of graceful degradation.
ENGINEERINGDistributed Locks: Getting It Right
Six traps in a Redis lock, fencing tokens, and the fact that most cases need no lock at all.
ENGINEERINGLeader Election: Who Does This Job?
Four copies, all running the health check. Leases, split-brain and fencing tokens.
ENGINEERINGPrompt Cache: A Design Decision, Not a Discount
Same traffic, 2.5 times the LLM cost: the customer name had moved to the first line of the system prompt. The fixed prefix, hidden variables, hit rate and time to first token.
ENGINEERINGWorker Sharding: The Right Way to Split Work
Four workers, one at 100% CPU and three idle. Hot keys, rebalancing and ordering guarantees.
ENGINEERINGHow Pod Scaling Actually Works
Pods went from 4 to 24 and the system got slower. Why is CPU the wrong metric?
ENGINEERINGToken Dashboard: An Early Warning, Not an Invoice
The March invoice was $11,400 against a $4,000 budget, and the increase had started 25 days earlier. Four tags on every request, a daily budget and spike alert, the input/output ratio and the most expensive 1%.
ENGINEERINGHow Many Times Does a Message Arrive?
At-least-once, at-most-once and the exactly-once myth. Idempotency keys and dedup, with code.
ENGINEERINGQueue Pathologies
Poison messages, DLQs, jitter, head-of-line blocking, reordering and backpressure — all five.
ENGINEERINGHallucination in Production: Field Signals, Not Evals
The eval set passed 48 of 50 questions, and the assistant told a customer the wrong commission. Five field signals and their precision, a number check, a weekly review and feeding errors back into the eval set.
ENGINEERINGFailure Modes
Thundering herds, cache stampedes, cascades, circuit breakers, gray failure and failover loss.
ENGINEERINGDistributed Transactions: Why 2PC Blocks and Saga Saves You
One leg filled, the other rejected. Compensating transactions and the escrow pattern.
ENGINEERINGThe Model Changed Quietly: Drift, Not a Version
No deploy for nine days, same prompt, and the refusal rate went from 1.4% to 7.9%. Pinned snapshots, watching the output distribution, and 40 canary questions every morning.
ENGINEERINGThe Dual Write Problem: Outbox, Inbox and CDC
The order was written but the event never published. Why atomic dual writes are impossible.
ENGINEERINGCQRS and Materialized Views
The portfolio screen takes 8 seconds, the order path needs 3 ms. Separating reads from writes.
ENGINEERINGCrewAI: Borrowed Assumptions, Not a Ready-Made Team
A three-agent prototype worked in one day, then missed a complaint with regulatory risk: the compliance agent never saw that sentence. The same job in plain code: 71k tokens instead of 186k.
ENGINEERINGWhere Did the 200 ms Go?
Four services add up to 40 ms while the customer waits 240 ms. Correlation IDs and traces.
ENGINEERINGVibe Coding Rituals
Why does three hours of work turn into two days of debugging? Ten habits that work.
ENGINEERINGMonolith to Microservices: What Five Months Taught Us
The distributed monolith, double shipments, a cancelled order at the door, and the two services we merged back — with numbers.
ENGINEERINGLogs and Metrics in Microservices: One Screen, Not Six Log Files
Prometheus, Loki and Grafana on one screen. The four golden signals, structured logs, a 2.4-million-series cardinality accident, and why the CPU alert was useless.
ENGINEERINGDon’t Say “I Have an SMS Code, I’m Safe”: Step-Up Authentication with Passkeys for Withdrawals
The JWT was valid and $4,800 went to a new IBAN. A challenge bound to amount and IBAN, a free margin re-check, and the back door on the “add passkey” screen.
ENGINEERINGSOLID: A Tool, Not a Goal — When to Use It, When to Skip It
Two interfaces, two classes and a factory for a one-line welcome message. The five principles with code; the signals where they help, the cases where they only add files; YAGNI and the rule of three.
ENGINEERINGAgent, Local LLM, MCP Server: Who Decides, Who Executes?
An agent on an internal server that answers “how many dollars were deposited today?” with no data leaving the company. Three parts, three responsibilities; how the docstring decides tool selection, a 20/20 benchmark and the two brakes of the agent loop.
ENGINEERINGMonorepo: Team Structure, Not Code Structure
One line in the shared library, 7 bump PRs, 3 days, one service forgotten. Nine repos were right for 14 people; with 3 people they protect nobody. Why we moved from poly repo to monorepo, its five traps, the numbers 6 months later, and the real question: how many services for 3 people?
ENGINEERINGAgent Orchestrator: A Guided Flow, Not One Clever Agent
The nightly reconciliation job took 17 turns and 9 minutes in one agent, and the answer was wrong. In a 4-step flow: 2 turns, 38 seconds. Who decides, three patterns and four brakes.
ENGINEERINGMulti-Agent: Work You Can Split, Not Free Speed
We split 34 records across 4 agents: 11 minutes became 3, tokens went up 3.4 times, and two agents wrote to the same file. Split keys, the sub-agent contract and the merge step.
ENGINEERINGContext: The Right Summary, Not a Longer Window
The same question is wrong in a 118k-token window and right in a 9k one. A model does not remember, it re-reads: trim tool output, move run state to disk, and know what a summary costs.
ENGINEERINGTool Design: A Decision Surface, Not an API Wrapper
We went from 11 tools to 23 and accuracy fell from 96% to 71%. The code was fine; the menu was not. Separate decision or parameter, the four-line description, and error messages that instruct.
ENGINEERINGThe Agent Called It Twice: Idempotency, Not Autonomy
The tool answered in 32 seconds and the timeout was 30: the same withdrawal was approved twice. Who generates the key, what the second call returns, and why irreversible work is two-stage.
ENGINEERINGPrompt Injection: Input That Runs as Instruction
One sentence in a customer note reached the agent’s plan and put 3 withdrawals up for approval. Models do not separate data from instruction; the defence is in permissions, not the prompt.
ENGINEERINGThe Agent’s Trail: A Chain of Decisions, Not Logs
It took 40 minutes to find the drift in a 400-line log; now it takes 12 seconds. A run is a trace and a step is a span: what to record, how to mask customer data, how to replay a run.
ENGINEERINGEval: A Regression Test, Not a Demo
I changed one word in the prompt, the demo was perfect, and two days later 4 questions went to the wrong tool. Golden sets, exact checks, judge biases and “green but unhappy”.
ENGINEERINGPrompt and Model Versions: Code, Not Text
A “small fix” from an admin screen was a deploy with no record. Repo or config, why the model version is pinned, shadow runs and rollback thresholds.
ENGINEERINGRAG: Building Context, Not Searching
The agent stated the rule correctly; the rule had changed 14 days earlier. RAG or tool, who draws chunk boundaries, how to measure recall, and teaching a model to say “I do not know”.
ENGINEERINGAgent Cost: Cost per Task, Not per Token
“It is our own server, tokens are free” — one report kept the GPU busy for 96 seconds. Re-sent context, retries, the fixed prefix and the arithmetic of a cheap model with 3 attempts.
ENGINEERINGOn-Call: Handover, Not Heroics
03:12 at night, the alert fired and the runbook page was empty. The three questions on-call must answer, the rule that deleted 23 of 60 alerts, night authority, a 15-minute handover and the shadow week.
ENGINEERINGShipping: Release Management, Not Deploys
The rollback took 9 minutes; with a flag it would have taken 4 seconds. Separating deploy from release, four flag types and their lifetimes, a 61-flag graveyard, and automatic rollback thresholds.
ENGINEERINGSchema Changes: Two Stages, Not a Friday Night
80,000 rows in test, 47 million in production; a “five minute” migration locked the table for 40. The five steps of expand/contract, batched backfill, a table of unsafe operations and a drop-by date.
ENGINEERINGCode Review: Spreading Knowledge, Not Guarding a Gate
The PR waited three days and got 41 comments: 38 about formatting, 0 about architecture. The inverse relationship between size and value (PRs over 800 lines get FEWER comments), comment labels, the 24-hour rule and where a second approval is really needed.
ENGINEERINGArchitecture Decision Records: One Page, Not a Meeting
“Why did we do it this way?” — three people, three answers; we made the same decision twice, for different reasons. A one-page template, the rejected-options section, which decisions to record and why records are never deleted.
ENGINEERINGInheriting a System: The First 90 Days
The README was four lines and two were wrong; the answer to “how do we ship” was a name, and that person was leaving in three weeks. The single goal of week one, building the map from git, the 90-day rule and five questions for the person leaving.