Security Audits: Collecting Evidence, Not Paperwork

From the field: 214 requests and the last week

We have one audit season a year: the regulator’s information systems audit, and the ISO 27001 surveillance audit in the same period. Until last year we prepared for both the same way. Three weeks before the audit, the compliance team sends a spreadsheet. It has 214 rows, and every row is a request for evidence. Four engineers spend three weeks filling it in. The total cost is roughly 260 hours.

Most requests were easy: “Show the password policy configuration.” One screenshot, done. The hard ones asked about the past: “Show the approval records for these 25 production changes from the year.” For 7 of the 25, the approval was a chat message that said “ok, ship it”. Two had no approval at all. The changes were correct and had been reviewed, but there was no record that showed it.

The audit ended with 6 findings. Four were versions of the same sentence: the control worked, but there was no evidence. The other two were real, and they bothered me more. A contractor who had left still had an active account 40 days later. And we had never tested a restore from backup. If the auditor had not asked, we would not have done it for another year.

The mistake was mine. I thought the audit was the compliance team’s job, and we were just the people who sent screenshots when asked. But everything they asked about was something engineering produced: approvals, access, logs, backups. We designed the systems that could produce the evidence. We just had not designed them to do it.

What does the auditor ask?

ISO 27001, SOC 2 or a regulator’s audit: the names and control lists are different, but the logic is the same. The auditor looks at three things:

1. Design. Is the control written down? “Every production change is approved by someone other than its author.”
2. Implementation. Does the control really exist? Is the branch rule turned on?
3. Operating effectiveness. Did the control work all year? This is where sampling comes in: 25 random changes, 25 approvals.

A screenshot is enough for the first two. It is not enough for the third, because the question is about the past. You cannot prove May’s access list with a screenshot in January. Either someone saved it on that day, or nobody did. The whole point is not to leave that record to someone’s memory.

Evidence as a side effect

We sat down and looked at where the evidence for each control came from. The surprise was that almost all of it was already being produced. PR approvals existed in the system. Access groups existed in the identity provider (IdP). Deploy logs existed. The problem was not producing it. It was keeping it: PRs were squashed and the approval was lost, the access group only showed its current state, and deploy logs were deleted after 30 days.

Every row in this table is an engineering decision. The compliance team can write the rule “access is removed within 24 hours”. But we choose the query that connects the HR system to the identity provider, the threshold that turns the gap into an alert, and where the record is kept.

The evidence collector: every night, one job

We tied all of it to a single nightly job. What it does is simple: for each control, query the source, write the result to a dated file, and put the file in write-once (object lock) storage. When the auditor asks for something, the answer is a folder path.

# every night at 02:00. output: evidence/2025/05/14/ACCESS-01.json
# storage: write-once (object lock), kept as long as the policy says
controls:
  - id: CHANGE-01            # no unapproved production change
    source: deploy log -> commit sha -> PR -> approver
    rule:   approver != author
    breach: alert + open a ticket (24 hours if it was an emergency fix)

  - id: ACCESS-01            # who has production access
    source: idp group "prod-access" member list
    rule:   keep the list every day, write the diff separately

  - id: LEAVER-01            # leavers are disabled within 24 hours
    source: hr leaving date - idp disable date
    rule:   gap > 24 hours --> alert (on THAT day, not at the audit)

  - id: LOG-01               # log retention
    source: date of the oldest readable record in every log source
    rule:   today - oldest < policy period --> alert

The real value of the collector did not show on audit day. It showed during the rest of the year. In April, on its very first night, it found three accounts that should have been closed. The oldest had been open for 19 days. Last year the auditor found the same kind of problem, on day 40. This year we found it ourselves, on the first night.

Three engineering decisions

1. An emergency path

On the day we made “the approver must not be the author” a hard branch rule, the first question came: what if someone has to ship a fix alone at 03:00? We did not loosen the rule. We wrote a second path. An emergency fix can ship without prior approval, but the deploy tool marks it, opens a ticket automatically and asks for approval within 24 hours. In 2025 there were 9 emergency releases, and all 9 were approved within 24 hours. When the auditor asked about this path, we showed the tickets. No finding.

2. Break-glass instead of permanent access

Nine people had permanent access to the production database. Every quarterly review, we asked “is this still needed?”, and every time the answer was “yes, rarely, but yes”. So we changed the question: permanent access is zero. When someone needs it, they open break-glass access with a ticket number. It closes by itself after 4 hours, and the session is recorded. In 2025, 31 sessions were opened. The reason for each one is in its ticket. The auditor’s question changed from “who had access?” to “why did they use it?”, and the answer was already written down.

3. Write only the policy you can follow

Our old policy said: “Every production change is approved by two people.” Nobody did that. An auditor checks against the written policy, not against your good intentions. Two approvals on paper and one in practice: a finding in every sample. We rewrote the policy as “one approval + all automated tests pass”, because we could produce evidence for both, 100% of the time. It looks like a weaker rule. In fact it is stronger, because now it is actually followed.

How it goes wrong

What changed this year?

We finished the second audit in the middle of this month. The number of requests went down to 186, because the auditor closed some controls by looking at the evidence folder directly. 141 of the 186 requests were answered with a folder path. The other 45 were still collected by hand; most were policy documents and meeting notes, things that are human work by nature.

The one finding taught us something. VPN logs were kept for 14 months, but the policy said 24. That log source was not in the collector, because we had not thought of it when we made the list. The lesson is simple: a control that is not in the collector is a control without evidence. Now, adding a line to the collector is one of the steps of setting up any new log source.

What did not work for me

• Buying a ready-made compliance tool. That was my first reflex. The tool listed the controls nicely, but to pull evidence from our systems we still had to build every connection ourselves. The real work was the connections, not the tool.
• Doing the quarterly review by email. We sent lists to managers and asked “do you approve?”. Half did not reply. Most of those who did said “ok” without reading the rows. We moved it to the ticket system and put a separate “keep / remove” choice on every row. In the first round, 11 access rights were removed.
• Keeping deploy logs like application logs. A 30-day retention period was right for application logs, not for change evidence. We moved deploy records to a separate stream with its own retention period.

What to track

Checklist

Conclusion

On that Monday night last year, four engineers were taking screenshots at 23:40, and half of what they captured did not answer the question. The problem was not that they were not working. They were working at the wrong time. The evidence should have been collected on the day things happened.

This year, on the night before the audit, everyone was at home. The audit was not easier; the auditor asked more questions. But the answer to every question had been piling up all year, on its own, without anyone thinking about it. I described the same idea in the error budget post: write the rule on a calm day, and do not negotiate it during the fire.

The test: if the auditor came tomorrow morning, would anyone stay late tonight? If yes, you do not have evidence. You have a last-minute rush.