Logging Prompts: Safely, Not Everything

From the field: a national ID number on the log screen

I was the one who turned this log on. While we were fixing the latency problems of our support assistant, we wanted to see why the prompt was 6,000 tokens long. So we started writing the full prompt and the full answer of every request to the log platform. It was a one-line change. Retention was the platform default: 90 days. Access was the default too: the whole engineering team, two analysts and one outside consultant. 23 people in total.

That Thursday, after I saw the log line, the first thing I did was scan the last 14 days. There were 41,200 chats. The result:

The second row hurt the most. The account summary we added to the prompt contained the customer’s full IBAN. Why? Because I had said “the model might need it”. The model never needed the full IBAN, not even once. There was an ironic detail too. We had an output filter that masked account numbers in the answer on the screen. The customer’s screen was clean. The log received the text before the filter.

The maths was simple. After 90 days the log would hold about 265,000 chats, and 23 people could read any of them. Nobody had bad intentions. It was just that nobody had thought of this log as a customer database.

Why you need the log, and why it is dangerous

Turning the log off looks like the easy fix. It is not. Debugging an LLM feature has three questions, and you cannot answer any of them without the content: what did the model see, what did it say, and why? You cannot fix “the assistant calculated the commission wrong” without seeing the prompt. Maybe the model made a mistake. Maybe we gave it the old price table.

There is also a difference from a classic service log: the input of an LLM is free text. In a classic service you decide what goes into which field, so you find personal data by field name and mask it. In a chat, a customer can write their ID number, their mother’s name, their debt or their illness anywhere they like. There is no field name. There is only guessing.

KVKK (the Turkish personal data protection law) has general principles that come down to two sentences here. Data must be relevant, limited and proportionate to its purpose. And it must not be kept longer than the purpose needs. “Debugging” is a valid purpose. “Keeping every message from every customer open to 23 people for 90 days, for debugging” is not proportionate.

First, dry up the source

The first fix was not about masking. It was about the prompt. We asked one question about every field in the account summary: does the model really use this to write the answer?

We saw no change in answer quality. The rate of negative feedback stayed where it was that week. The 8,900 chats with an IBAN from our side went to zero in one step. This is not masking. It is better: data you never write cannot leak.

Masking: before it reaches the log

You cannot dry up what the customer types. That you have to mask. The real decision is where to mask. Our answer: inside the application, in the logging library itself. Developers do not have to remember to mask. Every piece of text that goes to the log passes through the same function.

TCKN = re.compile(r"\b[1-9](?:\s?\d){10}\b")          # also catch IDs written with spaces
IBAN = re.compile(r"\bTR\d{2}(?:\s?\d{4}){5}\s?\d{2}\b", re.IGNORECASE)
CARD = re.compile(r"\b(?:\d[ -]?){15,16}\b")

def tckn_valid(s):
    d = [int(c) for c in s if c.isdigit()]
    tenth = ((d[0]+d[2]+d[4]+d[6]+d[8]) * 7 - (d[1]+d[3]+d[5]+d[7])) % 10
    return d[9] == tenth and d[10] == sum(d[:10]) % 10   # ~1% of random 11 digits pass

def token(kind, value):
    # NOT a plain hash: a keyed digest. people with log access do not have the key.
    digest = hmac.new(MASK_KEY, normalize(value), "sha256").hexdigest()[:8]
    return f"<{kind}:{digest}>"       # same customer, same token: still traceable

def mask(text):
    text = TCKN.sub(lambda m: token("TCKN", m[0]) if tckn_valid(m[0]) else m[0], text)
    text = IBAN.sub(lambda m: token("IBAN", m[0]), text)
    text = CARD.sub(lambda m: token("CARD", m[0]) if luhn(m[0]) else m[0], text)
    return text

log.content(chat_id, prompt=mask(prompt), answer=mask(answer))

There are three decisions in this code, and each one comes from a mistake.

The check digit test. If you mask every 11-digit number, you also mask order numbers, and then you cannot debug. The Turkish national ID number (TCKN) has two check digits, and about 1% of random 11-digit numbers pass the check. So 1% of our 11-digit order numbers are masked for no reason. We solved it like this: record numbers, such as order and customer numbers, are written in clear text in a separate structured field, not taken from the free text. If the copy in the free text gets masked, we lose nothing.

A keyed digest. In the first version we hashed ID numbers with plain SHA-256 and called it masking. A colleague from the security team reversed it during a lunch break. The first nine digits decide the rest, so there are 900 million candidates, and an ordinary laptop tries all of them in minutes. In a small space, a plain hash is only a slow lookup table. With a keyed digest (HMAC) nobody can try candidates without the key, and the key lives where no one with log access can reach it.

A token, not a deletion. <TCKN:7f3a91c2> does not say who the customer is, but it can link three different chats from the same customer. That is enough for the question “does this bug always happen to the same customer?”.

The code also misses things, and it is important to say so openly: names, addresses, free sentences like “my mother’s illness”. We did not try to catch these with regex. The two layers and the access rules below are there for them.

Not everything: two layers and sampling

We split the log in two. The first layer is written for every request and contains not a single word from the customer. The second carries content, but it is masked, sampled and short-lived.

The logic of sampling is simple: the chats you need to debug are not spread at random. Complaints come from negative feedback, from chats handed over to an agent, and from errors, so we keep all of those. The random 5% is there to check whether “normal-looking” chats hide a problem. In total, about 11% of chats reach the content log. With 30 days of retention, the log holds about 9,700 chats at any time, not 265,000.

We keep metadata for 13 months because of trends. “How much did the prompt grow in the last six months?” needs numbers, not content. There is no customer text in it, so keeping it longer costs nothing.

Who sees what

The number of people with access to content went from 23 to 6. The outside consultant lost access completely. Every content view is also recorded: who looked at which chat, and when. This record does two jobs. First, it discourages casual looking. Second, when someone asks “who can access customer data?”, we can answer with a list, not a sentence — the same evidence idea I described in the security audit post.

When you really need the raw text

Sometimes masking hides the bug itself. In the first week of the new setup, a bug came in: the assistant did not recognise IBANs written with spaces. In the masked log every IBAN looked like <IBAN:…>, so the problem was invisible.

The key idea is this: the raw text already exists somewhere. It is in the chat database, under customer data rules, with its own retention and permissions. The log does not need to keep a second copy. When someone needs the raw text, a person from the assistant team opens a request with the chat number, security approves it, that one chat opens for 24 hours, and the access is recorded. In the first two weeks of the new setup, 12 of 14 bug reports were solved with the masked log. For the other 2 we used this path.

What did not work for me

• Masking in the log platform. This was the first reflex: a masking rule on the platform’s ingest pipeline. But before the raw text reached that pipeline, it passed through a message queue with its own 7-day retention. Backups are another story. If masking does not happen at the first place the data leaves, every stop on the way is a copy.
• Asking another model to find personal data. “Let a model read the text and mark the personal data” — we tried it. It added 400 ms to every request and doubled the cost. And the irony: to mask customer text, we were sending it to one more model. Regex and check digits are boring, but they are fast, cheap and measurable.
• “Keep everything for now, we will clean it later.” We planned to clean the old log twice, and both times it was postponed. In the end we deleted the whole log and started again with the new rules. A log you plan to clean later never gets cleaned.

What to track

Masking is written as a rule, but it works as a test. We built a job that scans the masked log every night with looser versions of the same patterns. In the first week it found 14 leaks: ID numbers written with spaces, IBANs starting with a lowercase “tr”. We fixed all of them, and the code above is the result of those 14 leaks.

Checklist

Conclusion

I still remember the log line I read that Thursday, because I was the one who put that information there. Yes, the customer typed their own ID number. But I added the IBAN to the prompt, I turned the log on, and I left the 90 days and the 23 people as defaults.

The fix was not to turn the log off. Debugging still needs content. The fix was to treat the log like a customer database: do not put in data you do not need, mask what you do put in before you write it, keep little of it for a short time, and know who is looking.

The test: if you showed your log screen to one of your customers today, what would they read about themselves? If the answer is “more than debugging needs”, your log is not working for you. It is working for the next leak.