A protection I never saw refuse anything does not exist

I built a site meant for three people, for one week. All the content had to stay behind a password, images and video included. Nothing was to remain after the deadline.

So the requirements were the reverse of a normal site. I was not looking for an audience, I was looking for the absence of a leak. And an absence is not something you notice by watching the site work.

The principle I ended up adopting

A check that always passes checks nothing.

When I write a control, I start by giving it a reason to fail. I deliberately introduce the flaw it is supposed to detect, I check that it detects it, then I remove the flaw. If the control turns green in both cases, it is useless, and worse than useless: it reassures me.

I applied this principle in two places. Each time, I found a real flaw.

First control: nothing from the content may reach the browser

My texts live in typed files, imported only by components rendered on the server. In theory, no sentence can end up in the JavaScript sent to the browser. In theory.

I wrote a check in two parts. The first walks the import graph and fails if a component marked as client-side reaches a content file. The second looks for the text itself in the compiled files.

To test it, I injected a leak into a page, then ran the check. It failed, as expected. I then injected a partial leak: not a whole text, but a sentence taken from the middle of a paragraph.

The check turned green.

It was looking for complete strings. An isolated sentence, a quote, a truncation: all of that slipped past it. I rewrote it to also look for whole sentences and sliding windows of six consecutive words, plus the encoded variants the build produces. I replayed both injections: both are detected.

Without that second attempt, I would have had a green check and a possible leak. That is exactly the scenario I dread, because it does not look like a failure.

Second control: refuse what comes from elsewhere

My login page refuses requests coming from another site. I wrote the control and tested it from the command line with a foreign origin: refused. With the right origin: accepted. Green control.

Then I opened a browser. I could not log in.

My site sends a header asking the browser not to send any referrer. It is good practice; I had added it to avoid revealing the address of my pages. But with that instruction, the browser announces an empty origin instead of the real one. So my control saw an unknown origin and refused everyone, while my command line tests dutifully sent the right origin.

My check did not test what a browser does. It tested what my test tool does.

I switched to another header, one that browsers send and that says whether the request comes from the same site, and kept the origin as a fallback. Then I redid both tests in a real browser: the form hosted elsewhere is refused, mine goes through.

What these mistakes have in common

In each case, my protection existed, my test existed, and they never met.

The first test checked one form of leak instead of checking the leak. The second checked the behaviour of a client that is not the one my users have. Neither would have been found by rereading the code, because the code was correct with respect to what I thought I was testing.

I took three habits from this, and I keep them.

I write the negative control before the positive one. I build the flaw first, I watch the test fail, and only then do I fix it. A test I have never seen fail has no demonstrated value.

I test with the real client. A command line does not send the same headers as a browser, does not apply the same cookie rules, and is not subject to the instructions I send myself. For anything that touches authentication, I end up in a browser.

I prefer a noisy control to an elegant one. My leak check looks for windows of six words, which produces a few false alarms when a common phrase turns up in a library. I would rather dismiss three alerts by hand than miss the only one that matters.

The limit I know of

These checks prove the absence of what I know how to look for. They do not prove the absence of a leak. Data leaving by a path I have not thought of would pass green without any trouble.

That is why I never present a green result as a guarantee. I present it for what it is: the list of what I tried to break, and what held.