# A protection I never saw refuse anything does not exist

Language: en

Identifiant : article:controle-negatif

Noa Matout

I built a private site, then tried to prove it was leaking: two of my protections did not hold.

I built a site meant for three people, for one week. All the content had to
stay behind a password, images and video included. Nothing was to remain after
the deadline.

So the requirements were the reverse of a normal site. I was not looking for an
audience, I was looking for the absence of a leak. And an absence is not
something you notice by watching the site work.

## The principle I ended up adopting

A check that always passes checks nothing.

When I write a control, I start by giving it a reason to fail. I deliberately
introduce the flaw it is supposed to detect, I check that it detects it, then I
remove the flaw. If the control turns green in both cases, it is useless, and
worse than useless: it reassures me.

I applied this principle in two places. Each time,
I found a real flaw.

## First control: nothing from the content may reach the browser

My texts live in typed files, imported only by components rendered on the
server. In theory, no sentence can end up in the JavaScript sent to the
browser. In theory.

I wrote a check in two parts. The first walks the import graph and fails if a
component marked as client-side reaches a content file. The second looks for
the text itself in the compiled files.

To test it, I injected a leak into a page, then ran the check. It failed, as
expected. I then injected a partial leak: not a whole text, but a sentence taken
from the middle of a paragraph.

The check turned green.

It was looking for complete strings. An isolated sentence, a quote, a
truncation: all of that slipped past it. I rewrote it to also look for whole
sentences and sliding windows of six consecutive
words, plus the encoded variants the build produces. I replayed both
injections: both are detected.

Without that second attempt, I would have had a green check and a possible
leak. That is exactly the scenario I dread, because it does not look like a
failure.

## Second control: refuse what comes from elsewhere

My login page refuses requests coming from another site. I wrote the control
and tested it from the command line with a foreign origin: refused. With the
right origin: accepted. Green control.

Then I opened a browser. I could not log in.

My site sends a header asking the browser not to send any referrer. It is good
practice; I had added it to avoid revealing the address of my pages. But with
that instruction, the browser announces an empty origin instead of the real
one. So my control saw an unknown origin and refused everyone, while my
command line tests dutifully sent the right origin.

My check did not test what a browser does. It tested what my test tool does.

I switched to another header, one that browsers send and that says whether the
request comes from the same site, and kept the origin as a fallback. Then I
redid both tests in a real browser: the form hosted elsewhere is refused, mine
goes through.

## What these mistakes have in common

In each case, my protection existed, my test existed, and they never met.

The first test checked one form of leak instead of checking the leak. The
second checked the behaviour of a client that is not the one my users have.
Neither would have been found by rereading the code, because the code was
correct with respect to what I thought I was testing.

I took three habits from this, and I keep them.

I write the negative control before the positive one. I build the flaw first, I
watch the test fail, and only then do I fix it. A test I have never seen fail
has no demonstrated value.

I test with the real client. A command line does not send the same headers as a
browser, does not apply the same cookie rules, and is not subject to the
instructions I send myself. For anything that touches authentication, I end up
in a browser.

I prefer a noisy control to an elegant one. My leak check looks for windows of
six words, which produces a few false alarms when
a common phrase turns up in a library. I would rather dismiss three alerts by
hand than miss the only one that matters.

## The limit I know of

These checks prove the absence of what I know how to look for. They do not
prove the absence of a leak. Data leaving by a path I have not thought of would
pass green without any trouble.

That is why I never present a green result as a guarantee. I present it for
what it is: the list of what I tried to break, and what held.

## Scope of the measurements

controls : 2 controls. Controls tried on my private site, each of which revealed a real flaw.

windowWords : 6 consecutive words. Size of the sliding windows used in my search for text leaks.