I tested whether AI explanations hold up

I saw circuit tracing from Anthropic, which maps how a language model gets to its answer [1]. But I was wondering how much an explanation like that can really be trusted, so I decided to test it myself.

I started simple and did not test circuit tracing itself. I took a small open-source model, Qwen 3 0.6B, asked it 394 questions and compared a gradient explanation with what really happens when I switch off parts of its network. Everything here is a real result, computed in advance on my MacBook. What you click stays in your browser.

How I tested it

  1. Question28 layersAnswer

    1. I ask a question

    The model answers in one short sentence and I score it against a list of accepted answers. It always picks its most likely next word, so the same prompt gives the same answer.

  2. 2. I switch off one spot

    I replace one position at one layer with that layer's average and let the model answer again, for every spot. This is called activation patching [2]. Meng et al. located facts in a model in a similar way [3].

  3. BeforeAfter

    3. I compare the answers

    If the answer changes or its probability halves, that spot mattered. The circuit tracing authors validate their graphs the same way, by changing the real model and watching what happens [1].

Loading results.