Finding What a Generative Model Made Up
A generative model's answer mixes what the data supports with what the model filled in from training. However, both parts look the same to the reader. Therefore, we propose a method that finds the invented parts using only the model, and we prove it in simulation, where the physics provides an exact answer key, so that it can be used on real sensors, where the physics is only approximately known.
- 1The probleminvented parts look real
- 2The answer keyphysics knows the truth
- 3An examplethe mug behind the corner
- 4The methoda map from the model alone
- 5The testfive hypotheses, two stages
- 6The usereal sensors, physics unsure
Invented parts look real
Powerful models now answer questions people cannot check, such as long proofs, large programs, or a scene no camera can see. However, a model's confident answer may rest partly on evidence and partly on what it guessed from training, and the two look the same. Therefore, overseeing such a model, a task known as scalable oversight, needs a way to point at the invented parts.
| Difficulty | Stage | How we test it |
|---|---|---|
| 1. No ground truth when the model is used | 1 | The simulator knows the truth; the method never sees it |
| 2. The model is stronger than its overseer | 2 | The overseer gets a smaller model and a few probes |
| 3. Nobody knows how the evidence links to the answer | 1 | The method is not given the physics |
| 4. The world changes after training | 1 | Simple simulation, then detailed simulation, then real data |
| 5. The model works against its overseer | 2 | A model trained to hide its invented parts |
Why we study it with physical sensors
On real hardware nobody knows the exact physics, and many reconstructors are black-box networks, so the invented parts of a real reconstruction cannot be computed and no method can be graded there. However, in a simulator the physics is exact and states which parts of a scene the measurements can reveal. Therefore, we use the simulator's physics as an answer key, and give the method nothing but the data and the model, exactly as it would have on real hardware.
The mug behind the corner
In Figure 1, light returns only from the front of the mug, the measured part. No light can return from the back, the null part. However, the model still draws a handle there, because most mugs it saw had one. Therefore, the goal is a method that marks the handle as invented without being told the physics.
A first idea, and why it fails. One could split the answer with the physics and check whether the model's confidence is honest on each part. This fails for three reasons:
- If training matches the world, the confidence on the null part is already honest, only wide.
- If training does not match the world, the null part is wrong, which is obvious from the mathematics.
- Fixing the confidence needs the truth at the time of use, which an overseer never has, and real hardware or a black-box network has no exact physics to split with.
Therefore, the question is not "is the confidence honest?" but "can the invented parts be found from the model alone?"
A map from the model alone, graded by the answer key
The answer key: how much of each part of the scene the physics cannot see.
\[ m^{\star}_j = \frac{\| P^{\tau}_{\text{null}}\, e_j \|^2}{\| e_j \|^2} \in [0, 1] \]The map from the model alone: a part that barely moves when the data is nudged does not rest on the data.
\[ \hat m_j = 1 - \frac{\| \nabla_y \hat x_j(y) \|}{\max_k \| \nabla_y \hat x_k(y) \|} \]The score: 1 is a perfect match, 0.5 is chance.
\[ S = \text{AUROC}\Big( \hat m,\; \mathbb{1}\big[\, m^{\star} > 0.5 \,\big] \Big) \]The hard part. The data cannot see the null part, yet the model's guess for it still moves when the data moves, because the model learned that the hidden part goes with the seen part:
\[ \underbrace{\frac{\partial y}{\partial x_{\text{null}}} = A\,P_{\text{null}} = 0}_{\text{the data cannot see it}} \qquad\text{but}\qquad \underbrace{\frac{\partial \hat x_{\text{null}}}{\partial y} \neq 0}_{\text{the guess still moves}} \]Therefore, the simple map above is only a first candidate. Better candidates compare the spread of the model's answers before and after seeing the data, or change only the part of the input the physics cannot see, and the score decides which candidate works.
Stage 1, then stage 2
Stage 1 runs the test of Figure 2 as the world changes, from simple simulation to detailed simulation to real data. However, a small model is neither stronger than its overseer nor trying to fool it. Therefore, stage 2 adds both conditions, as Figure 3 shows.
The hiding generator's training goal, where a larger weight means trying harder to fool:
\[ \min_{\theta}\; \mathcal{L}_{\text{recon}}(\theta) \;+\; \lambda\, S(\theta), \qquad \lambda \ge 0 \]Both are stand-ins, as in current oversight research: a larger model is not a truly more capable one, and a trained deceiver did not choose to deceive.
Five hypotheses
Stage 1. The map from the model alone finds the true null part.
Stage 1. As the world shifts by an amount δ, errors gather in the null part, and the map still marks it.
Stage 1. An overseer can check only B parts of each answer. Choosing them by the map catches more invented content than choosing them by the model's uncertainty, with no ground truth.
Stage 2. The method degrades gently, not suddenly, as the overseer falls behind.
Stage 2. A generator trained to hide cannot push the overseer down to chance.
From the simulator to real sensors
| Simulator (proof) | Real sensor (use) | Powerful AI system (long-term goal) | |
|---|---|---|---|
| Answer | Reconstructed scene | Reconstructed scene | The system's answer |
| Evidence | Simulated measurements | Real photon or radar measurements | The input the system was given |
| Physics linking evidence to answer | Exact: an answer key | Approximate: timing jitter, wall shape, reflectance, antenna motion errors | Unknown |
| Reconstructor | Diffusion model with the physics | Diffusion model or a black-box network | A black box |
| Probe the method uses | Nudge the measurements | Nudge the measurements | Change the input |
| Invented part | Handle on the unseen side | Structure the measurements never constrained | Part of the answer no input supports |
On real hardware the physics is only approximately known, and many reconstructors are black-box networks, so the invented parts of a real reconstruction cannot be computed from the physics. However, the method needs only the data and the model, so it applies there unchanged. Therefore, the simulator is where the method is proven against an exact answer key, real sensors are where it is needed, and any system whose answers cannot be checked is where the same idea points.
What exists, and what is missing
| Work | What it does | What it lacks here |
|---|---|---|
| Learned NLOS reconstructors (e.g. Li et al. 2023, NLOST) | Fast reconstruction of real captures with deep networks | No marking of what the network invented |
| Bhadra et al. 2021 | Maps invented content in CT reconstructions | Needs the known physics |
| Teneggi et al. 2023 | Statistical guarantees on diffusion confidence | Needs the known physics; no shift |
| Sandwiching (Cotra 2021; Bowman et al. 2022) | Tests oversight where the truth is hidden from the overseer | No exact answer key |
| Weak-to-strong (Burns et al. 2023) | A small model supervises a large one | Not about invented content |
| AI control (Greenblatt et al. 2023) | A model trained to sabotage, against a monitor | No exact answer key |
| Scaling laws for oversight (2025) | Oversight success falls as the capability gap grows | Games, not reconstructions |
No work finds invented content from the model alone and grades it against an exact answer key.
How an answer divides into two parts
The measurement model. The hidden scene is a list of unknown numbers. The sensor records a list of measured numbers, made by known physics from the scene, plus noise.
\[ y = A x + \varepsilon \] \[ \begin{aligned} x &\in \mathbb{R}^{n} && \text{the scene} \\ y &\in \mathbb{R}^{m} && \text{the measurements} \\ A &\in \mathbb{R}^{m \times n} && \text{the physics} \\ \varepsilon & && \text{the noise} \end{aligned} \]In NLOS the scene is the brightness of every small cube of the hidden space, and the measurements are the photon-timing histograms. The physics usually loses information: there are fewer useful measurements than unknowns, or some patterns barely register.
Break the physics into simple pieces (the singular value decomposition).
\[ A = U \Sigma V^{\top}, \qquad U^{\top}U = I_m, \qquad V^{\top}V = I_n \] \[ \Sigma = \operatorname{diag}(\sigma_1,\dots,\sigma_r,0,\dots,0), \qquad \sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0, \qquad r = \operatorname{rank}(A) \]The columns of V are perpendicular directions in scene space, one pattern each. Each sigma says how strongly its pattern shows up in the measurements.
\[ v_1, \dots, v_n = \text{the columns of } V, \qquad \sigma_i = \text{the strength of pattern } v_i \text{ in the data} \]Split the patterns into seen and unseen.
\[ V = \big[\, V_r \;\big|\; V_0 \,\big], \qquad V_r = [v_1,\dots,v_r] \;\text{(seen)}, \qquad V_0 = [v_{r+1},\dots,v_n] \;\text{(unseen)} \]The seen patterns are the ones the sensor responds to (the row space). The unseen patterns are the ones it cannot see at all (the null space). The two groups are perpendicular:
\[ V_r^{\top}V_0 = 0 \]Build the two projections. Each one keeps the part of a scene along its own patterns and drops the rest.
\[ P_{\text{meas}} = V_r V_r^{\top}, \qquad P_{\text{null}} = V_0 V_0^{\top} \]The two projections add up to the identity. V is orthogonal, so V times its transpose is the identity:
\[ I_n = VV^{\top} = \big[V_r \,|\, V_0\big]\begin{bmatrix} V_r^{\top} \\ V_0^{\top} \end{bmatrix} = V_rV_r^{\top} + V_0V_0^{\top} = P_{\text{meas}} + P_{\text{null}} \]So every scene splits into two parts. Multiply step 5 by the scene:
\[ x = I_n\,x = P_{\text{meas}}\,x + P_{\text{null}}\,x = x_{\text{meas}} + x_{\text{null}} \]The split is unique, and the two parts are perpendicular:
\[ P_{\text{meas}}P_{\text{null}} = V_r\,(V_r^{\top}V_0)\,V_0^{\top} = 0 \]The sensor cannot see the null part. Apply the physics to it:
\[ A\,P_{\text{null}}\,x = U\Sigma V^{\top}V_0V_0^{\top}x \] \[ V^{\top}V_0 = \begin{bmatrix} V_r^{\top}V_0 \\ V_0^{\top}V_0 \end{bmatrix} = \begin{bmatrix} 0 \\ I_{n-r} \end{bmatrix} \quad\Longrightarrow\quad \Sigma V^{\top}V_0 = \begin{bmatrix} \operatorname{diag}(\sigma_1,\dots,\sigma_r) \cdot 0 \\ 0 \cdot I_{n-r} \end{bmatrix} = 0 \] \[ \Longrightarrow\quad A\,P_{\text{null}}\,x = 0 \quad \text{exactly} \]The last diagonal entries of Sigma, one for each unseen pattern, are zero. So every unseen pattern is multiplied by zero.
So the measurements depend only on the measured part. Put step 6 into the model and use step 7:
\[ y = A(x_{\text{meas}} + x_{\text{null}}) + \varepsilon = A\,x_{\text{meas}} + \underbrace{A\,x_{\text{null}}}_{=\,0} + \varepsilon = A\,x_{\text{meas}} + \varepsilon \]Two scenes that differ only in their null part produce the same measurements. No amount of data can tell them apart.
With noise, "zero" becomes "too small to see". This is the ≈ in the claim. A pattern with a tiny sigma is seen in principle, but its signal is buried in the noise. Pick a threshold tied to the noise level, and call every pattern below it effectively unseen:
\[ \text{signal of pattern } i = \sigma_i\,(v_i^{\top}x), \qquad \text{noise level} \approx \sigma_{\varepsilon}, \qquad \text{threshold} = \tau \] \[ P^{\tau}_{\text{meas}} = \sum_{i:\,\sigma_i \ge \tau} v_i v_i^{\top}, \qquad P^{\tau}_{\text{null}} = I_n - P^{\tau}_{\text{meas}} = \sum_{i:\,\sigma_i < \tau} v_i v_i^{\top} \]How much an effectively unseen part can still change the measurements:
\[ \begin{aligned} \big\| A\,P^{\tau}_{\text{null}}\,x \big\|_2 &= \Big\| \sum_{\sigma_i < \tau} \sigma_i\,(v_i^{\top}x)\,u_i \Big\|_2 \\ &= \Big( \sum_{\sigma_i < \tau} \sigma_i^2\,(v_i^{\top}x)^2 \Big)^{1/2} \\ &\le \tau\,\big\| P^{\tau}_{\text{null}}\,x \big\|_2 \end{aligned} \]The middle step holds because the columns of U are orthonormal. So the effectively unseen part changes the measurements by at most the threshold times its own size, which is below the noise. This "effective null space" is the one used in practice.
What a Bayesian model does with each part. By step 8 the likelihood sees only the measured part:
\[ p(y \mid x) = p(y \mid x_{\text{meas}}) \]Apply Bayes' rule, then the chain rule to the prior:
\[ \begin{aligned} p(x_{\text{meas}}, x_{\text{null}} \mid y) &\propto p(y \mid x_{\text{meas}})\; p(x_{\text{meas}}, x_{\text{null}}) \\ &= p(y \mid x_{\text{meas}})\; p(x_{\text{meas}})\; p(x_{\text{null}} \mid x_{\text{meas}}) \end{aligned} \]Integrate out the measured part to get what the model believes about the unseen part:
\[ p(x_{\text{null}} \mid y) = \int p(x_{\text{null}} \mid x_{\text{meas}})\; p(x_{\text{meas}} \mid y)\; dx_{\text{meas}} \]The data enter only through the belief about the measured part. Everything the model says about the unseen part comes from the prior, conditioned on the measured part, not from the bare prior. A learned (diffusion) prior ties the two together, which is how it invents plausible content in the unseen part: hallucination.
Why confocal NLOS makes this cheap. After the light-cone transform, the physics acts as a 3D convolution with one kernel. A convolution is diagonal in the Fourier basis, so its patterns are Fourier modes and its strengths are the kernel's Fourier magnitudes:
\[ A = F^{-1}\operatorname{diag}(\hat h)\,F \quad\Longrightarrow\quad \sigma_k = |\hat h_k|, \qquad v_k = \text{Fourier mode } k \] \[ P^{\tau}_{\text{meas}} = F^{-1}\operatorname{diag}\big(\mathbb{1}[\,|\hat h_k| \ge \tau\,]\big)\,F, \qquad P^{\tau}_{\text{null}} = I - P^{\tau}_{\text{meas}} \]So the split is one FFT, a mask and an inverse FFT, with no large SVD. To verify against O'Toole, Lindell and Wetzstein (Nature 2018) before relying on it. For nonlinear physics (the rich simulator, SAR), replace the physics matrix by its Jacobian at the reconstructed scene, valid only near that scene:
\[ A \;\longrightarrow\; J = \frac{\partial f}{\partial x} \]How this turns into the test. Take the AI's posterior samples, with mean written as mu. Split the error the same way, and measure calibration on each part separately:
\[ x - \mu = P_{\text{meas}}(x-\mu) + P_{\text{null}}(x-\mu) \] \[ \begin{aligned} \text{coverage}_{\text{meas}} &= \Pr\big[\, P_{\text{meas}}x \in \text{interval}_{\text{meas}} \,\big] \\ \text{coverage}_{\text{null}} &= \Pr\big[\, P_{\text{null}}x \in \text{interval}_{\text{null}} \,\big] \end{aligned} \]Coverage is a secondary measurement here. The main test is step 13.
The new test: find the split from the model alone.
We compare two maps over every part of the scene. One comes from the physics and is the answer key. The other comes from the AI alone and is the guess.
The answer key, from the physics. For each part of the scene, how much of it the sensor cannot see:
\[ m^{\star}_j = \frac{\| P^{\tau}_{\text{null}}\, e_j \|^2}{\| e_j \|^2} \] \[ \text{where}\quad e_j = \text{the scene with only part } j \text{ switched on} \]A value near 1 means the sensor cannot see that part at all. A value near 0 means it sees it fully.
The guess, from the AI alone. Nudge the measurements a little and watch how much each part of the AI's answer moves. If a part barely moves, the answer there does not depend on the evidence.
\[ s_j = \big\| \nabla_{y}\, \hat x_j(y) \big\| \] \[ \hat m_j = 1 - \frac{s_j}{\max_k s_k} \]This uses only the AI's own answer and its own input. It never uses the physics.
The score. How well the guess matches the answer key:
\[ \text{score} = \text{AUROC}\Big( \hat m,\; \mathbb{1}\big[\, m^{\star} > 0.5 \,\big] \Big) \]A score of 1 is a perfect match. A score of 0.5 is no better than guessing. H1 asks whether a method scores well here.
The hard part. A learned prior ties the hidden part of the scene to the seen part. So when the measurements change, the hidden part of the answer can move too, even though no evidence reaches it. A good method must tell "moves with the evidence" apart from "is decided by the evidence". Other methods to compare: how much the AI's spread shrinks compared with its prior spread, and how much its answer changes when only the unseen part of the input changes.
Where the truth comes from
| Source | Real hardware | True shape | Role |
|---|---|---|---|
| Our simulator, simple linear physics (LCT) | No | Yes | H1, H3: the answer key |
| Rich simulator (mitransient: multi-bounce, SPAD noise, odd walls) | No | Yes | Controlled shift, H2 |
| DENALI, CVPR 2026 (MIT, TU Berlin) | Yes | Yes, CAD + 6-DoF poses | Real sim-to-real test |
| Stanford f-k, Wisconsin phasor, NLOST | Yes | No | Check of the measured part only |
| Sayem's SAR simulator | No | Yes | Second testbed, same question |
DENALI is the only real NLOS dataset with known shapes: 72,000 captures of 3D-printed objects, each with a matching simulated capture. The gap between each real capture and its simulated twin is exactly the physics mismatch the method must cope with on real hardware.
Timeline
| Months | Work | Tests |
|---|---|---|
| 0 to 2 | Review, go or no-go | |
| 2 to 6 | Answer key and maps | H1 |
| 6 to 10 | Shift and DENALI | H2 |
| 10 to 14 | Overseer budget, paper 1 | H3 |
| 14 to 18 | Stronger model | H4 |
| 18 to 22 | Model that hides, paper 2 | H5 |
Risks and mitigations
| Risk | Mitigation |
|---|---|
| No map from the model alone finds the null part | Compare several maps (sensitivity, spread versus prior, response to unseen-only changes); a negative result still shows where such methods fail |
| On real data there is no exact answer key | Grade on real DENALI captures by their known shapes, and use the simulated twins to measure how physics mismatch changes the score |
| Real data with known shapes is scarce | DENALI, with a simulated twin for every capture |
| Stage 2 stand-ins are too simple | Report trends as the gap and the hiding strength grow, not single numbers |
Invented parts of a reconstruction look real, and on real hardware the physics is not exact enough to find them. However, in a simulator the physics is exact. Therefore, we use it as an answer key to prove a method that finds invented content from the model alone, under all five difficulties of oversight, and then use it on real sensors, as a concrete step toward checking systems whose answers cannot be verified.