Research idea · draft · 2026-10-25

Finding What a Generative Model Made Up

A generative model's answer mixes what the data supports with what the model filled in from training. However, both parts look the same to the reader. Therefore, we propose a method that finds the invented parts using only the model, and we prove it in simulation, where the physics provides an exact answer key, so that it can be used on real sensors, where the physics is only approximately known.

  1. 1The probleminvented parts look real
  2. 2The answer keyphysics knows the truth
  3. 3An examplethe mug behind the corner
  4. 4The methoda map from the model alone
  5. 5The testfive hypotheses, two stages
  6. 6The usereal sensors, physics unsure
Figure 0. The story of this page in six steps; each section below is one step.
Step 1 · The problem

Invented parts look real

Powerful models now answer questions people cannot check, such as long proofs, large programs, or a scene no camera can see. However, a model's confident answer may rest partly on evidence and partly on what it guessed from training, and the two look the same. Therefore, overseeing such a model, a task known as scalable oversight, needs a way to point at the invented parts.

Table 1. Five difficulties of overseeing a powerful model, and where this work addresses each one.
DifficultyStageHow we test it
1. No ground truth when the model is used1The simulator knows the truth; the method never sees it
2. The model is stronger than its overseer2The overseer gets a smaller model and a few probes
3. Nobody knows how the evidence links to the answer1The method is not given the physics
4. The world changes after training1Simple simulation, then detailed simulation, then real data
5. The model works against its overseer2A model trained to hide its invented parts
Step 2 · The answer key

Why we study it with physical sensors

On real hardware nobody knows the exact physics, and many reconstructors are black-box networks, so the invented parts of a real reconstruction cannot be computed and no method can be graded there. However, in a simulator the physics is exact and states which parts of a scene the measurements can reveal. Therefore, we use the simulator's physics as an answer key, and give the method nothing but the data and the model, exactly as it would have on real hardware.

Step 3 · An example

The mug behind the corner

In Figure 1, light returns only from the front of the mug, the measured part. No light can return from the back, the null part. However, the model still draws a handle there, because most mugs it saw had one. Therefore, the goal is a method that marks the handle as invented without being told the physics.

A first idea, and why it fails. One could split the answer with the physics and check whether the model's confidence is honest on each part. This fails for three reasons:

Therefore, the question is not "is the confidence honest?" but "can the invented parts be found from the model alone?"

Step 4 · The method

A map from the model alone, graded by the answer key

SIMULATOR · knows the truth True scene x the real mug Physics A the sensor model Answer key m* the true null part what A cannot see METHOD · sees only the data and the model Data y y = A x + noise Generative model diffusion prior Answer x̂ mug with handle Probe nudge y, rerun model Map m̂ parts that barely move Score S does m̂ find m*? SIMULATOR · knows the truth True scene xthe real mug Physics Athe sensor model Answer key m*the true null part METHOD · data and model only Data yy = A x + noise Generative modeldiffusion prior Answer x̂mug with handle Probenudge y, rerun model Map m̂parts that barely move Score Sdoes m̂ find m*?
Figure 2. The simulator makes the data from the true scene and gives the answer key. The method sees only the data and the model: the model gives the answer, and probing the model gives the map. The score compares the map with the answer key.

The answer key: how much of each part of the scene the physics cannot see.

\[ m^{\star}_j = \frac{\| P^{\tau}_{\text{null}}\, e_j \|^2}{\| e_j \|^2} \in [0, 1] \]

The map from the model alone: a part that barely moves when the data is nudged does not rest on the data.

\[ \hat m_j = 1 - \frac{\| \nabla_y \hat x_j(y) \|}{\max_k \| \nabla_y \hat x_k(y) \|} \]

The score: 1 is a perfect match, 0.5 is chance.

\[ S = \text{AUROC}\Big( \hat m,\; \mathbb{1}\big[\, m^{\star} > 0.5 \,\big] \Big) \]

The hard part. The data cannot see the null part, yet the model's guess for it still moves when the data moves, because the model learned that the hidden part goes with the seen part:

\[ \underbrace{\frac{\partial y}{\partial x_{\text{null}}} = A\,P_{\text{null}} = 0}_{\text{the data cannot see it}} \qquad\text{but}\qquad \underbrace{\frac{\partial \hat x_{\text{null}}}{\partial y} \neq 0}_{\text{the guess still moves}} \]

Therefore, the simple map above is only a first candidate. Better candidates compare the spread of the model's answers before and after seeing the data, or change only the part of the input the physics cannot see, and the score decides which candidate works.

Step 5 · The test

Stage 1, then stage 2

Stage 1 runs the test of Figure 2 as the world changes, from simple simulation to detailed simulation to real data. However, a small model is neither stronger than its overseer nor trying to fool it. Therefore, stage 2 adds both conditions, as Figure 3 shows.

A STRONGER MODEL Data y Generator large, grows Overseer small model, k probes k probes capability gap Δ: the generator grows, the overseer stays fixed A MODEL THAT HIDES Generator trained to fool the overseer Overseer map m̂ from probes Answer key m* from the simulator Score S training loss: reconstruction + λ S A STRONGER MODEL Data y Generatorlarge, grows Overseersmall model, k probes k probes gap Δ: generator grows, overseer fixed A MODEL THAT HIDES Generatortrained to fool Overseer map m̂from probes Answer key m*from the simulator Score S training loss: reconstruction + λ S
Figure 3. Left: a small overseer with a few probes checks an ever larger generator. Right: the generator is trained to lower the overseer's score while still reconstructing the scene.

The hiding generator's training goal, where a larger weight means trying harder to fool:

\[ \min_{\theta}\; \mathcal{L}_{\text{recon}}(\theta) \;+\; \lambda\, S(\theta), \qquad \lambda \ge 0 \]

Both are stand-ins, as in current oversight research: a larger model is not a truly more capable one, and a trained deceiver did not choose to deceive.

Five hypotheses

H1
\[ S > 0.5 \quad\text{clearly} \]

Stage 1. The map from the model alone finds the true null part.

H2
\[ R(\delta) = \frac{\mathbb{E}\,\| P_{\text{null}}(x - \hat x) \|^2}{\mathbb{E}\,\| P_{\text{meas}}(x - \hat x) \|^2}\ \text{grows with}\ \delta, \qquad S(\delta)\ \text{stays high} \]

Stage 1. As the world shifts by an amount δ, errors gather in the null part, and the map still marks it.

H3
\[ \text{caught}_{\text{map}}(B) > \text{caught}_{\text{uncertainty}}(B) \]

Stage 1. An overseer can check only B parts of each answer. Choosing them by the map catches more invented content than choosing them by the model's uncertainty, with no ground truth.

H4
\[ S(\Delta)\ \text{falls gradually as}\ \Delta\ \text{grows} \]

Stage 2. The method degrades gently, not suddenly, as the overseer falls behind.

H5
\[ S_{\text{hiding}}(\lambda) > 0.5 \quad\text{for every tested}\ \lambda \]

Stage 2. A generator trained to hide cannot push the overseer down to chance.

Step 6 · The use

From the simulator to real sensors

Table 2. The same method where it is proven, where it is needed in imaging, and where the long-term goal lies.
Simulator (proof)Real sensor (use)Powerful AI system (long-term goal)
AnswerReconstructed sceneReconstructed sceneThe system's answer
EvidenceSimulated measurementsReal photon or radar measurementsThe input the system was given
Physics linking evidence to answerExact: an answer keyApproximate: timing jitter, wall shape, reflectance, antenna motion errorsUnknown
ReconstructorDiffusion model with the physicsDiffusion model or a black-box networkA black box
Probe the method usesNudge the measurementsNudge the measurementsChange the input
Invented partHandle on the unseen sideStructure the measurements never constrainedPart of the answer no input supports

On real hardware the physics is only approximately known, and many reconstructors are black-box networks, so the invented parts of a real reconstruction cannot be computed from the physics. However, the method needs only the data and the model, so it applies there unchanged. Therefore, the simulator is where the method is proven against an exact answer key, real sensors are where it is needed, and any system whose answers cannot be checked is where the same idea points.

Prior work

What exists, and what is missing

Table 3. Closest prior work.
WorkWhat it doesWhat it lacks here
Learned NLOS reconstructors (e.g. Li et al. 2023, NLOST)Fast reconstruction of real captures with deep networksNo marking of what the network invented
Bhadra et al. 2021Maps invented content in CT reconstructionsNeeds the known physics
Teneggi et al. 2023Statistical guarantees on diffusion confidenceNeeds the known physics; no shift
Sandwiching (Cotra 2021; Bowman et al. 2022)Tests oversight where the truth is hidden from the overseerNo exact answer key
Weak-to-strong (Burns et al. 2023)A small model supervises a large oneNot about invented content
AI control (Greenblatt et al. 2023)A model trained to sabotage, against a monitorNo exact answer key
Scaling laws for oversight (2025)Oversight success falls as the capability gap growsGames, not reconstructions

No work finds invented content from the model alone and grades it against an exact answer key.

Mathematical formulation

How an answer divides into two parts

answer x from the model P_meas x · measured part The measurements constrain it. We can check it against data. P_null x · null part The measurements say nothing. Only the prior decides it. answer xfrom the model measured partThe measurements constrain it. null partThe measurements say nothing; the prior decides.
Figure 4. The null part is where hallucination lives: the model fills it from what it learned, given the seen part, not from evidence. Physics makes this split exact, which general AI cannot.
  1. The measurement model. The hidden scene is a list of unknown numbers. The sensor records a list of measured numbers, made by known physics from the scene, plus noise.

    \[ y = A x + \varepsilon \] \[ \begin{aligned} x &\in \mathbb{R}^{n} && \text{the scene} \\ y &\in \mathbb{R}^{m} && \text{the measurements} \\ A &\in \mathbb{R}^{m \times n} && \text{the physics} \\ \varepsilon & && \text{the noise} \end{aligned} \]

    In NLOS the scene is the brightness of every small cube of the hidden space, and the measurements are the photon-timing histograms. The physics usually loses information: there are fewer useful measurements than unknowns, or some patterns barely register.

  2. Break the physics into simple pieces (the singular value decomposition).

    \[ A = U \Sigma V^{\top}, \qquad U^{\top}U = I_m, \qquad V^{\top}V = I_n \] \[ \Sigma = \operatorname{diag}(\sigma_1,\dots,\sigma_r,0,\dots,0), \qquad \sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0, \qquad r = \operatorname{rank}(A) \]

    The columns of V are perpendicular directions in scene space, one pattern each. Each sigma says how strongly its pattern shows up in the measurements.

    \[ v_1, \dots, v_n = \text{the columns of } V, \qquad \sigma_i = \text{the strength of pattern } v_i \text{ in the data} \]
  3. Split the patterns into seen and unseen.

    \[ V = \big[\, V_r \;\big|\; V_0 \,\big], \qquad V_r = [v_1,\dots,v_r] \;\text{(seen)}, \qquad V_0 = [v_{r+1},\dots,v_n] \;\text{(unseen)} \]

    The seen patterns are the ones the sensor responds to (the row space). The unseen patterns are the ones it cannot see at all (the null space). The two groups are perpendicular:

    \[ V_r^{\top}V_0 = 0 \]
  4. Build the two projections. Each one keeps the part of a scene along its own patterns and drops the rest.

    \[ P_{\text{meas}} = V_r V_r^{\top}, \qquad P_{\text{null}} = V_0 V_0^{\top} \]
  5. The two projections add up to the identity. V is orthogonal, so V times its transpose is the identity:

    \[ I_n = VV^{\top} = \big[V_r \,|\, V_0\big]\begin{bmatrix} V_r^{\top} \\ V_0^{\top} \end{bmatrix} = V_rV_r^{\top} + V_0V_0^{\top} = P_{\text{meas}} + P_{\text{null}} \]
  6. So every scene splits into two parts. Multiply step 5 by the scene:

    \[ x = I_n\,x = P_{\text{meas}}\,x + P_{\text{null}}\,x = x_{\text{meas}} + x_{\text{null}} \]

    The split is unique, and the two parts are perpendicular:

    \[ P_{\text{meas}}P_{\text{null}} = V_r\,(V_r^{\top}V_0)\,V_0^{\top} = 0 \]
  7. The sensor cannot see the null part. Apply the physics to it:

    \[ A\,P_{\text{null}}\,x = U\Sigma V^{\top}V_0V_0^{\top}x \] \[ V^{\top}V_0 = \begin{bmatrix} V_r^{\top}V_0 \\ V_0^{\top}V_0 \end{bmatrix} = \begin{bmatrix} 0 \\ I_{n-r} \end{bmatrix} \quad\Longrightarrow\quad \Sigma V^{\top}V_0 = \begin{bmatrix} \operatorname{diag}(\sigma_1,\dots,\sigma_r) \cdot 0 \\ 0 \cdot I_{n-r} \end{bmatrix} = 0 \] \[ \Longrightarrow\quad A\,P_{\text{null}}\,x = 0 \quad \text{exactly} \]

    The last diagonal entries of Sigma, one for each unseen pattern, are zero. So every unseen pattern is multiplied by zero.

  8. So the measurements depend only on the measured part. Put step 6 into the model and use step 7:

    \[ y = A(x_{\text{meas}} + x_{\text{null}}) + \varepsilon = A\,x_{\text{meas}} + \underbrace{A\,x_{\text{null}}}_{=\,0} + \varepsilon = A\,x_{\text{meas}} + \varepsilon \]

    Two scenes that differ only in their null part produce the same measurements. No amount of data can tell them apart.

  9. With noise, "zero" becomes "too small to see". This is the ≈ in the claim. A pattern with a tiny sigma is seen in principle, but its signal is buried in the noise. Pick a threshold tied to the noise level, and call every pattern below it effectively unseen:

    \[ \text{signal of pattern } i = \sigma_i\,(v_i^{\top}x), \qquad \text{noise level} \approx \sigma_{\varepsilon}, \qquad \text{threshold} = \tau \] \[ P^{\tau}_{\text{meas}} = \sum_{i:\,\sigma_i \ge \tau} v_i v_i^{\top}, \qquad P^{\tau}_{\text{null}} = I_n - P^{\tau}_{\text{meas}} = \sum_{i:\,\sigma_i < \tau} v_i v_i^{\top} \]

    How much an effectively unseen part can still change the measurements:

    \[ \begin{aligned} \big\| A\,P^{\tau}_{\text{null}}\,x \big\|_2 &= \Big\| \sum_{\sigma_i < \tau} \sigma_i\,(v_i^{\top}x)\,u_i \Big\|_2 \\ &= \Big( \sum_{\sigma_i < \tau} \sigma_i^2\,(v_i^{\top}x)^2 \Big)^{1/2} \\ &\le \tau\,\big\| P^{\tau}_{\text{null}}\,x \big\|_2 \end{aligned} \]

    The middle step holds because the columns of U are orthonormal. So the effectively unseen part changes the measurements by at most the threshold times its own size, which is below the noise. This "effective null space" is the one used in practice.

  10. What a Bayesian model does with each part. By step 8 the likelihood sees only the measured part:

    \[ p(y \mid x) = p(y \mid x_{\text{meas}}) \]

    Apply Bayes' rule, then the chain rule to the prior:

    \[ \begin{aligned} p(x_{\text{meas}}, x_{\text{null}} \mid y) &\propto p(y \mid x_{\text{meas}})\; p(x_{\text{meas}}, x_{\text{null}}) \\ &= p(y \mid x_{\text{meas}})\; p(x_{\text{meas}})\; p(x_{\text{null}} \mid x_{\text{meas}}) \end{aligned} \]

    Integrate out the measured part to get what the model believes about the unseen part:

    \[ p(x_{\text{null}} \mid y) = \int p(x_{\text{null}} \mid x_{\text{meas}})\; p(x_{\text{meas}} \mid y)\; dx_{\text{meas}} \]

    The data enter only through the belief about the measured part. Everything the model says about the unseen part comes from the prior, conditioned on the measured part, not from the bare prior. A learned (diffusion) prior ties the two together, which is how it invents plausible content in the unseen part: hallucination.

  11. Why confocal NLOS makes this cheap. After the light-cone transform, the physics acts as a 3D convolution with one kernel. A convolution is diagonal in the Fourier basis, so its patterns are Fourier modes and its strengths are the kernel's Fourier magnitudes:

    \[ A = F^{-1}\operatorname{diag}(\hat h)\,F \quad\Longrightarrow\quad \sigma_k = |\hat h_k|, \qquad v_k = \text{Fourier mode } k \] \[ P^{\tau}_{\text{meas}} = F^{-1}\operatorname{diag}\big(\mathbb{1}[\,|\hat h_k| \ge \tau\,]\big)\,F, \qquad P^{\tau}_{\text{null}} = I - P^{\tau}_{\text{meas}} \]

    So the split is one FFT, a mask and an inverse FFT, with no large SVD. To verify against O'Toole, Lindell and Wetzstein (Nature 2018) before relying on it. For nonlinear physics (the rich simulator, SAR), replace the physics matrix by its Jacobian at the reconstructed scene, valid only near that scene:

    \[ A \;\longrightarrow\; J = \frac{\partial f}{\partial x} \]
  12. How this turns into the test. Take the AI's posterior samples, with mean written as mu. Split the error the same way, and measure calibration on each part separately:

    \[ x - \mu = P_{\text{meas}}(x-\mu) + P_{\text{null}}(x-\mu) \] \[ \begin{aligned} \text{coverage}_{\text{meas}} &= \Pr\big[\, P_{\text{meas}}x \in \text{interval}_{\text{meas}} \,\big] \\ \text{coverage}_{\text{null}} &= \Pr\big[\, P_{\text{null}}x \in \text{interval}_{\text{null}} \,\big] \end{aligned} \]

    Coverage is a secondary measurement here. The main test is step 13.

  13. The new test: find the split from the model alone.

    We compare two maps over every part of the scene. One comes from the physics and is the answer key. The other comes from the AI alone and is the guess.

    The answer key, from the physics. For each part of the scene, how much of it the sensor cannot see:

    \[ m^{\star}_j = \frac{\| P^{\tau}_{\text{null}}\, e_j \|^2}{\| e_j \|^2} \] \[ \text{where}\quad e_j = \text{the scene with only part } j \text{ switched on} \]

    A value near 1 means the sensor cannot see that part at all. A value near 0 means it sees it fully.

    The guess, from the AI alone. Nudge the measurements a little and watch how much each part of the AI's answer moves. If a part barely moves, the answer there does not depend on the evidence.

    \[ s_j = \big\| \nabla_{y}\, \hat x_j(y) \big\| \] \[ \hat m_j = 1 - \frac{s_j}{\max_k s_k} \]

    This uses only the AI's own answer and its own input. It never uses the physics.

    The score. How well the guess matches the answer key:

    \[ \text{score} = \text{AUROC}\Big( \hat m,\; \mathbb{1}\big[\, m^{\star} > 0.5 \,\big] \Big) \]

    A score of 1 is a perfect match. A score of 0.5 is no better than guessing. H1 asks whether a method scores well here.

    The hard part. A learned prior ties the hidden part of the scene to the seen part. So when the measurements change, the hidden part of the answer can move too, even though no evidence reaches it. A good method must tell "moves with the evidence" apart from "is decided by the evidence". Other methods to compare: how much the AI's spread shrinks compared with its prior spread, and how much its answer changes when only the unseen part of the input changes.

Data

Where the truth comes from

Table 4. Data sources, whether each comes from real hardware, and whether the true shape is known.
SourceReal hardwareTrue shapeRole
Our simulator, simple linear physics (LCT)NoYesH1, H3: the answer key
Rich simulator (mitransient: multi-bounce, SPAD noise, odd walls)NoYesControlled shift, H2
DENALI, CVPR 2026 (MIT, TU Berlin)YesYes, CAD + 6-DoF posesReal sim-to-real test
Stanford f-k, Wisconsin phasor, NLOSTYesNoCheck of the measured part only
Sayem's SAR simulatorNoYesSecond testbed, same question

DENALI is the only real NLOS dataset with known shapes: 72,000 captures of 3D-printed objects, each with a matching simulated capture. The gap between each real capture and its simulated twin is exactly the physics mismatch the method must cope with on real hardware.

Plan

Timeline

0246810121416182022 month Review, go or no-go Answer key and maps · H1 Shift and DENALI · H2 Overseer budget · H3 Stronger model · H4 Model that hides · H5 paper 1 paper 2
Figure 5. Stage 1 (months 0 to 14) and stage 2 (months 14 to 22). Months are a draft.
Figure 5 as a table (phone layout).
MonthsWorkTests
0 to 2Review, go or no-go
2 to 6Answer key and mapsH1
6 to 10Shift and DENALIH2
10 to 14Overseer budget, paper 1H3
14 to 18Stronger modelH4
18 to 22Model that hides, paper 2H5
Risks

Risks and mitigations

Table 5. Risks and how we handle them.
RiskMitigation
No map from the model alone finds the null partCompare several maps (sensitivity, spread versus prior, response to unseen-only changes); a negative result still shows where such methods fail
On real data there is no exact answer keyGrade on real DENALI captures by their known shapes, and use the simulated twins to measure how physics mismatch changes the score
Real data with known shapes is scarceDENALI, with a simulated twin for every capture
Stage 2 stand-ins are too simpleReport trends as the gap and the hiding strength grow, not single numbers
Summary

Invented parts of a reconstruction look real, and on real hardware the physics is not exact enough to find them. However, in a simulator the physics is exact. Therefore, we use it as an answer key to prove a method that finds invented content from the model alone, under all five difficulties of oversight, and then use it on real sensors, as a concrete step toward checking systems whose answers cannot be verified.