Differentiable Polarized Path Tracing

European Conference on Computer Vision (ECCV), 2026

Pramod Rao1,2,3, Jérémy Riviere4, Xilong Zhou1,2, Abhijeet Ghosh4, Abhimitra Meka4, Thabo Beeler4, Marc Habermann1,2, Christian Theobalt1,2,3, Delio Vicini4

1Max Planck Institute for Informatics    2VIA Research Center    3Saarland Informatics Campus    4Google

Paper Supplementary Code

Project Summary

Differentiable Polarized Path Tracing makes polarized inverse rendering practical. Polarization reveals surface and material cues that intensity images discard, but standard path replay becomes unstable when polarized light transport requires inverting rank-deficient Mueller matrices. DPPT avoids this inversion by caching suffix radiance along each path, enabling stable reverse-mode gradients with much lower memory than conventional automatic differentiation.

Overview of differentiable polarized path tracing.
DPPT matches conventional automatic differentiation on polarized gradient tests, while using substantially less memory and avoiding rank-deficient Mueller-matrix inversions.
Abstract

Physically based differentiable rendering has proven to be a powerful tool for inverse rendering problems (e.g., 3D reconstruction, reflectance estimation, lighting estimation). However, most existing methods operate solely on radiometric intensity, discarding valuable polarization cues that constrain scene geometry and material properties. While forward simulation of polarized light is well-defined via Mueller-Stokes calculus, extending reverse-mode differentiation to this domain presents significant challenges. The rank-deficient nature of common polarimetric operators, such as linear polarizers and diffuse reflections, violates the invertibility assumptions of standard gradient estimators like path replay backpropagation and results in numerical instability. We address this by proposing a robust, polarization-aware differentiable path tracing method. Our approach estimates unbiased gradients through a combination of path replay and local caching. This formulation enables efficient and stable optimization of material and lighting parameters in complex scenes, broadening the applicability of physically based inverse rendering.

Overview

Making polarization practical for inverse rendering

Intensity images measure how much light reaches the camera. Polarization also captures how light is oriented, revealing cues about surface shape, material response, and specular reflection that are otherwise difficult to separate.

Differentiable Polarized Path Tracing (DPPT) brings these cues into physically based inverse rendering. It extends path replay backpropagation with cached suffix replay, avoiding unstable Mueller-matrix inversions while keeping memory far below conventional automatic differentiation.

The cue

Polarization adds information beyond intensity. It helps distinguish surface orientation, material response, and specular effects that can look similar in ordinary rendered images.

The challenge

In polarized rendering, light transport is represented with Stokes vectors and Mueller matrices. Common interactions such as diffuse reflection and linear polarizers can be rank-deficient, making standard path replay unreliable.

The solution

DPPT caches the suffix radiance along each sampled path. During replay, it loads this cached quantity directly instead of recovering it through an unstable matrix inverse.

Challenge

Why standard path replay cannot simply be polarized

Path replay backpropagation is memory-efficient because it replays the same light path and locally cancels detached path contributions. In scalar rendering, this cancellation is just a division.

Scalar replay
\[ L_i = f_i L_{i+1} \quad \Rightarrow \quad L_{i+1} = \frac{L_i}{f_i} \]
Polarized replay
\[ \mathbf{S}_i = \mathbf{M}_i \mathbf{S}_{i+1} \quad \Rightarrow \quad \mathbf{S}_{i+1} = \mathbf{M}_i^{-1}\mathbf{S}_i \]

In polarized rendering, radiance is no longer a scalar. Light is represented by Stokes vectors, and each scattering event applies a \(4 \times 4\) Mueller matrix. The same replay step therefore becomes a matrix inverse.

This inverse is often not defined. Common polarimetric interactions, such as diffuse reflection or a linear polarizer, discard part of the incident polarization state. Their Mueller matrices are rank-deficient, so a direct polarized extension of PRB becomes unstable or produces incorrect gradients.

The hard case: Not every singular matrix is equally problematic. Fully depolarizing diffuse interactions can still be handled with a scalar PRB-style update. The difficult case is a polarization-aware interaction that is non-invertible: it affects the polarization-dependent gradient, but the missing incident polarization state cannot be recovered by replay inversion.

DPPT addresses this by avoiding the inverse altogether: instead of recovering the missing suffix radiance through \(\mathbf{M}^{-1}\), it caches the suffix during the primal pass and reuses it during replay.

Method

Cache the suffix, do not invert it

Standard polarized path replay tries to recover the remaining path contribution by undoing the current Mueller operator. This requires \(\mathbf{M}_i^{-1}\), which is not defined for rank-deficient polarized interactions.

DPPT avoids this recovery step. In the primal pass, it traces the sampled path, stores the local Mueller transport factors, and folds the path backward to build a suffix-radiance cache. During the adjoint replay, the same path is sampled again, but the remaining radiance after vertex \(i\) is loaded directly from the cache as \(L_{i+1}\). The local gradient can then be accumulated using the replayed prefix, the differentiated Mueller operator, and the cached suffix.

This keeps the path-replay structure of PRB, but removes the unstable inversion that makes a direct polarized extension unreliable.

1. Trace

Follow a sampled polarized light path and store the local transport factors.

2. Fold backward

Accumulate the remaining light contribution after every vertex, producing a suffix cache.

3. Replay

Replay the same path and use the cached suffix directly when computing local gradients.

Cached suffix replay method diagram.
Cached suffix replay stores the remaining light contribution after each sampled interaction. During replay, DPPT uses this cached suffix to compute local gradients, rather than reconstructing it through an inverse Mueller matrix.
Method at a glance Cached suffix replay

Instead of recovering the suffix through a Mueller-matrix inverse, DPPT stores it once and reuses it during replay.

PRIMAL PASS
# Trace once and build the suffix cache
path = trace_path(ray)
for vertex in path:
    store_emission(vertex)
    store_mueller_transport(vertex)
suffix = fold_backward(path)
# suffix[i+1] stores the remaining radiance after vertex i
ADJOINT PASS
# Replay the same path and load cached suffixes
β = identity()
for i, vertex in replay_path(ray):
    M = evaluate_mueller_operator(vertex)
    # Use the cached suffix directly.
    # No inverse of M is required.
    grad += backward(β @ M @ suffix[i+1])
    β = β @ M / pdf(vertex)
For very deep paths, the supplement introduces a hybrid variant that stores suffix values only at checkpoint intervals and recomputes missing suffixes when needed.

Results

What DPPT enables

We evaluate DPPT along four axes: gradient correctness, the diversity of polarization-dependent gradients, inverse-rendering reconstruction quality, and runtime/memory behavior for long polarized light paths.

Correct gradients

DPPT closely matches conventional AD across reverse- and forward-mode gradient tests, while avoiding the unstable inverse used by direct polarized PRB.

Complementary cues

Different polarizer configurations produce different gradient structures, revealing information that intensity-only rendering does not provide.

Better reconstructions

Polarized optimization improves material, texture, normal-map, and geometry recovery compared with unpolarized baselines.

Lower memory

Cached replay keeps memory far below conventional AD and is faster than the unbiased polarized RB baseline in the long-path benchmark.

DPPT matches conventional AD gradients without unstable replay inversion

We first verify that DPPT computes the right derivatives. Conventional automatic differentiation provides the reference gradients. DPPT closely matches this reference across scene parameters, while the noise-regularized polarized PRB baseline deviates when rank-deficient Mueller operators strongly affect the objective.

Gradient correctness comparison across scenes, methods, and parameters.

How to read the figure: lower relative error means closer agreement with conventional AD. P-PRB is fast but inaccurate in several polarized cases; DPPT matches AD without relying on unstable Mueller-matrix inversion.

Polarization changes the optimization signal

Changing the analyzer configuration changes the derivative field. For roughness, specular reflectance, and surface normals, different polarization states reveal different spatial structures. These complementary gradients give inverse rendering additional cues for separating material and shape parameters.

Reverse-mode gradients for different polarization configurations.

Polarized observations improve inverse-rendering reconstructions

Once the gradients are correct, polarization can be used directly inside inverse rendering. Across material, texture, polarizer-angle, and normal-map optimization tasks, DPPT uses polarized observations to recover parameters that are closer to the reference than an unpolarized PRB baseline. Conventional AD provides correct gradients, but it runs out of memory in several of these larger settings.

Polarized inverse-rendering comparison across scenes and methods.

Camera polarizer

DPPT optimizes the polarizer angle and suppresses glare. Unpolarized PRB cannot differentiate this polarization parameter.

Diffuse + roughness

Cross/parallel polarized observations help separate diffuse and specular effects, improving texture and roughness recovery.

Diffuse texture

Polarized observations improve floor-texture reconstruction in the Living Room scene.

Normal map

Multi-view polarized optimization recovers sharper normal details on the Marble Bust, especially in fine geometric regions.

Polarization improves single-view geometry reconstruction

We also combine polarized light transport with visibility-aware shape gradients. Under the same initialization and optimization schedule, polarized observations recover sharper details and more faithful silhouettes than the unpolarized baseline.

0.1874 → 0.0904Chamfer distance
65.61° → 46.72°Normal error
Polarized and unpolarized 3D reconstruction comparison.

Cached replay keeps polarized differentiation practical

Polarized rendering is expensive because each RGB radiance value becomes a per-channel Mueller representation. DPPT avoids the full memory cost of conventional AD by using local suffix caches. In the long-path benchmark, it keeps memory low up to depth 32, remains far below conventional AD, and is approximately twice as fast as the unbiased polarized RB baseline.

Stress-test scene
Modern Hall stress-test scene with a stack of linear polarizers.
Increasing path depth
Rendered appearance as maximum path depth increases.
Peak GPU memory
Peak GPU memory benchmark plot; lower is better.
Gradient runtime
Reverse-mode gradient runtime benchmark plot; lower is better.
Long-path benchmark. We stress-test polarized differentiation in the Modern Hall scene by inserting a stack of linear polarizers, which forces long polarized light paths. The top row shows the stress-test scene and the effect of increasing the maximum path depth (N). The bottom row reports peak GPU memory and reverse-mode gradient runtime for optimizing the floor albedo. Conv. AD gives correct gradients but becomes memory-intensive and eventually runs out of memory. P-PRB is fast and memory-efficient, but its gradients are inaccurate in rank-deficient polarized settings. P-RB produces unbiased gradients but is slower. Ours keeps memory low while preserving correct gradients, and is about (2\times) faster than P-RB in this benchmark.

Lower is better for both memory and runtime.

Takeaway: low runtime alone is not sufficient here—P-PRB is efficient but inaccurate. DPPT targets the useful operating point: correct gradients with much lower memory than Conv. AD and lower runtime than P-RB.

Conv. AD

Correct gradients, but high memory and OOM in large settings.

P-PRB

Low memory/runtime, but inaccurate gradients for rank-deficient polarized paths.

P-RB

Unbiased gradients and low memory, but slower runtime in the long-path benchmark.

DPPT

Correct gradients with much lower memory than AD and faster runtime than P-RB.

Hybrid Extension

Sparse cached replay for deeper polarized paths

Fully cached DPPT is efficient for the path depths used in our main inverse-rendering experiments. For deeper paths, however, storing a suffix at every interaction increases the local cache size. The hybrid extension stores suffixes only at checkpoint intervals and recovers missing suffixes inside each block using a polarization-aware rule.

\[ \mathcal{C}_k = \{L_0, L_k, L_{2k}, \ldots\}, \qquad k = 1 \Rightarrow \text{fully cached DPPT}. \]
\[ \text{suffix recovery} = \begin{cases} \text{scalar PRB update}, & \mathrm{DI}(\bar{\beta}) < \gamma, \\[4pt] \bar{\beta}^{-1} L_c, & \mathrm{DI}(\bar{\beta}) \geq \gamma \;\land\; |\det(\bar{\beta})| > \epsilon, \\[4pt] \text{recursive recomputation}, & \text{otherwise}. \end{cases} \]

Here, \(\bar{\beta}\) is the detached block throughput since the last checkpoint, \(L_c\) is the cached suffix at the next checkpoint, \(\gamma\) controls when polarized replay is retained, and \(\epsilon\) prevents unstable inversion of near-singular Mueller matrices.

The depolarization index decides whether polarization still matters inside the block. If the block is effectively depolarizing, scalar PRB is sufficient. If polarization remains important and the block throughput is safely invertible, we recover the suffix by local inversion. Otherwise, we avoid the unstable inverse and recompute the missing suffix recursively.

Depth 64 and 128 benchmark takeaway

Sparse checkpointing keeps memory approximately constant at larger path depths. The parameter \(k\) controls memory, while \(\gamma\) controls how often the algorithm stays in the polarized branch. In practice, \(\gamma = 0.9\) remains close to the fully cached method in runtime while preserving the lower-memory behavior.

The benchmark below uses the same long-path stress test as the main paper, but focuses on deeper paths. Larger checkpoint intervals reduce memory, while the depolarization threshold \(\gamma\) controls how often the algorithm stays in the polarized branch.

Depth 64 memory, \(\gamma = 0.9\)
Hybrid replay memory benchmark at depth 64 with gamma 0.9.
Depth 128 memory, \(\gamma = 0.9\)
Hybrid replay memory benchmark at depth 128 with gamma 0.9.
Depth 64 runtime, \(\gamma = 0.9\)
Hybrid replay runtime benchmark at depth 64 with gamma 0.9.
Depth 128 runtime, \(\gamma = 0.9\)
Hybrid replay runtime benchmark at depth 128 with gamma 0.9.

Hybrid benchmark at large path depths. Sparse checkpointing keeps memory nearly constant for longer paths. Runtime depends on \(\gamma\): when \(\gamma=0\), the polarized branch is selected more often, causing more recursive recomputation; with a practical threshold such as \(\gamma=0.9\), the hybrid method remains closer to the fully cached runtime while preserving the lower-memory behavior.

Lower is better for both memory and runtime.

Hybrid replay preserves the same gradients

The hybrid variant preserves the same gradient behavior as the fully cached method, while offering a different memory-runtime tradeoff.

Changing \(k\) and \(\gamma\) controls how much the hybrid method caches and when it recomputes, but the resulting gradients remain consistent with the fully cached method.

Scene
Hybrid
\(k=2, \gamma=0.5\)
Hybrid
\(k=4, \gamma=0.9\)
Fully cached
Ours
Conv. AD
Reference
Brass Vase scene. Brass Vase Roughness texture
Brass Vase gradient from hybrid replay with k 2 and gamma 0.5.
Brass Vase gradient from hybrid replay with k 4 and gamma 0.9.
Brass Vase gradient from fully cached DPPT.
Brass Vase reference gradient from conventional automatic differentiation.
Staircase scene. Staircase Diffuse texture
Staircase gradient from hybrid replay with k 2 and gamma 0.5.
Staircase gradient from hybrid replay with k 4 and gamma 0.9.
Staircase gradient from fully cached DPPT.
Staircase reference gradient from conventional automatic differentiation.
Hybrid gradient analysis. The first two result columns show two hyperparameter settings of the hybrid extension: \(k=2, \gamma=0.5\) and \(k=4, \gamma=0.9\). We compare them against the fully cached method and conventional AD. Each result tile reports relative error (RE) with respect to Conv. AD; lower is better. The hybrid configurations visually match the fully cached method and preserve nearly the same RE, showing that sparse checkpointing changes the memory-runtime tradeoff without changing the gradient behavior.

RE = relative error to Conv. AD. Lower is better.

Both hybrid columns are the same algorithm with different checkpointing and depolarization settings.

Experiment details behind the inverse-rendering results

Kitchen

A camera linear polarizer is optimized to suppress glare in the cooking area. This directly tests differentiation with respect to a polarization parameter.

Veach

Parallel and cross-polarized observations are used to jointly optimize plate roughness and diffuse texture, helping separate diffuse and specular effects.

Living Room

The same parallel/cross-polarized setup is used for diffuse floor-texture reconstruction.

Marble Bust

A 12-view polarized setup jointly optimizes surface normals and diffuse reflectance to recover a 1K normal map.

Implementation detail: local memory. The suffix cache is local to a sampled path. It is not stored as a global num_rays x path_depth buffer. In Dr.Jit, this is implemented with local memory through alloc_local, so memory use is bounded by concurrently active work items rather than by all paths in the image.

Citation

@InProceedings{rao2026dppt,
  title = {Differentiable Polarized Path Tracing},
  author = {Rao, Pramod and Riviere, Jeremy and Zhou, Xilong and Ghosh, Abhijeet and Meka, Abhimitra and Beeler, Thabo and Habermann, Marc and Theobalt, Christian and Vicini, Delio},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year = {2026}
}

Acknowledgements

This work was supported by the ERC Consolidator Grant 4DRepLy (770784) and the Saarbrücken Research Center for Visual Computing, Interaction and AI. We thank Peter Kultis for helpful discussions, the anonymous reviewers, in particular Reviewer 3, for their constructive feedback and support, and Shrisha Bharadwaj for insightful discussions, careful proofreading, and invaluable support throughout the project.