Mixed Reality (MR) headsets promise a future of immersive telepresence where virtual humans blend indistinguishably into their real or virtual surroundings. Achieving this vision requires a method capable of capturing a user's motion, estimating their appearance under novel lighting, and understanding the surrounding environment -- all from the constrained viewpoint of a head-mounted display (HMD). Existing approaches treat these as isolated problems: they either focus on driving avatars with baked-in lighting or rely on complex studio setups for relighting. In this paper, we present \methodname, a holistic framework for egocentric telepresence that simultaneously captures full-body human performance, synthesizes photorealistic and relightable appearance, and estimates high dynamic range (HDR) environment maps from a single HMD. First, to ensure accurate motion and surface reconstruction, we propose an egocentric perception module that leverages stereo down-facing cameras to extract dense depth maps, which serve as robust geometric control signals to drive a mesh-based avatar. Second, we introduce a novel neural appearance model that learns to synthesize view-dependent specular and view-independent diffuse shading separately. By employing a specialized ray-sampling strategy, our model generalizes to unseen illumination without relying on restrictive analytical BRDF priors. Third, we enable seamless avatar integration into the physical world via a test-time inverse rendering process, which recovers an HDR environment map by matching the pre-trained avatar’s appearance to the live egocentric camera observations. We demonstrate the efficiency of our system through a social telepresence application, where remote users are coherently relit according to their local physical environment. Extensive experiments show that our individual components and the integrated system significantly outperform the best combinations of state-of-the-art baselines in both geometric accuracy and rendering as well as relighting fidelity.
Overview of EgoRelight. At test time, EgoRelight enables driving a photoreal and relightable full-body avatar solely from egocentric stereo views. Moreover, it supports affordable environment illumination estimation just using the front-facing cameras of an HMD. In detail, we first estimate egocentric pose (green box) and depth maps (purple box), which then drive a mesh-based human character (red box). The recovered surface that encodes the pose of the human then serves as input for our relightable avatar (grey box), which also takes a target illumination for relighting (blue box).
@article{chen2026egorelight,
title={EgoRelight: Egocentric Human Capture and Illumination Recovery for Relightable and Photoreal Avatar Rendering},
author = {Chen, Jianchun and Zhang, Yinda and Pandey, Rohit and Beeler, Thabo and Habermann, Marc and Theobalt, Christian},
journal={ACM Transactions on Graphics (ToG)},
volume={45},
number={4},
pages={1--19},
year={2026},
publisher={ACM New York, NY, USA}
}
This project was supported by the Saarbrücken Research Center for Visual Computing, Interaction, and AI. The authors thank the anonymous reviewers for their thoughtful feedback and suggestions. The authors also thank Guoxing Sun, Pramod Rao, and Andrea Boscolo Camiletto for their assistance with hardware setup, data capture, and help on egocentric pose estimation.