Brain2Face

Brain2Face

A Study on the Relation of EEG Signals and Spatio-temporal Head Avatars

  • 1Max Planck Institute for Informatics, Saarland Informatics Campus
  • 2German Research Center for Artificial Intelligence (DFKI)
Watch video BibTeX
Papersoon Datasetsoon

Input64-channel EEG, 16 shown

Output3D Gaussian head avatar

Subject 01 “raise upper lip, scrunch nose” seen type, held-out session Each avatar is driven by its own subject’s held-out EEG.

Abstract

Driving digital head avatars is classically done by hand-crafted artist design or by image-based facial performance capture. In this work, we ask whether spatio-temporal tracking of the human’s head geometry and appearance can be performed solely from brain signals, i.e., Electroencephalography (EEG). Since prior work mostly focused on activity and emotion recognition, this task is relatively unexplored without even having publicly available high-quality datasets and benchmarks.

Thus, we contribute a dataset featuring paired EEG and human head geometry as well as appearance annotations. We synchronize 64-channel EEG with per-frame expression latent codes of a personalized Gaussian head avatar. Second, we introduce a baseline to the task of expression recovery of a head avatar from EEG signals. To this end, we propose a dedicated EEG-to-expression model that takes a temporal sequence of EEG signals as input and regresses the latent expression parameters of the 3D Gaussian head avatar. Lastly, we validate our capture and algorithmic design choices through a benchmark protocol that we introduce to this task. We believe this work paves the way towards better understanding of the relation between EEG signals and facial surface as well as appearance changes.

Overview video

A dataset of paired EEG and 3D faces

Six participants performed 75 predefined facial expressions in two sessions each, wearing a 64-channel EEG cap inside a multi-camera rig. We fit a personalized Gaussian head avatar to every participant, and a frozen image encoder turns each video frame into a 256-dimensional expression code: the regression target for EEG.

Twelve of the synchronized camera views and the EEG recorded with them (16 of 64 channels). Subject 03, “scream, eyebrows up”.
Inside the capture rig: cameras mounted around a white, octagonal frame
The multi-camera capture rig.
Participants
6

two sessions each

Expression types
75

per session

Paired segments
891

five seconds each

EEG channels
64

recorded at 500 Hz

Cameras
26-33

per session, 29.97 fps

Capture time
4 h 15 min

over 12 sessions

Fitted avatar targets for all six participants, four expressions each.

From video frames to targets

We fit the Universal Head Avatar Prior (UHAP) of UniGAHA to each participant’s multi-view frames. Fitting yields a 512-dimensional identity code and a personalized decoder. A frozen image encoder maps every frontal video frame to a 256-dimensional expression code, and decoding these codes gives the meshes, 3D landmarks and renders that supervise and evaluate the EEG model.

A manual trigger and three jaw clenches at the start and end of each session synchronize the EEG with the video and its audio.

Seen and unseen expressions

The 75 expression types fall into four fixed groups. Seen types are tested in one session but trained in the other; unseen types never appear in training. Each subject gets its own model.

Each square is one expression type. Point at a square to find the same expression in the other session.

    Segments across the six subjects; each subject has its own model.

    From EEG windows to avatar expressions

    An EEG encoder and a convolutional decoder predict one second of expression codes at a time. A frozen, personalized avatar decoder turns them into 3D Gaussians and geometry.

    1. EEG window

      64 channels, one second at 200 Hz, band-passed to 0.5-99.5 Hz

    2. frozenor fine-tuned

      EEG encoder

      Pretrained REVE-base. A one-second window gives one token per electrode.

    3. trained

      Decoder

      A projection, residual blocks across the electrode tokens, a learned resampler and a framewise MLP give : 30 frames of 256 values.

    4. frozen

      Avatar decoder

      The personalized UHAP decoder takes the identity code and outputs Gaussians and a guide mesh.

    5. 3D Gaussian avatar

    Training objective

    with

    Targets come from the fitted avatar. Geometry losses backpropagate through the frozen avatar decoder, so only the EEG encoder and decoder are updated.

    Mean squared error of the expression codes
    Mean absolute error of the decoded mesh vertices, in mm
    Wing loss on 200 3D landmarks of the mesh, in mm
    Code error of an auxiliary head on intermediate decoder features, used only in training

    At test time, consecutive one-second windows each predict 30 frames, and the chunks are concatenated without overlap or smoothing.

    EEG-driven reconstructions

    Each subject has its own model, trained on the cross-session split. Every clip is a held-out recording, drawn at random: five seen and five unseen expression types per subject.

    Input EEG: 10 of 64 channels. Predictions are the raw model output, five consecutive one-second windows with no smoothing. Landmark Wing is the clip’s mean error.

    What matters for decoding faces from EEG

    Which EEG foundation model?

    We freeze nine pretrained EEG encoders (BENDR, BIOT, CBraMod, CodeBrain, EEGPT, LaBraM, LUNA, Signal-JEPA and REVE) and train the same MLP decoder on each. REVE gives the lowest error on all five metrics. Training REVE from scratch, with the encoder unfrozen and the same data and compute, is clearly worse.

    Subject 03, seen expression types.

    Show the table

    The implementations of all nine foundation models come from Braindecode.

    Which electrodes matter?

    We retrain the fine-tuned model with groups of electrodes set to zero and compare it with the model that sees all 64 channels. Bad channels are detected per subject with PyPREP; frontal and temporal electrodes lie closest to the frontalis and temporalis muscles, and occipital electrodes lie over the visual cortex.

    Error against the all-channel model

    Axis range
    Show the table

    Zeroing frontal or temporal electrodes raises the error on seen expression types the most, while zeroing the motor region or the detected bad channels changes it little. On unseen types every variant stays close to the all-channel model, and zeroing the frontal group is marginally best on every metric.

    Mean over all six subjects, fine-tuned REVE-base with .

    Limitations

    Sideways jaw shift, unseen type. The mouth opens, but the jaw does not move to the left. Subject 04, “mouth open, jaw left, show teeth”.
    Large mouth opening, seen type. The predicted scream is weaker and comes later than the target. Subject 01, “scream, eyebrows up”.
    • Six participants Each was recorded in two sessions. Larger studies with more participants and sessions are needed to assess generalization.
    • No new sessions or people Both sessions contribute training data, so the protocol does not test transfer to an unseen session or participant.
    • Noisy signals EEG is far noisier than images, and the reconstructions miss details that image-driven trackers capture.
    • Faces only The model covers facial expressions, not movements of other body parts, which can also affect the signal.

    Citation

    @misc{martinez2026brain2face,
  title  = {Brain2Face: A Study on the Relation of EEG Signals
            and Spatio-temporal Head Avatars},
  author = {Mart{\'\i}nez Mart{\'\i}nez, Camilo Andr{\'e}s and
            D{\"u}nkel, Olaf and Teotia, Kartik and Rekrut, Maurice and
            Theobalt, Christian and Habermann, Marc},
  year   = {2026}
}
    The paper's overview figure: data pairing, offline avatar targets, EEG-to-expression regression and avatar decoding with latent and geometry losses