Where are scanpath predictions used?
Goal-directed scanpath models such as GazeFormer take an image and a query ("find the fork") and predict
where a person will look, in what order, and for how long. Predicted scanpaths can inform foveated rendering, help
interactive systems work out what a user means, and guide assistive interfaces, so each of these systems acts
directly on where the model says attention will land. A corrupted scanpath can propagate directly into downstream
system behavior if there is no intervention point between model output and system action.
How does an attacker get in?
Eye-tracking data is costly to collect, so practitioners fine-tune on a handful of public datasets or reuse
third-party pretrained weights. An attacker only needs to control one upstream artifact: a few poisoned samples in
a shared dataset, or a backdoored checkpoint on a public model hub. Because the model is multimodal, the trigger
can live in the image, the text query, or both.
Why don't classification backdoors carry over?
Classic backdoors flip a discrete label. A scanpath is a continuous, variable-length, spatiotemporal
sequence, and natural outputs are expected to vary from scene to scene. A backdoor that collapses every triggered
prediction onto one fixed path stands out in simple statistics of the poisoned training data, and activation
clustering flags it in several configurations (see below). Our attacks condition the poison
scanpath on each scene, so triggered outputs stay diverse and plausible.
Contributions
- Two scene-conditioned backdoor attacks on multimodal goal-directed scanpath prediction. Spatial misdirection redirects the search toward an attacker-chosen object; duration inflation lengthens the search while still reaching the right target. Both work with visual, text, and multimodal triggers at 2.5 to 10% poison ratios, reaching up to 93.5% attack success rate (ASR) from as few as 540 poisoned samples (2.5%) and up to 75.5% targeted redirection when the attacker target is in the scene, with clean utility close to the unmodified model.
- A defense evaluation against five existing post-training defenses. Fine-pruning comes closest to removing the backdoor but pushes clean utility below the usable threshold; the utility-preserving defenses leave substantial residual ASR. SecureGaze, a defense built for gaze estimation models, does not detect either attack.
- Generalization across trigger designs (size, color, shape, position, word), a second architecture (ART), and a second dataset with a different task (AiR-D, visual question answering).






