Follow My Eyes

Backdoor Attacks on Goal-Directed
Scanpath Prediction

Diana Romero*1, Mutahar Ali*1, Momin Ahmad Khan*2, Habiba Farrukh1, Fatima Anwar2, Salma Elmalaki1
1University of California, Irvine 2University of Massachusetts Amherst
*Equal contribution
WACV 2027
Pervasive Autonomy Lab

Same model, same query. A trigger in the image, the text query, or both makes a backdoored scanpath model either search for a different object (the knife) or take longer to find the right one, while behaving normally on clean inputs. Every scanpath shown is a real GazeFormer prediction on COCO-Search18.

Abstract

Scanpath prediction models forecast the sequence of fixations a person makes while searching a scene, and gaze-driven applications use their predictions to anticipate where users will look. Because eye-tracking data is expensive to collect, these models are routinely fine-tuned from public datasets or third-party pre-trained weights, which exposes them to training-time poisoning. We present the first backdoor attacks on multimodal scanpath prediction. Unlike classification, this task outputs a continuous, variable-length sequence of fixations, introducing new attack surfaces. We find that a fixed-path backdoor is easy to implant, but it clusters poisoned samples away from clean data, making it detectable.

Thus, we design two backdoor attacks that condition the output to each scene, keeping the triggered outputs diverse and plausible: a spatial misdirection attack that redirects the predicted search toward an attacker-chosen object instead of the queried one, and a duration inflation attack that lengthens the predicted search by inserting extra fixations while preserving correct localization.

We show that our attacks are successful across visual, textual, and multimodal triggers, with the duration inflation attack reaching up to 93.5% attack success from as few as 540 poisoned samples (2.5% of the training data), and the spatial misdirection attack redirecting the search to an attacker-chosen object in up to 75.5% of triggered inputs. We evaluate our attacks against five existing backdoor defenses, including fine-tuning, fine-pruning, neural attention distillation, contrastive learning, and trigger inversion, and show that none removes the backdoor without degrading the model output below the usable threshold. Our attacks generalize across models and datasets, showing that scanpath prediction models are vulnerable to backdoors via data poisoning, and designing an effective defense remains an open problem.

At a Glance

93.5%
of triggered searches can be made to take longer by duration inflation
75.5%
of triggered searches can be redirected to the attacker-chosen object by spatial misdirection
3
trigger types (visual, text, and multimodal) work for both attacks
540
poisoned samples (2.5% of the training data) can implant either backdoor
None
of five existing defenses removes the backdoor while keeping the model usable

Why Backdoor a Scanpath Model?

Where are scanpath predictions used?
Goal-directed scanpath models such as GazeFormer take an image and a query ("find the fork") and predict where a person will look, in what order, and for how long. Predicted scanpaths can inform foveated rendering, help interactive systems work out what a user means, and guide assistive interfaces, so each of these systems acts directly on where the model says attention will land. A corrupted scanpath can propagate directly into downstream system behavior if there is no intervention point between model output and system action.

How does an attacker get in?
Eye-tracking data is costly to collect, so practitioners fine-tune on a handful of public datasets or reuse third-party pretrained weights. An attacker only needs to control one upstream artifact: a few poisoned samples in a shared dataset, or a backdoored checkpoint on a public model hub. Because the model is multimodal, the trigger can live in the image, the text query, or both.

Why don't classification backdoors carry over?
Classic backdoors flip a discrete label. A scanpath is a continuous, variable-length, spatiotemporal sequence, and natural outputs are expected to vary from scene to scene. A backdoor that collapses every triggered prediction onto one fixed path stands out in simple statistics of the poisoned training data, and activation clustering flags it in several configurations (see below). Our attacks condition the poison scanpath on each scene, so triggered outputs stay diverse and plausible.

Contributions

  1. Two scene-conditioned backdoor attacks on multimodal goal-directed scanpath prediction. Spatial misdirection redirects the search toward an attacker-chosen object; duration inflation lengthens the search while still reaching the right target. Both work with visual, text, and multimodal triggers at 2.5 to 10% poison ratios, reaching up to 93.5% attack success rate (ASR) from as few as 540 poisoned samples (2.5%) and up to 75.5% targeted redirection when the attacker target is in the scene, with clean utility close to the unmodified model.
  2. A defense evaluation against five existing post-training defenses. Fine-pruning comes closest to removing the backdoor but pushes clean utility below the usable threshold; the utility-preserving defenses leave substantial residual ASR. SecureGaze, a defense built for gaze estimation models, does not detect either attack.
  3. Generalization across trigger designs (size, color, shape, position, word), a second architecture (ART), and a second dataset with a different task (AiR-D, visual question answering).

Threat Model

Attacker

A data provider who injects poisoned samples into the training corpus, or a model supplier who ships a pretrained model with the backdoor already implanted. Controls a small fraction of training data; cannot change the architecture or training algorithm.

Goals

Utility: behave like a clean model on clean inputs. Attack success: produce the malicious scanpath when the trigger is present. Stealth: triggered outputs must look enough like clean ones to escape corpus screening and post-training detection.

Defender

Holds the possibly compromised model and a small trusted clean set (1,081 samples, 5% of training data). Does not know the trigger modality, pattern, poison ratio, or target behavior.

Method

A fraction ρ of training samples get a trigger τ(I, o) added to the image, the query, or both, and their ground-truth scanpath is replaced with a poison scanpath S†. The backdoored model M′ matches the clean model on clean inputs and outputs S† when the trigger is present.

Backdoor attack overview: training with clean and poisoned data, then deployment with and without trigger
Backdoor pipeline. Poisoned training samples carry a trigger and an attacker scanpath. At deployment the model behaves normally until the trigger appears.
Construction of poison scanpaths for the spatial misdirection and duration inflation attacks
Poison scanpaths. (a) A clean model searches for the attacker's target. (b) Fixations are inserted at neighbor midpoints, each inheriting its predecessor's duration.

Spatial misdirection

Make the model search for an attacker-chosen object o† instead of the queried one. The poison label comes from querying a clean reference model for o† on the same image, M(I, o†) → S†. Because the label is a real search of each scene, poison scanpaths differ image to image. We use o† = knife, a safety-sensitive object among the 18 COCO-Search18 targets.

Duration inflation

Keep the search pointed at the correct target but make it take longer. The poison label is the sample's own clean scanpath with extra fixations inserted at the midpoint of neighboring fixations, each inheriting its predecessor's duration; the fixations displaced to the end get durations sampled from the empirical fixation-duration distribution. We insert two fixations.

Triggers

  • Visual: a 128×128 white patch at the top center, about 0.9% of a 1680×1050 image.
  • Text: a zero-width space (U+200B) in the query. It is invisible and gives the smallest shift in the RoBERTa embedding of the query among the candidates we tested (5.14, against 6.38 to 13.79 for alternatives).
  • Multimodal: both at once.

Why Not a Fixed Path?

The fixed-path backdoor, the scanpath analogue of a fixed-label backdoor, maps every triggered input to one trajectory (here, two fixations at fixed points, 250 ms each). The poisoned corpus then shows sharp spikes at the attacker's positions that a basic exploratory pass over the data reveals, before any defense is run. Activation clustering also flags the fixed-path poison in several configurations (up to 669 poisoned samples), but flags no samples for either of our attacks.

Fixation heatmap of fixed-path poisoned data at 10%
Most frequent fixation points in fixed-path poisoned data at 10%

Fixed-path poisoned COCO-Search18 at ρ = 10%. Left: fixation heatmap. Right: most frequent fixation points. The same artifacts appear at 2.5% and 5% (paper supplement).

Attacks in Action

All scanpaths are real GazeFormer outputs on COCO-Search18, replayed fixation by fixation (with the model's predicted durations in the duration inflation example). Triggered inputs carry the visible white patch. The same scanpaths appear in the paper (main paper Fig. 4b; supplement Fig. 8).

Query: cup, knife in the scene (target-present). On the clean input, the model goes from the image center to the cup. With the trigger, the same query sends the search to the knife beside the fork.

Query: mouse, no knife in the scene (target-absent). On the clean input, the model goes straight to the mouse. With the trigger, the search turns the other way, toward a pen-like object on the left of the desk, and never reaches the mouse.

The triggered output depends on the scene rather than collapsing to one fixed path. Triggered scanpaths can still pass through the queried region when the queried object and the knife are semantically related (fork), share spatial context in cluttered scenes, or when a single salient region dominates, which is why spatial misdirection does not succeed on every input.

Query: mouse, 383 ms → 873 ms. Both searches end on the mouse. The triggered one adds three fixations (orange diamonds) and takes 490 ms longer (+128%).

Results

Attack effectiveness (GazeFormer, COCO-Search18)

Spatial misdirectionDuration inflation
TriggerρUtility ↑ASRTA (%) ↑ASRTP (%) ↑Utility (SSt) ↑ASR (%) ↑Added time
Clean model–0.8660.022.50.4515.5–
Visual2.5%0.78833.826.80.4406.9+7 ms
5%0.79758.529.50.44267.0+111 ms
10%0.83562.561.90.44187.1+259 ms
Text2.5%0.81551.875.50.43189.9+177 ms
5%0.82255.071.90.44390.2+188 ms
10%0.81057.068.50.43995.1+224 ms
Multimodal2.5%0.79746.764.60.44293.5+208 ms
5%0.81054.364.60.43692.7+200 ms
10%0.82058.067.90.43689.5+204 ms

ρ = 2.5 / 5 / 10% is 540 / 1,081 / 2,162 of 21,622 training samples. Attack success rate (ASR) is how often the trigger produces the attacker's intended behavior. Spatial utility is the BBox hit ratio, the fraction of clean inputs whose final fixation lands in the queried target's box. ASRTA (target-absent) is the percent drop in that rate under the trigger on test images without the attacker target (knife). ASRTP (target-present) is the percent of triggered inputs whose final fixation lands in the knife's box, measured on 302 image–query pairs from 207 held-out MS-COCO images that contain both objects; the clean model's 22.5% is the baseline to beat. All other columns use the COCO-Search18 test set. Duration utility is SSt (usable threshold 0.403, the next-best predictor). Duration ASR is the share of triggered inputs whose total viewing time grows by more than δ = 11.5 ms, the 95th percentile of the trigger's effect on a clean model (5% false-positive rate, calibrated on validation, scored on test). Added time is the mean increase in predicted viewing time under the trigger.

Text and multimodal triggers are the most budget-efficient: even at 2.5% poisoning they exceed 45% ASRTA, 64% ASRTP, and 89% duration ASR. We attribute this to the discrete token, which gives the model a signal it learns more easily than a small patch, even from a few (540) poisoned examples. The visual trigger needs more poison: its duration ASR rises from 6.9% at 2.5% to 87.1% at 10%, and its ASRTP stays within a few points of the clean model's 22.5% until 10% (61.9%), although it reaches the highest ASRTA at 10%. The target-absent drop reflects redirection rather than generic failure: at 10% poisoning, on 62.5 to 63.9% of those images the triggered scanpath is more similar to the clean model's search for the knife than to its search for the queried object, against 2.2% for the clean model. Clean utility stays within 0.788 to 0.835 against 0.866 for the clean model, and SSt stays well above the 0.403 threshold in every configuration.

Defenses: none removes the backdoor and keeps the model usable

Legend: fine-tuning, fine-pruning, contrastive, and NAD defenses; visual, text, and multimodal triggers; clean model
Spatial misdirection defense trade-off, target-absent
Spatial misdirection: target-absent.
Spatial misdirection defense trade-off, target-present
Spatial misdirection: target-present.
Duration inflation defense trade-off
Duration inflation.

Clean utility (right is better) against residual ASR (down is better), ρ = 10%. The star is the clean model, where a successful defense would land; the dashed line is the level at which the backdoor counts as removed.

VisualTextMultimodal
DefenseUtility ↑ASRTA ↓ASRTP ↓Utility ↑ASRTA ↓ASRTP ↓Utility ↑ASRTA ↓ASRTP ↓
No defense0.83562.561.90.81057.068.50.82058.067.9
Fine-tuning0.76346.456.60.75552.373.20.76548.865.6
Fine-pruning0.6601.826.20.69313.536.80.71420.940.4
Contrastive0.71433.637.10.70644.062.30.70644.760.9
NAD0.80750.959.90.77653.178.50.78849.873.2
SecureGazeBackdoor not detected, so mitigation never runs (ASR unchanged)
VisualTextMultimodal
DefenseSSt ↑ASR ↓SSt ↑ASR ↓SSt ↑ASR ↓
No defense0.44187.10.43995.10.43689.5
Fine-tuning0.42770.10.43389.20.43391.8
Fine-pruning0.2696.10.1970.70.2358.0
Contrastive0.37766.20.38494.40.38387.8
NAD0.44122.40.43982.40.44576.6
SecureGazeBackdoor not detected, so mitigation never runs (ASR unchanged)

ρ = 10%, defender holds 1,081 clean samples; columns as in the attack table. Red marks SSt below the usable threshold (0.403). Fine-pruning reduces ASR the most, but at a steep utility cost: for spatial misdirection, targeted redirection persists, especially for text and multimodal triggers (ASRTP 36.8 to 40.4%), while utility falls to 0.660 to 0.714; for duration inflation, SSt collapses to 0.197 to 0.269. Fine-tuning and NAD preserve utility but retain substantial targeted redirection (ASRTP 56.6 to 78.5%). For duration inflation, NAD is the only defense that keeps SSt above the threshold while substantially cutting ASR, and only for the visual trigger (22.4%); text and multimodal stay above 70%. SecureGaze's RAV stays between 0.14 and 0.25 for every backdoored model, above its 0.1 collapse threshold, likely because our triggered outputs vary with the scene instead of collapsing.

Robust to trigger design

Trigger (spatial misdirection, ρ = 10%)Utility ↑ASRTA (%) ↑ASRTP (%) ↑
Clean model0.8660.022.5
White 128×128, top center (default)0.83562.561.9
White 64×64, top center0.77558.357.9
White 128×128, bottom right0.81464.663.6
Yellow 128×128, top center0.82765.757.9
Circle r = 64, center0.79959.262.9
Word "shiny" in query0.81561.172.5
Word "red" in query0.78659.974.8

Each model is evaluated with the trigger it was trained with; the clean-model row uses the multimodal trigger. The backdoor does not depend on a specific patch size, color, shape, location, or token, so a range of everyday objects or words could plausibly serve as triggers.

Generalizes to another model and another dataset

ART modelUtility ↑ASR (%) ↑
Clean0.6700.0
Backdoored0.67562.0

Spatial misdirection on ART, a second VLM-based scanpath predictor trained on RefCOCO-Gaze, with the zero-width-space text trigger at ρ = 5%. Knife is not a RefCOCO-Gaze category, so the attacker target is "person". ASR is the percent drop in how often the final fixation lands on the queried target under the trigger. Most goal-directed predictors share this encoder-decoder, multimodal-conditioning design, which indicates that the vulnerability is not specific to GazeFormer.

AiR-DρSpatial dep. (px) ↑Duration ASR (%) ↑
Visual2.5%−0.17.8
5%0.48.1
10%1.19.4
Text2.5%6.639.4
5%14.461.9
10%26.183.1
Multimodal2.5%5.334.5
5%15.549.2
10%23.683.1

AiR-D conditions on full GQA reasoning questions instead of a single category word (307 test questions). Spatial departure is the backdoor-induced final-fixation shift from the clean target, net of the clean model's shift on the same triggered input; the BBox hit ratio is uninformative here because only 69 of the 307 questions have an answer box. Duration ASR uses the same δ-calibrated test, so the visual trigger stays near the 5% clean false-positive floor. Clean ScanMatch stays at 0.262 to 0.271 against 0.270 for the clean model. At ρ = 10%, NAD only lowers duration ASR from 83% to 49% (text) and 53% (multimodal); fine-pruning reaches 27% and 37% at a higher utility cost.

Scope and Limitations

When this threat applies. The attack matters for teams that build gaze-driven systems on top of fine-tuned scanpath predictors, where training data or weights come from outside sources. It needs write access to a small slice of training data (or the ability to publish a checkpoint) and a way to place a trigger at inference time, such as a patch in the image or a token in the query.

  • Spatial misdirection fits settings where the predicted target drives an action, such as interactive systems that work out what a user means, since it changes which object the system believes the user is looking for.
  • Duration inflation fits latency-sensitive settings, such as gaze-contingent rendering and interaction, since it can add hundreds of milliseconds while preserving correct localization.

Limitations. Our evaluation rests mainly on two datasets (COCO-Search18 and MS-COCO) and one model (GazeFormer). The ART (RefCOCO-Gaze) and AiR-D results show the attacks transfer, but they do not cover the full range of models and datasets at risk. We also do not measure end-to-end impact on a downstream system (for example, rendering quality or interaction latency), so the real-world severity of a redirected or inflated scanpath remains to be quantified.

Code

Code for both attacks and for the defense evaluation will be released in the future.

BibTeX

@inproceedings{romero2027followmyeyes,
  title     = {Follow My Eyes: Backdoor Attacks on Goal-Directed Scanpath Prediction},
  author    = {Romero, Diana and Ali, Mutahar and Khan, Momin Ahmad and Farrukh, Habiba and Anwar, Fatima and Elmalaki, Salma},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

Acknowledgements

This work is supported by the U.S. National Science Foundation (NSF) under grant numbers 2339266, 2237485, 2452819, and 2544561. The findings and conclusions presented are solely those of the authors and do not necessarily represent the views of the NSF.