World Models as a Persistent Latent Memory
Store for Long-Context Wearable AI

Abstract

This paper investigates whether the Joint-Embedding Predictive Architecture (V-JEPA), trained to predict scene dynamics rather than align pixels to language, can serve as a compact, database-free memory substrate for egocentric video question answering on wearable devices. We bridge a frozen V-JEPA vision encoder to a frozen Qwen2-VL-2B-Instruct language model (~2B parameters) via a 2-layer MLP projection (~3.94M trainable parameters, under 0.2% of the combined ~2.3B-parameter system), after pooling the encoder's 8,192 output tokens down to 16 via adaptive average pooling.

Evaluated on 513 verified instances from the EgoLongQA benchmark, the system achieves 1.2% exact-match accuracy and mean ROUGE-L F1 of 0.23, with qualitative analysis showing a consistent pattern: causal and physical scene understanding — reasoning about building shadows preserving snow, re-identifying objects across viewpoint changes — is preferentially preserved, while fine-grained alphanumeric detail (prices, names, printed text) is frequently lost.

Retrieval quality remains roughly stable across a 5.8× range of memory library sizes (110 to 635 videos), a result we show is consistent with the bridge already operating in an overfitting regime, where additional training reduces training loss without improving held-out performance. We discuss this as an efficiency–fidelity tradeoff intrinsic to extreme latent compression, and identify a systematic no-JEPA baseline and held-out-loss-based checkpoint selection as the most direct next steps.

Keywords: egocentric video understanding · world models · persistent memory · latent compression

1 · Introduction

The advent of next-generation wearable devices, equipped with first-person, head-mounted cameras, has catalyzed interest in real-time multimodal contextual assistants. Unlike traditional AI systems, these wearable agents are expected to "watch" the world through the user's eyes, anticipating needs based on hours of continuous egocentric visual data. Achieving this requires solving a long-context memory challenge: maintaining a coherent, queryable account of an extended stream, under the strict power and compute constraints of on-device inference.

We investigate V-JEPA, a video encoder trained to predict a scene's own latent dynamics rather than align visual content with language, as a candidate memory substrate for this setting. We compress V-JEPA's output to 16 tokens via adaptive average pooling and bridge it to a frozen Qwen2-VL-2B-Instruct decoder through a small trainable MLP, avoiding both an external retrieval index and the growing decoder-side sequence-length cost of passing full-length visual sequences through the decoder at query time. Evaluated on 513 verified EgoLongQA instances, we find a consistent split in what survives this compression: causal and physical scene understanding is reliably preserved, while precise alphanumeric detail is not — a pattern we tie to V-JEPA's training objective and characterize quantitatively in Sections 5 and 6, alongside a training-dynamics finding that bears directly on how the library-size results should be interpreted.

We frame our contribution as an empirical scoping study rather than a demonstration of a finished system: we test whether a minimal, frozen-encoder, frozen-decoder, lightweight-bridge design is sufficient for this task, and report both what it preserves and an overfitting pattern that complicates any simple story about scaling.

2 · Related work

2.1 · Egocentric video understanding

Egocentric vision research has been largely shaped by large-scale datasets such as Ego4D, EPIC-KITCHENS, and Ego-Exo4D, which provide extensive first-person video for action recognition and spatial reasoning. These datasets often focus on short-duration clips or specific task labels, whereas long-form question answering over 10+ minute streams — the setting we study here — requires temporal grounding across a much longer horizon, and models trained primarily on short-clip datasets often struggle to maintain a coherent account of events across such extended durations.

2.2 · Vision-language models and CLIP-based architectures

The current generation of multimodal LLMs frequently uses CLIP-based vision encoders, trained via contrastive image–text alignment, to translate visual frames into linguistic tokens. This objective rewards preserving the kind of fine-grained, literal detail a caption would need to name. V-JEPA, in contrast, is trained to predict a scene's own latent dynamics, giving it no direct incentive to preserve caption-relevant detail.

This is a plausible explanation for the pattern reported in Sections 5 and 6, but we note it as a hypothesis consistent with the architecture's training objective, not something tested directly: this paper does not include a trained CLIP-bridge baseline evaluated on the same 513 instances, and that comparison remains the most direct way to confirm whether the tradeoff is specific to JEPA or common to any frozen, pooled encoder bridge.

There also exists VL-JEPA, which extends JEPA to vision-language tasks by predicting continuous text embeddings rather than autoregressive tokens, achieving comparable VQA performance with fewer parameters and selective decoding.

2.3 · Long-term memory without external retrieval

Existing long-memory systems for egocentric video often rely on external databases or retrieval-augmented generation pipelines to manage temporal history. Our approach instead pools the video encoder's own latent tensor to a fixed 16-token summary and treats it directly as memory, avoiding a separate retrieval index and limiting the sequence length passed to the decoder at query time regardless of how much video has been encoded.

3 · Methodology

Our framework investigates whether V-JEPA's latent world-model representation, compressed to a fixed-size summary, can serve as a usable memory substrate for egocentric video question answering.

3.1 · System architecture

• V-JEPA encoder: pre-trained vjepa2-vitl-fpc64-256, producing a (1, 8192, 1024) tensor per video.
• Adaptive average pooling: torch.nn.functional.adaptive_avg_pool1d reduces the 8,192-token sequence to a fixed 16 tokens per video, applied once at encoding time and cached.
• MLP bridge: Linear(1024 → 1536) → GELU → Linear(1536 → 1536), approximately 3.94M trainable parameters, mapping pooled JEPA features into Qwen2-VL-2B-Instruct's embedding space.
• Language decoder: frozen Qwen2-VL-2B-Instruct (~2B parameters); pooled, projected tokens are concatenated with the text prompt and injected via inputs_embeds.

Both the video encoder and the language model are frozen throughout; only the MLP bridge receives gradient updates.

3.2 · The 16-token bottleneck

Pooling to 16 tokens is applied once per video at encoding time, not per query, so the decoder's per-query sequence length does not grow with the size of the memory library. We hypothesize this bottleneck functions as a semantic filter, prioritizing global spatial-temporal gist over local, high-frequency detail; Sections 5 and 6 evaluate this directly.

4 · Experimental setup

4.1 · Dataset and tasks

Our experiments use the Wearable AI Dataset, a benchmark for egocentric video understanding consisting of 2,100 high-resolution (1080p, 15 fps) videos across three tasks. We evaluate on the EgoLongQA task, which targets a 700-video subset requiring long-form (10+ minute), compositional question answering with dense temporal grounding. Results reported in Sections 5 and 6 reflect 513 verified instances from this subset; the remaining instances could not be verified against source logs at time of submission and are excluded rather than reported.

4.2 · Implementation details

The architecture uses Qwen2-VL-2B-Instruct and vjepa2-vitl-fpc64-256. To manage the dataset's raw footprint (317 GB) on standard hardware, we used a Stream–Process–Purge pipeline: each raw video is encoded once by the frozen V-JEPA encoder, pooled to 16 tokens via adaptive average pooling, cached to disk (on the order of tens of kilobytes per video), and the source .mp4 is then deleted. All training and evaluation operate entirely on these cached, pooled embeddings; no raw video is re-read after this initial encoding step.

4.3 · Training dynamics

Checkpoints in our training log were selected on training-loss improvement, not held-out loss. Across logged epochs, average training loss fell from 1.89 to 1.26, while held-out loss simultaneously rose from 1.95 to as high as 2.98 — a standard overfitting signature. We report this here as an experimental-setup fact and address its implications for the library-size results directly in Section 6.

5 · Results and analysis

5.1 · Quantitative evaluation

We evaluate on N = 513 verified EgoLongQA instances drawn from the Wearable AI Dataset. Exact-match accuracy is 1.2% (6/513). Loss and ROUGE-L F1 are moderately anti-correlated (r = −0.36), consistent with loss serving as a rough proxy for output quality without fully determining it.

ROUGE-L F1 remains within a comparatively narrow band (0.17–0.25) across a 5.8× range in library size, with no consistent monotonic trend in either direction. Teacher-forced loss, however, is at its highest at the largest library size tested (635), a point we return to in Section 6 in light of the overfitting pattern: since checkpoints were selected on training loss rather than held-out loss, later, larger-library snapshots are not guaranteed to reflect a more capable model, only a more optimized one on the training set.

5.2 · Qualitative error analysis

Automated lexical-overlap metrics penalize responses that are semantically correct but phrased differently from the reference answer, and reward responses that share incidental words with the reference despite being factually wrong. To characterize what the compressed latent actually preserves, we manually inspect a sample of model outputs across three tiers of agreement with ground truth.

Category-consistent hits. In several cases the model's output matches the ground truth's underlying state even when the two are lexically distinct. For a loop-closure question on video 2d40456b, asking whether the user later returned to the stone-pillared entrance where they first entered the park, the model answered "Yes," matching ground truth exactly — indicating the pooled representation retains enough spatial context to resolve a return-to-origin query over a multi-segment walk. On video 3d23a421, the model correctly re-identified a red "RI" installation viewed first from outside a tram station and later from inside the tram ("Yes, I saw it from a different angle on the tram"), matching the ground truth's description of a viewpoint change — evidence of some viewpoint-invariant object persistence in the latent representation.

Semantic near-misses. A larger share of outputs land in an intermediate zone: the model identifies the correct general context or category but errs on a specific entity within it. Asked to track two cyclists (video 2a35434e), the model correctly identified which cyclist was still being followed at the end of the video (black jacket, neon helmet) but described the other cyclist as having "fallen off their bike," where the ground truth states only that the second cyclist was "no longer visible" — a plausible but unsupported elaboration. Similarly, when asked about an artificial-reef exhibit (video f4614dd9), the model's answer ("created to provide a home for marine life, similar to an artificial reef") captures the general ecological function described in the ground truth ("grows coral to be relocated to artificial reefs") without reproducing the specific relocation mechanism.

Physical and causal reasoning. One notable case (video f06bcd6a) involved a question about why snow patches persisted on grass along a walking route. The model's answer — that the patches of snow were on grass shaded by apartment buildings, and that this pattern repeated during the walk — matches the causal structure of the ground truth (buildings casting shade, preserving snow, encountered repeatedly along the route). This is a single example and should be read as illustrative rather than representative; we do not yet have an aggregate rate at which this kind of multi-step causal inference succeeds versus fails.

Fine-grained bounding failures. The clearest and most consistent failure mode involves alphanumeric and low-level perceptual detail — colors, prices, brand names, and printed text — rather than category-level understanding. Representative cases:

• Scaffolding described as "white and black" versus ground truth "white and silver" (video 2ba0accb)
• A hoodie described as "light green" versus "blue" (video 33a1126a)
• A price tag read as "$125.00" versus a ground truth of "$69.99" (video ealebfee), where the model correctly identified that a price was present but not its value
• A pharmacy sign read as "P. J. Feltner Pharmacy" versus "Curson Pharmacy" (video 13604bf4)
• A lug-nut replacement task where the model gave a plausible but incorrect material-based explanation instead of the alloy-versus-steel distinction in the ground truth (video 0efc2689)

In every case above, the model correctly recovers the category of the answer (a color was present, a price was present, a named business was present) but not the exact value — consistent with the hypothesis that the 16-token bottleneck filters out low-level perceptual detail while retaining scene- and task-level structure.

5.3 · Parameter efficiency

A distinguishing property of the proposed bridge is its parameter footprint relative to the frozen backbones it connects. The V-JEPA 2 ViT-L encoder contributes approximately 300M frozen parameters; the Qwen2-VL-2B-Instruct language decoder contributes approximately ~2B frozen parameters. The trainable component, a 2-layer MLP (Linear(1024 → 1536) → GELU → Linear(1536 → 1536)) mapping the pooled JEPA feature space into the language embedding space, comprises approximately 3.94M parameters — under 0.2% of the combined ~2.3B-parameter system.

6 · Discussion: the latent information hierarchy

The systemic pattern running through our empirical results is not fundamentally a constraint of computational efficiency; rather, it exposes a distinct hierarchy of what information survives when translating abstract physical world-model features into a text vector space. At the apex of this hierarchy sits scene-level causal and temporal tracking: what event is occurring, why it is occurring, and how chronological moments connect across extended timelines. Our framework preserves this macro-physics pattern with reasonable stability, correctly deriving that building shadows preserve snow patches via localized microclimates, re-identifying an art installation across a substantial viewpoint shift, and capturing the general topic and context of a scene.

Conversely, literal, low-level alphanumeric detail — price magnitudes, garment colors, exact naming strings — is where generation shows systematic drift. Exact-match string accuracy rests at a low baseline, and a meaningful share of errors are wrong in a specific, informative way: correct category, incorrect instance. A pharmacy sign is decoded as a plausible alternate pharmacy name; a blue hoodie is described as light green; a price tag's magnitude is localized but its exact value is not.

We tie this divergence to V-JEPA's training objective: unlike CLIP-style encoders optimized to align visual content directly with language, V-JEPA is trained to predict how a scene evolves in latent space, with no direct incentive to preserve the fine-grained visual precision that OCR-like or exact-color tasks require. Compressing this representation through a small trainable bridge into a frozen language model plausibly sharpens this tendency further, since the tokens that survive translation are those carrying coarser, dynamics-relevant signal, while literal specifics are exactly what a small bridge has the least capacity to preserve. We present this as a hypothesis consistent with the architecture and our observations, not as something this paper tests directly against an alternative encoder.

A second, and we believe more important, pattern concerns library-size scaling. Expanding the memory library from 110 to 635 videos leaves ROUGE-L F1 within a comparatively narrow 0.17–0.25 band with no consistent monotonic trend, while teacher-forced loss is at its highest at the largest library size tested. Read in isolation, this could support two very different stories: either library size genuinely has little effect on retrieval quality within the range tested, or the apparent stability (and the loss increase at 635) reflects something else entirely. Section 4.3 supplies the relevant evidence (Figure 1): checkpoints in our training log were selected on training-loss improvement, not held-out loss, and across logged epochs, training loss fell from 1.89 to 1.26 while held-out loss rose from 1.95 to as high as 2.98 — a standard overfitting signature. This is consistent with a bridge already operating in an overfitting regime by the time these evaluations ran, where checkpoint selection on training loss, rather than held-out loss, does not reliably track held-out performance. Under this reading, the library-size results should not be interpreted as evidence that scale is irrelevant to this architecture; they are at least partly a symptom of a checkpoint-selection procedure that was not optimizing for the thing we actually want. Future work should select checkpoints on held-out loss directly, which would allow the scaling question to be asked cleanly, independent of this confound.

7 · Limitations

Our verified evaluation set (N = 513) is a subset of the 700-video EgoLongQA target; the remaining instances could not be verified against source logs at time of submission, and we exclude rather than estimate them — which we flag as a methodological gap rather than a resolved filtering procedure.

Our library-size breakdown draws on differently-sized samples per library-size checkpoint (ranging from N = 11 to N = 102), which limits the precision of any single row and, combined with the checkpoint-selection confound discussed in Section 6, means that breakdown should be read as suggestive rather than conclusive. We use a single frozen language model and a single V-JEPA checkpoint, and do not know whether our findings hold for other model sizes, other pretraining objectives, or pooling strategies other than fixed-size adaptive average pooling.

Our automatic metrics (word overlap, ROUGE-L, teacher-forced loss) are weak proxies for answer quality on their own; we do not report human or LLM-judge evaluation in this version of the paper, and the dataset's own official ConvQA protocol uses an LLM judge specifically because automatic overlap metrics are known to correlate imperfectly with human judgment. Finally, our reported metrics are not the dataset's official EgoLongQA leaderboard metric (four-way multiple-choice accuracy); we train and evaluate on free-text generation because it is a stronger test of representational adequacy, but mapping these findings onto the official multiple-choice protocol remains an open next step.

8 · Conclusion

This paper investigates whether V-JEPA, compressed to a 16-token summary via adaptive average pooling and bridged to a frozen Qwen2-VL-2B-Instruct decoder through a small 2-layer MLP (~3.94M trainable parameters, ~0.17% of the ~2.3B-parameter system), can serve as a persistent, database-free memory for egocentric video question answering.

Across 513 verified EgoLongQA instances, the system shows low exact-match accuracy (1.2%) alongside a consistent qualitative pattern: causal and physical understanding is preserved, precise alphanumeric detail is not. This held roughly steady from 110 to 635 videos in memory — a stability we show is consistent with the bridge already overfitting on training loss by the time these evaluations were logged. This is an important caveat, and the clearest concrete next step, alongside a systematic no-JEPA baseline, a direct comparison against a pooled CLIP-encoder bridge, and held-out-loss-based checkpoint selection to disentangle genuine scaling behavior from this confound.

Status

This work was submitted to the ECCV 2026 Wearable AI Workshop and it is published here as a preprint. The authors welcome feedback and discussion on the findings and methodology.

Cite

@misc{srivastava2026worldmodels,
  title  = {World Models as a Persistent Latent Memory Store for
            Long-Context Wearable AI},
  author = {Srivastava, Arjun and Karakoti, Gaurav},
  year   = {2026},
  note   = {Preprint. Submitted to the ECCV 2026
            Wearable AI Workshop.},
  url    = {https://deuxvoir.example/research/world-model-latents-as-memory}
}