VideoMemBench

How Well Do MLLMs Remember
in Video Conversations?

Explore the benchmark

Zhengdong Hu^1,*Chao Wang^1,*Kaituo Feng²Huanjin Yao³Mingfei Lu¹Hehe Fan⁴Yi Yang^4,†

¹ University of Technology Sydney² The Chinese University of Hong Kong³ The Hong Kong University of Science and Technology⁴ Zhejiang University

* Equal contribution · † Corresponding author

Paper

Code

Dataset Soon Leaderboard

01 / THE BENCHMARK

Beyond answering one question at a time.

A useful video conversation is connected: later questions revisit what happened earlier, or build on an answer already given. VideoMemBench measures this ability to retrieve and reuse information across turns—in-context video memory.

01 / EPISODIC DEPENDENCY

Remember what you have seen.

A later question can return to a scene, object, or event discussed earlier in the conversation.

The model must reconnect that reference with the right visual episode and retrieve the relevant details.

EpiDepAccAccuracy on turns that revisit an earlier visual episode.

Episodic dependency: an earlier video scene is revisited by a later conversation turn. — An earlier visual episode anchors a later question.

02 / ANSWER DEPENDENCY

Remember what you have said.

A later question can build on an answer the model has already given.

That earlier answer becomes part of the reasoning. A correct response earns credit only when its full answer-dependency chain is also correct.

AnsDepAccCredit for a correct response and its answer ancestors.

Answer dependency: earlier model answers carry forward through a chain of later conversation turns. — Earlier answers carry forward through the conversation.

A video conversation, turn by turn.

Follow a conversation across sports and product launches. Each question can return to an earlier scene or build on an earlier answer.

Loading video…

UCF vs. Arizona00:00

The video plays once. Questions follow their evidence timestamps. Select a turn to pause and revisit, or pause to read.

Questions & answers

Loading questions…

02 / MAIN RESULTS

A substantial memory gap remains.

25 models evaluated with 128 frames and their own conversation history.

Main results from Table 2. Scores are percentages; higher is better.
Rank	Model	All turns	Episodic dependency	Answer dependency	Mean score
1	Claude Opus 5Closed-source	52.3	49.1	35.7	45.7
2	Gemini 3.8 FlashClosed-source	50.4	46.6	30.3	42.4
3	Kimi K3Open-weight	50.5	49.0	27.1	42.2
4	GPT-5.6-solClosed-source	50.0	47.8	27.1	41.7
5	Gemini 3.5 FlashClosed-source	49.6	45.5	29.7	41.6
6	GPT-5.5Closed-source	48.0	46.5	27.0	40.5
7	Claude Sonnet 5Closed-source	47.8	45.3	27.3	40.2
8	Gemini 3.1 Pro PreviewClosed-source	45.7	44.8	27.0	39.1
9	GPT-5.4Closed-source	44.4	42.6	21.9	36.3
10	Qwen3-VL-235B-A22BOpen-weight	43.1	41.6	18.6	34.4
11	Qwen3.5-27BOpen-weight	42.2	39.8	20.9	34.3
12	Qwen3.5-397B-A17BOpen-weight	42.3	40.1	20.3	34.2
13	Qwen3.5-122B-A10BOpen-weight	42.4	39.5	19.9	33.9
14	Qwen3.5-35B-A3BOpen-weight	38.2	36.3	18.0	30.8
15	Qwen3-VL-32BOpen-weight	39.3	35.9	17.2	30.8
16	Qwen3-VL-30B-A3BOpen-weight	36.4	35.1	15.8	29.1
17	Qwen3.5-9BOpen-weight	34.6	33.4	12.9	27.0
18	Qwen3-VL-8BOpen-weight	34.8	32.1	12.3	26.4
19	Qwen3.5-4BOpen-weight	33.8	32.6	12.5	26.3
20	Qwen3-VL-4BOpen-weight	32.4	29.4	12.9	24.9
21	Qwen2.5-VL-32BOpen-weight	31.7	32.6	9.6	24.6
22	Qwen2.5-VL-7BOpen-weight	26.2	24.9	8.4	19.8
23	Keye-VL-1.5-8BOpen-weight	25.4	24.5	8.2	19.4
24	Qwen2.5-VL-3BOpen-weight	23.5	24.4	5.7	17.9
25	Qwen3.5-0.8BOpen-weight	21.6	21.3	5.7	16.2

25 modelsWe will continue evaluating new models and updating this leaderboard.

Main results from Table 2 of the manuscript. All model scores use 128 fixed frames and sequential self-rollout. Scores are percentages.

Evaluation with fewer frames

13 open-weight models evaluated with 64 or 32 fixed frames per video.

64 frames

Table 4: 64 fixed frames and sequential self-rollout. Scores are percentages; higher is better. Rank is by Overall score.
Rank	Model	TurnAccAll turns	EpiDepAccEpisodic dependency	AnsDepAccAnswer dependency	OverallMean score
1	Qwen3-VL-235B-A22B	40.3	39.3	15.6	31.7
2	Qwen3.5-122B-A10B	37.8	36.5	15.2	29.8
3	Qwen3.5-27B	37.9	36.7	14.8	29.8
4	Qwen3-VL-32B	37.8	36.1	14.5	29.5
5	Qwen3.5-35B-A3B	37.0	36.7	13.5	29.1
6	Qwen3-VL-30B-A3B	33.6	32.9	11.5	26.0
7	Qwen3-VL-8B	32.7	32.0	11.9	25.5
8	Qwen3.5-9B	33.4	33.2	9.6	25.4
9	Qwen3-VL-4B	31.3	30.3	12.1	24.5
10	Keye-VL-1.5-8B	24.5	24.9	6.3	18.5
11	Qwen2.5-VL-7B	24.1	22.7	6.4	17.7
12	Qwen2.5-VL-3B	22.5	22.5	5.1	16.7
13	Qwen3.5-0.8B	21.0	21.1	4.7	15.6

32 frames

Table 4: 32 fixed frames and sequential self-rollout. Scores are percentages; higher is better. Rank is by Overall score.
Rank	Model	TurnAccAll turns	EpiDepAccEpisodic dependency	AnsDepAccAnswer dependency	OverallMean score
1	Qwen3-VL-235B-A22B	36.3	37.8	14.5	29.5
2	Qwen3.5-122B-A10B	35.0	35.2	10.7	27.0
3	Qwen3.5-35B-A3B	34.5	34.1	12.3	27.0
4	Qwen3.5-27B	34.9	34.2	11.7	26.9
5	Qwen3-VL-32B	34.1	32.6	13.1	26.6
6	Qwen3-VL-30B-A3B	32.1	32.5	8.0	24.2
7	Qwen3.5-9B	29.8	30.4	8.6	22.9
8	Qwen3-VL-8B	29.4	29.4	9.2	22.7
9	Qwen3-VL-4B	29.0	29.0	8.8	22.3
10	Qwen2.5-VL-7B	23.2	23.7	4.9	17.3
11	Keye-VL-1.5-8B	23.3	23.1	5.1	17.2
12	Qwen2.5-VL-3B	22.6	23.4	4.7	16.9
13	Qwen3.5-0.8B	20.1	21.8	4.7	15.5

Table 4 · Fixed frames and sequential self-rollout. Scores are percentages.

03 / EVALUATION

The same video. The model’s own history.

Visual input stays fixed; the conversation unfolds one turn at a time.

Evaluation protocolFixed input, sequential conversation, answer grading

Evaluation pipeline: fixed visual input, sequential self-rollout, answer grading, dependency-aware metrics

Fix the visual input
Uniformly sample 128 timestamped frames once per video. Reuse the same frames for every model and every turn.
Roll out the dialogue
Ask questions in order, retaining the model’s own full historical responses. Ground-truth answers are not inserted into history.
Grade each response
Extract option letters for multiple-choice questions. Grade open-ended answers with GPT-5.4 mini under the paper’s binary semantic criterion.

TurnAccTurn accuracy

Evaluation subset: All conversation turns.
Credit rule: A turn receives credit when the current response is correct.

EpiDepAccEpisodic dependency accuracy

Evaluation subset: Turns with at least one Episodic parent.
Credit rule: The current response must be correct. Credit is independent of whether the parent responses were correct.

AnsDepAccAnswer dependency accuracy

Evaluation subset: Turns with at least one Answer parent.
Credit rule: The current response and every recursive ancestor reached through Answer edges only must all be correct. Episodic edges are not followed when tracing these ancestors.

OverallEqual-weight mean of the three metrics

(TurnAcc + EpiDepAcc + AnsDepAcc) / 3

The three metrics contribute equally to the overall score. All scores are reported as percentages; higher is better.

Run the evaluation

Frame preparation, self-rollout, answer grading, and dependency-aware scoring.

Evaluation guide ↗

04 / CITATION

Cite VideoMemBench

BibTeX

@misc{videomembench,
      title={VideoMemBench: How Well Do MLLMs Remember in Video Conversations?}, 
      author={Zhengdong Hu and Chao Wang and Kaituo Feng and Huanjin Yao and Mingfei Lu and Hehe Fan and Yi Yang},
      year={2026},
      eprint={2610.40131},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2610.40131}
}