Peak-End-Net Brings the Peak-End Rule to Video Aesthetic Assessment

Introduction

Video aesthetic assessment asks a deceptively difficult question: how visually pleasing is a video? Unlike image aesthetics, video aesthetics involves not only composition, color, and visual quality in individual frames, but also temporal flow, pacing, memorable moments, and the impression left by the ending.

The arXiv paper “Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment,” accepted to ACM MM 2026, proposes a framework called Peak-End-Net. Its key idea is to borrow from psychology: according to the peak-end rule, people often judge an experience largely by its most intense or salient moments and by how it ends. The authors apply this intuition to video aesthetic assessment, arguing that a model should not simply average visual impressions across time.

Key points

• A psychology-grounded approach: Peak-End-Net treats the most aesthetically salient moments and the ending of a video as central cues for predicting overall aesthetic appeal.
• Transferring image aesthetic knowledge: The framework introduces a pretrained image aesthetic assessment head to generate frame-wise aesthetic priors. These priors act as surrogate signals for locating visually important moments in the video.
• Peak-end temporal aggregation: Rather than relying only on uniform pooling, the model uses the frame-level priors to guide aggregation in a way aligned with the peak-end rule.
• Modeling aesthetic rhythm: Videos are temporal experiences, so the authors add an aesthetic rhythm encoder to capture how visual appeal evolves over time beyond isolated frames.
• Dynamic gated fusion: A gated fusion mechanism combines different sources of evidence and is designed to improve robustness when the test distribution differs from the training distribution.
• Parameter-efficient design: Peak-End-Net is built on a frozen vision transformer backbone and trains only a small number of parameters, making the approach more scalable.

Why it matters

The contribution is notable because it reframes video aesthetic assessment as a human-perception problem rather than only a visual feature extraction task. Many video scoring methods reduce a video to frame-level features and then aggregate them statistically. Peak-End-Net instead suggests that subjective judgment is uneven: a striking shot or a strong ending can disproportionately shape the final impression.

This has potential relevance for short-video recommendation, creator tools, automatic editing, and evaluation of generated videos. A video with an ordinary beginning but a powerful final sequence may be undervalued by simple averaging methods; a peak-end-inspired model could better capture that human-like response.

The paper also highlights the continuing challenges in this field. Aesthetic judgment remains subjective, and benchmarks for video aesthetic assessment are still limited compared with other visual tasks. The broader message is that future evaluation models may need to combine visual representation learning with temporal structure and cognitive principles, instead of relying on scale alone.

Source: arXiv