Motion-to-Attention: Enhancing Attention Maps to Improve Performance of Text-Guided Video Editing Models

Seong-Hun Jeong^*, Inhwan Jin^*, Haesoo Choo^*, Hyeonjun Na^*, Kyeongbo Kong^†

Pusan National University, Pukyong National University
^*Indicates Equal Contribution
^†Corresponding Author

Paper Code

Red words indicate the prompt that Prompt-to-prompt editing is applied ('replace' or 'refine'),
Blue words indicate the prompt that Attention Flow with Motion Map Injection module is applied.

Abstract

Recent text-guided video editing research attempts to expand from image to video based on the text-guided image editing model. To this end, most research focuses on achieving temporal consistency between frames as a primary challenge in text-guided video editing. However, despite their efforts, the editability is still limited when there is a prompt indicating motion, such as ``floating". In our experiment, we found that this phenomenon was due to the inaccurate attention map of the motion prompt. In this paper, we suggest the Motion-to-Attention (M2A) module to perform precise video editing by explicitly taking motion into account. First, we convert the optical flow extracted from the video into a motion map. During conversion, users can selectively apply direction information to extract the motion map. The proposed M2A module uses two methods: ``Attention-Motion Swap", which directly replaces the motion map with the attention map, and ``Attention-Motion Fusion", which uses the association between the motion map and the attention map, measured by a Fusion metric, as a weight to enhance the attention map using the motion map. The Text-to-Video editing model with the proposed M2A module showed better quantitative and qualitative results compared to the existing model.

Our Contribution

Our proposed Motion-to-Attention (M2A) module gives a way to effectively inject motion extracted from video into the attention map of the prompt. We found that editing a video with the motion extracted from the video improves general editing performance and enables selective editing according to the direction in which the object moves.

This figure visualizes the results of the T2V model for the input video and its corresponding attention map, confirming the inaccurate estimation of the motion prompt (e.g., floating, moving). The existing T2V model failed to accurately estimate the attention map for the motion prompt, resulting in restricted editability. The proposed Motion-to-Attention (M2A) module improves the attention map of the entire prompt, demonstrating enhanced editability for existing video editing models.

Framework

This figure illustrates the overall processing flow of the proposed method, including the Motion-to-Attention (M2A) module. First, the Text-to-Video (T2V) Model generates an attention map by receiving video and prompts as input. Simultaneously, the optical flow estimation model estimates the optical flow from the input video frames. The estimated optical flow is converted to a motion map by default using only magnitude information. Optionally, when direction information is provided by the user, the Direction Control converts the optical flow to a motion map that only shows movement in the user-specified direction. Then, the motion map is injected into the attention map of the T2V-Model in two ways from the M2A module: Attention-Motion Fusion and Attention Motion Swap. After that, text-to-video editing is performed using the attention map enhanced by the motion map. The central part of the figure illustrates how the Attention-Motion Swap and Attention-Motion Fusion of the M2A module enhance the attention map using the motion map.

Qualitative Results

First, we provide results for 8 frames and 24 frames. It is possible to use not only 8 frames, which are commonly used by most video editing models, but also up to 24 frames.

8-Frame Results

24-Frame Results

We coducted extensive exprements on existing T2V models (vid2vid-zero, FateZero) including Video-P2P. Followings are those expermental results.

Frame-wise Results

Furthermore, we visualize, on a frame-by-frame basis, the motion maps utilized by the M2A module, as well as the editing results before and after applying M2A, across multiple examples.

Direction Control

Since optical flow has the information on direction of pixel movement, as well as magnitude, our model can be applied to allow the user to edit contents in a specific direction by rotating the optical flow according to the direction provided by the user before injecting it.

Qualitative Results on
Multi-Object, Multi-Motion Video Editing

We evaluate video editing performance on complex videos containing multiple objects and diverse motion patterns. Existing text-guided video editing methods often fail to accurately localize and preserve object motions under such challenging conditions, whereas our method consistently produces coherent and accurate editing results.

Qualitative Results
on Videos with Camera Motion

Furthermore, we conduct additional experiments on videos that include camera motion in addition to multiple objects and multiple motion patterns. Despite the increased complexity introduced by camera motion, our method continues to deliver high-quality and stable editing results.

Video Captioning–Based Prompts

To reflect realistic scenarios where no user-provided prompt is available for the input video, we evaluate our method using automatically generated prompts obtained from a video captioning model. When applying M2A, we observe that the editing results produced using user-written prompts and those generated by the video captioning model are both accurate and reliable, demonstrating the robustness of our approach.

* Zhang, Boqiang, et al. "Videollama 3: Frontier multimodal foundation models for image and video understanding." arXiv preprint arXiv:2501.13106 (2025).

Ablation Study

In ablation study, correlation-based normalized methods (Normalized Cross Correlation, Normalized Cross Coefficient) outperform other metrics including Spectral angle Mapper. In some case, both the Structural Similarity Index and Mutual Information shows novel results that were not observed within other template matching methods. Through this results, it is important to choose the template matching metrics well.

Limitation

The proposed M2A module enhances attention distributions using optical-flow-based motion cues. As a result, its performance may degrade in challenging scenarios where reliable motion estimation is difficult. Such cases include videos with severe motion blur, low-texture regions, heavy occlusions, or complex crowd scenes. In these situations, inaccurate or ambiguous optical flow leads to noisy motion maps, which in turn limits the effectiveness of motion-guided attention enhancement and results in suboptimal editing quality.

Followup Works

We used Video-P2P: Video Editing with Cross-attention Control model as base-model which is based on Text-to-Image diffusion. And we expanded experments to show that our module is generally suitable for other T2V models, vid2vid and FateZero.

BibTex

@article{2024Motiontoattention,
title={Motion-to-Attention: Enhancing Attention Maps to Improve Performance of Text-Guided Video Editing Models},
year={2024}
}