Enhanced Multimodal Video Retrieval System: Integrating Query Expansion and Cross-modal Temporal Event Retrieval

Van-Thinh Vo, Minh-Khoi Nguyen, Minh-Huy Tran, Anh-Quan Nguyen-Tran, Duy-Tan Nguyen, Loi Nguyen Khanh, Anh-Minh Phan

HCMUT EE Machine Learning & IoT Lab - HCMUT
THE 14TH INTERNATIONAL SYMPOSIUM ON INFORMATION AND COMMUNICATION TECHNOLOGY - 2025

Code

Abstract

Multimedia information retrieval from videos remains a challenging problem. While recent systems have advanced multimodal search through semantic, object, and OCR queries - and can retrieve temporally consecutive scenes - they often rely on a single query modality for an entire sequence, limiting robustness in complex temporal contexts. To overcome this, we propose a cross-modal temporal event retrieval framework that enables different query modalities to describe distinct scenes within a sequence. Another key contribution is the Kernel Density Gaussian Mixture Thresholding (KDE-GMM) algorithm, which adaptively determines decision thresholds for scene transition and slide change detection, ensuring optimal keyframe selection. These extracted keyframes act as compact, high-quality visual exemplars that retain each segment's semantic essence, improving retrieval precision and efficiency. Additionally, the system incorporates a large language model (LLM) to refine and expand user queries, enhancing overall retrieval performance. The proposed system's effectiveness and robustness were demonstrated through its strong results in the Ho Chi Minh AI Challenge 2025.

Method

Data Preprocessing

Overview of the data preprocessing pipeline. Raw videos are processed to extract representative keyframes, which are then analyzed through BEiT3/CLIP feature extraction, OCR, object detection, and color detection modules to generate structured metadata.

System Overview

Overview of the retrieval system featuring two main search engines: Embedding-based Search and Metadata-based Search, integrated to handle multimodal queries and temporal event retrieval.

User Interface

User interface with three main sections: (A) Query input and search parameters for embedding-based search, (B) Metadata search interface with 7x7 grid for object placement and logical operators, and (C) Results display with pagination and video player controls.

Results

Performance on Ho Chi Minh AI Challenge 2025

Our system (team name: EEIoT_newbie) achieved competitive results across three main tasks in the final round, demonstrating the effectiveness of our multimodal retrieval framework.

Task	Description	Performance
KIS	Known-Item Search	Strong performance
QA	Question Answering	Competitive results
TRAKE	Temporal Event Retrieval	Effective retrieval

Demo Test Cases

Query augmentation impact: comparison of retrieval results without and with Gemini-based query expansion, showing improved accuracy and relevance.

BibTeX

@inproceedings{vo2025enhanced,
  title={Enhanced Multimodal Video Retrieval System: Integrating Query Expansion and Cross-modal Temporal Event Retrieval},
  author={Vo, Van-Thinh and Nguyen, Minh-Khoi and Tran, Minh-Huy and Nguyen-Tran, Anh-Quan and Nguyen, Duy-Tan and Khanh, Loi Nguyen and Phan, Anh-Minh},
  booktitle={Proceedings of the 14th International Symposium on Information and Communication Technology (SOICT)},
  year={2025},
  organization={Springer}
}