ECCV 2026

Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild

AgentHOI reformulates human-object interaction detection as a structured reasoning-and-grounding process, requiring no HOI-specific training data or parameter updates.

Ting Lei* Jialin Liu* Zhu Xu Yuxin Peng Yang Liu

Wangxuan Institute of Computer Technology, Peking University

* Equal contribution

Code Paper Citation
AgentHOI compares traditional supervised HOI detectors with a training-free agentic framework.

Unlike traditional HOI detectors that rely on task-specific annotated data and fixed vocabularies, AgentHOI coordinates perception and localization foundation modules to discover and ground interactions without HOI-specific training.

Overview

Training-free HOI detection for open-world scenes

Human-object interaction detection has traditionally been formulated as supervised detection over predefined interaction categories. This closed-set paradigm limits generalization to open-world and compositional scenarios, where interaction patterns are diverse, long-tailed, and often ambiguous.

AgentHOI is a training-free, agentic framework that transfers the generalist multimodal reasoning capability of foundation models to structured HOI detection. Instead of learning interaction classifiers, it orchestrates complementary vision foundation modules for open-ended semantic reasoning and spatial grounding.

Training-free

No HOID data, task-specific classifier, or parameter update is required.

Open-ended reasoning

MLLMs infer plausible human-action-object triplets beyond fixed vocabularies.

Precise grounding

Instance-specific descriptions help localize ambiguous human-object pairs.

Method

Agentic reasoning and localization

AgentHOI coordinates a multimodal perception module and a spatial localization module. Context-aware multi-round reasoning progressively discovers missed interactions, while multifaceted interaction localization enriches grounding prompts with semantic, spatial, and appearance cues.

AgentHOI pipeline with multi-round reasoning and interaction localization.

Analysis

Robustness in complex real-world settings

Metrics comparing reasoning strategies in single-HOI and multi-HOI scenes.
Multi-round reasoning improves recall and F1, especially in scenes containing multiple concurrent HOI triplets.
Interaction localization without multifaceted descriptions.
Generic grounding queries can confuse visually similar people and objects, leading to mismatched interaction pairs.
Interaction localization with multifaceted descriptions.
Multifaceted interaction localization adds semantic, spatial, and appearance cues to disambiguate the correct human-object instances.

Quantitative Results

Main experimental results

AgentHOI achieves strong performance without HOI-specific training, showing competitive generalization on HICO-DET and SWIG-HOI. Table values are mAP unless otherwise noted.

Performance comparison with state-of-the-art zero-shot HOI detection methods on HICO-DET under original, style-transferred, and degraded settings.
HICO-DET zero-shot comparison. AgentHOI achieves strong unseen-class performance across original, style-transferred, and degraded images.
Performance comparison with state-of-the-art models on SWIG-HOI.
SWIG-HOI comparison. AgentHOI transfers to a large-vocabulary HOI benchmark without dataset-specific HOI training.

Visualization

Qualitative results

AgentHOI produces structured interaction predictions with localized human-object pairs across diverse real-world scenes.

Qualitative AgentHOI visualization results with localized human-object interaction predictions.
Visualization of original annotations (left) and the additional interactions supplemented by AgentHOI (right). Only the newly added interactions are shown on the right to keep the visualization concise.

Citation

BibTeX

@inproceedings{lei2026agenthoi,
  title={Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild},
  author={Lei, Ting and Liu, Jialin and Xu, Zhu and Peng, Yuxin and Liu, Yang},
  booktitle={European Conference on Computer Vision},
  year={2026}
}