PROOF OF CONCEPT OF ANALYZING MULTIMODAL PRAGMATIC MARKERS USING QWEN VL LLM

Abstract

Pragmatic markers in spoken interaction are realized not only through lexical and prosodic resources but also through facial expression, gesture, and posture. The manual coding of such multimodal markers is labor-intensive and difficult to scale, which has constrained the systematic inclusion of multimodal evidence in pragmatic research. This study reports a proof of concept for a reproducible pipeline that supports the description and annotation of multimodal pragmatic markers using an open-weight Vision-Language Large Language Model (VL LLM), with politeness strategies (Brown & Levinson, 1987) deployed as a concrete instantiation of multimodal pragmatic markers. The pipeline was implemented on a Google Colab Pro environment with an NVIDIA L4 GPU and comprises four sequential modules. Audio is transcribed at the word level using Whisper Large-v3 with inter-word pauses of 0.5 seconds or greater marked inline. Video frames are extracted at three-second intervals using ffmpeg. Transcribed speech is segmented into propositions using hybrid punctuation-plus-LLM rules and then aggregated into speech acts via rolling-window LLM decisions. For each speech act, three sampled frames are submitted to Qwen3-VL-4B-Instruct with a structured prompt that elicits facial, gestural, and emotional descriptions and assigns Likert-scaled scores (0–5) on the four Brown and Levinson politeness superstrategies, accompanied by an evidence-grounded rationale. To accommodate the 22 GB usable VRAM ceiling, the speech recognition model is unloaded before the VL LLM is loaded. Application to a twelve-minute public-speaking sample yielded 160 propositions aggregated into 40 speech acts with full annotation coverage and no inference errors. Bald On-Record emerged as the dominant superstrategy in 62.5% of units, consistent with the assertive declarative register of motivational public speaking, while multi-label scoring revealed substantial strategy blending across half of the corpus. Diagnostic flagging identified 20.0% of units as warranting expert review. The contribution of the study is methodological: the pipeline demonstrates technical feasibility and structural tractability for multimodal pragmatic annotation, providing a reproducible foundation for subsequent inter-rater validation and fine-tuning work.