The paper presents a causality-aware framework for LLM-based simultaneous speech-to-speech translation (Simul-S2ST). It introduces a data pipeline that generates causally aligned training segments with improved cross-lingual speaker fidelity using zero-shot TTS. The framework includes a factorized architecture (FAST) that decouples lexical and acoustic modeling, and a causality-aware adaptive policy (CAP) that decides when to read source input and write output based on available information.
Experiments on CVSS Spanish, German, and French show that FAST-CAP consistently improves the quality-latency trade-off. It achieves up to +1.2 BLEU and a 26% relative latency reduction over a fixed policy, and up to a 38.8% relative reduction in latency compared to state-of-the-art systems, despite using substantially less training data.
The work is interesting because it addresses the key challenge of alignment in simultaneous translation—learning when to wait and when to speak—without relying on fixed heuristics. It also preserves speaker identity via high-fidelity voice transfer, which is important for natural dialogue.