Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition
By: Martin Kocour , Martin Karafiat , Alexander Polok and more
Potential Business Impact:
Lets computers write down who said what.
We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to target-speaker ASR systems such as DiCoW, which decode each speaker separately, our approach performs joint decoding, allowing the decoder to condition on the context of all speakers simultaneously. Experiments show that the model outperforms existing SOT-based approaches and surpasses DiCoW on multi-talker mixtures (e.g., LibriMix).
Similar Papers
Scaling Multi-Talker ASR with Speaker-Agnostic Activity Streams
Audio and Speech Processing
Makes talking robots understand many people at once.
SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition
Sound
Helps computers understand who is talking in noisy rooms.
Joint ASR and Speaker Role Tagging with Serialized Output Training
Audio and Speech Processing
Lets computers know who is talking in a conversation.