espnet2.speechlm.dataloader.multimodal_loader.dialogue_loader.OmniIODialogueLoader

source

class espnet2.speechlm.dataloader.multimodal_loader.dialogue_loader.OmniIODialogueLoader(parquet_path: str, valid_ids: list | None = None)

Bases: OmniIOTextReader

Dict-like lazy dialogue reader using omniio parquets.

Extends OmniIOTextReader to parse JSON string output into validated dialogue messages with the same sanity checks as DialogueReader.

• Parameters:
  • parquet_path – Path to the parquet file containing dialogue metadata.
  • valid_ids – List of valid IDs to keep (optional, keeps all if None).
  • worker_id – Partition IDs by worker (optional, keeps all if None).
  • world_size – Used for worker partitioning.

items() → Iterator[Tuple[str, List[Tuple[str, str, str | Tuple[ndarray, int]]]]]

Return iterator over (id, validated_dialogue) pairs.

values() → Iterator[List[Tuple[str, str, str | Tuple[ndarray, int]]]]

Return iterator over validated dialogue values.