espnet2.speechlm.dataloader.multimodal_loader.text_loader.OmniIOTextReader

source

class espnet2.speechlm.dataloader.multimodal_loader.text_loader.OmniIOTextReader(parquet_path: str, valid_ids: list | None = None)

Bases: object

Dict-like lazy text reader using omniio parquets.

Reads compressed text data from omniio parquet files. Text is stored in binary format with compression and accessed via byte offsets.

• Parameters:
  • parquet_path – Path to the parquet file containing text metadata.
  • valid_ids – List of valid IDs to keep (optional, keeps all if None).

items() → Iterator[Tuple[str, str]]

Return iterator over (id, text) pairs.

keys() → Iterator[str]

Return iterator over IDs.

values() → Iterator[str]

Return iterator over text values.