espnet3.systems.f5tts.src.remove_long_short.RemoveLongShortRunner

source

class espnet3.systems.f5tts.src.remove_long_short.RemoveLongShortRunner(provider: EnvironmentProvider, batch_size: int | None = None, output_dir: str | Path | None = None, shard_subdir: str = '', resume: bool = True)

Bases: BaseRunner

Runner that checks audio durations against the bounds in parallel.

Each shard appends its status dicts to a shard-local results.jsonl file, and merge() reads every shard file back and re-sorts by idx, so callers receive results in manifest order regardless of shard completion order.

Initialize BaseRunner object.

static forward(idx: int | Iterable[int], entries: List[Tuple[str, str, str]], min_duration: float, max_duration: float, **env) → Dict[str, Any] | list

Check the duration bounds for one index or a batch of indices.

• Parameters:
  • idx – Single index or iterable of indices into entries.
  • entries – (utt_id, wav_path, line) tuples from the manifest.
  • min_duration – Minimum allowed duration in seconds (exclusive).
  • max_duration – Maximum allowed duration in seconds (exclusive).
  • **env – Additional environment entries, unused.
• Returns: A status dict for an int index, or a list of status dicts for an iterable. Each entry is {"idx": int, "utt_id": str, "keep": bool}.

Example

>>> RemoveLongShortRunner.forward(0, entries, 1.0, 20.0)
{'idx': 0, 'utt_id': 'utt_a', 'keep': True}
>>> RemoveLongShortRunner.forward([0, 1], entries, 1.0, 20.0)
[{'idx': 0, 'utt_id': 'utt_a', 'keep': True},
 {'idx': 1, 'utt_id': 'utt_b', 'keep': False}]

####### NOTE Both bounds are exclusive: an utterance lasting exactly min_duration or max_duration seconds is dropped, matching the awk filter in espnet2’s tts.sh.

merge(shard_dirs: List[Path]) → List[Dict[str, Any]]

Concatenate shard results and restore manifest (idx) order.

• Parameters:shard_dirs – Directories of every shard of the run.
• Returns: All status dicts, sorted by idx.

Example

Each shard’s results.jsonl holds one JSON object per line:

{"idx": 0, "utt_id": "103_1241_000000_000001", "keep": true}
{"idx": 1, "utt_id": "103_1241_000000_000002", "keep": false}
{"idx": 2, "utt_id": "103_1241_000001_000000", "keep": true}

>>> runner.merge([Path("data/shards/train/split.0")])
[{'idx': 0, 'utt_id': 'utt_a', 'keep': True}]

####### NOTE A shard directory without a results.jsonl is skipped rather than treated as an error.

static open_writers(shard_dir: Path | None, **env) → Dict[str, Any]

Open the shard-local JSONL results file.

• Parameters:
  • shard_dir – Directory of the shard being processed.
  • **env – Environment entries, unused.
• Returns:{"results": file}, where file is shard_dir/results.jsonl opened for writing.

Example

>>> writers = RemoveLongShortRunner.open_writers(
...     Path("data/shards/train/split.0")
... )
>>> sorted(writers)
['results']

####### NOTE The file is opened with mode "w", so a shard that is run again starts from an empty results file.

static write_record(writers: Dict[str, Any], result: Any, state: Dict[str, Any], **env) → None

Append one forward result (or batch of results) to the shard file.

• Parameters:
  • writers – The dict returned by open_writers().
  • result – A status dict, or a list of them for a batched call.
  • state – Per-shard mutable state, unused.
  • **env – Environment entries, unused.

Example

>>> RemoveLongShortRunner.write_record(
...     writers, {"idx": 0, "utt_id": "utt_a", "keep": True}, state={}
... )

which appends this line to results.jsonl:

{"idx": 0, "utt_id": "utt_a", "keep": true}