espnet3.systems.esp2_asr.metrics.ter.TER

source

class espnet3.systems.esp2_asr.metrics.ter.TER(bpemodel: str | Path, clean_types: Iterable[str] | None = None)

Bases: BaseMetric

Compute TER (token error rate) for a dataset.

TER is the error rate over the model’s subword (BPE) tokens: the reference and hypothesis text are tokenized with a SentencePiece model, then scored like WER over the resulting token sequences. This mirrors espnet2’s Stage 13 scoring, which computes ter at the bpe token level.

Reads data["ref"]/data["hyp"], the declared names; the metrics config’s inputs: binds them to a source (see check_metric_inputs).

############# Examples

>>> [f.name for f in TER.inputs]
['ref', 'hyp']

Initialize the TER metric.

• Parameters:
  • bpemodel – Path to the SentencePiece model used to tokenize text into subword tokens (typically the recipe’s trained bpe.model).
  • clean_types – Optional cleaner types passed to TextCleaner.

compute() → Dict[str, float]

Return the TER of everything added so far, from the summed counts.

• Returns:{"TER": <percentage>}, rounded to 2 places.
• Return type: Dict[str, float]
• Raises:ValueError – If the references have no tokens.

############# Examples

>>> metric = TER(bpemodel="bpe.model")
>>> metric.update(ref=["a b c"], hyp=["a x c"])
>>> metric.compute()
{'TER': 33.33}

inputs : ClassVar[Tuple[Field, ...]] = (Field(name='ref', kind='text', label='Ref', optional=False, channels=None), Field(name='hyp', kind='text', label='Hyp', optional=False, channels=None))

one Field per SCP input. The name is the key of data (and the default SCP file name); the kind says what the SCP values hold (text: the value itself, audio: a path). A class attribute for a fixed contract; set on self instead, before calling super().__init__(), for one built from the instance’s own configuration.

• Type: What this metric reads

outputs : ClassVar[Tuple[Field, ...]] = (Field(name='TER', kind='number', label='Ter', optional=False, channels=None),)

one Field per key of the result dict, all of kind number. Same class-attribute-or-self choice as inputs.

• Type: What this metric returns

reset() → None

Reset the counts and drop alignment text from an unfinished run.

############# Examples

>>> metric = TER(bpemodel="bpe.model")
>>> metric.reset()

update(, ref: Sequence[str], hyp: Sequence[str]) → None

Add the token error counts of one batch to the state.

• Parameters:
  • ref – Reference texts, one per utterance.
  • hyp – Hypothesis texts, aligned with ref.
• Raises:RuntimeError – If jiwer is not installed.

############# Examples

>>> metric = TER(bpemodel="bpe.model")
>>> metric.update(ref=["a b c"], hyp=["a x c"])

write_artifacts(test_dir: Path) → None

Write the alignments kept by this __call__ to test_dir/ter_alignment.

The file holds only the rows seen by the __call__ that wrote it, then one summary line from the summed counts; the kept text is cleared afterwards.

• Parameters:test_dir – The test set’s output directory, already created.

############# Examples

>>> metric.write_artifacts(Path("infer/test-clean"))