espnet3.systems.esp2_st.metrics.bleu.BLEU
espnet3.systems.esp2_st.metrics.bleu.BLEU
class espnet3.systems.esp2_st.metrics.bleu.BLEU(clean_types: Iterable[str] | None = None, tokenize: str = '13a', also_lowercase: bool = True, lc_remove_punctuation: bool = True, chrf: bool = True, ter: bool = True)
Bases: BaseMetric
Score translation hypotheses the way egs2/TEMPLATE/st1/st.sh does.
Mirrors espnet3.systems.esp2_asr.metrics.wer.WER in shape: it takes the declared ref/hyp texts, which the metrics config’s inputs: binds to a source, and writes a human-readable detail file beside the scores. Unlike WER’s counts, the texts themselves are the state: corpus BLEU is not the mean of per-batch BLEUs, so compute scores them all at once.
sacreBLEU is used rather than nltk’s sentence BLEU because it is what st.sh reports (scripts/utils/score_bleu.sh calls sacrebleu), and because corpus BLEU is not the mean of sentence BLEUs – scoring per sentence and averaging gives a different, lower number that is not comparable to published results.
Correspondence with st.sh (st.sh:1551-1620), which scores twice:
sacrebleu -m bleu chrf ter(st.sh:1604) ->BLEU,chrF2,TER;scripts/utils/remove_punctuation.pl(st.sh:1609-1612), thensacrebleu -lc -m bleu chrf ter(st.sh:1616) -> the same three keys suffixed_lc.
Note the asymmetry in sacreBLEU’s own CLI, which is reproduced here: -lc is --lowercase on BLEU only (dest='bleu_lowercase'). chrF keeps its separate --chrf-lowercase (off) and TER lowercases by default (--ter-case-sensitive is off), so in st.sh’s second pass chrF is case-SENSITIVE and TER is case-INsensitive in both passes. The _lc results therefore differ from the first pass mainly by punctuation removal.
Detokenization. st.sh word-tokenizes both sides and then runs Moses detokenizer.perl to undo it, because egs2’s ST data prep stores Moses tokenizer.perl output (Hello , world .). A recipe that keeps natural text on both sides – references straight from the corpus, hypotheses straight out of SentencePiece – is already at the state st.sh detokenizes to, so the round trip is a no-op and is omitted rather than reimplemented. If a recipe does feed this metric Moses-tokenized text, detokenize it in the inference output_fn first; scoring tokenized against detokenized text silently inflates BLEU.
Initialize the metric.
- Parameters:
- clean_types – Optional cleaner types passed to TextCleaner. st.sh applies its
--cleanerto the reference only and deliberately not to the hypothesis (st.sh:1563, “Don’t use cleaner for hyp”); this applies it to neither by default, which is the same thing whenever no cleaner is configured. - tokenize – sacreBLEU tokenizer.
13ais sacreBLEU’s default and the one egs2 scores with; usezh/ja-mecabfor those target languages, where whitespace is not a word boundary. - also_lowercase – Additionally report the case-insensitive pass, suffixed
_lc. st.sh reports both (case-sensitive at st.sh:1604, case-insensitive at st.sh:1616) and papers quote the case-sensitive number. Both passes come from ONE instance because espnet3 keys results by metric CLASS PATH (espnet3/systems/base/metric.py:109), so two BLEU entries in metrics.yaml would overwrite each other. - lc_remove_punctuation – Strip punctuation before the case-insensitive pass, as st.sh does with
scripts/utils/remove_punctuation.pl(st.sh:1609-1612). Lowercasing alone is NOT what egs2 calls case-insensitive BLEU, and scores a point or so lower. - chrf – Report chrF2, as
st.sh’s-m bleu chrf terdoes. - ter – Report TER (lower is better), likewise.
- clean_types – Optional cleaner types passed to TextCleaner. st.sh applies its
compute() → Dict[str, float]
Score both passes over everything added so far.
- Returns:
BLEU/chrF2/TERfor the case-sensitive pass, plus the BLEU brevity penalty and n-gram precisions, and the same keys suffixed_lcfor the case-insensitive pass. - Raises:RuntimeError – If
sacrebleuis not installed or is too old.
inputs : ClassVar[Tuple[Field, ...]] = (Field(name='ref', kind='text', label='Ref', optional=False, channels=None), Field(name='hyp', kind='text', label='Hyp', optional=False, channels=None))
one Field per SCP input. The name is the key of data (and the default SCP file name); the kind says what the SCP values hold (text: the value itself, audio: a path). A class attribute for a fixed contract; set on self instead, before calling super().__init__(), for one built from the instance’s own configuration.
- Type: What this metric reads
reset() → None
Reset the texts and drop detail lines from an unfinished run.
update(, ref: Sequence[str], hyp: Sequence[str]) → None
Keep one batch’s cleaned texts for the corpus-level score.
- Parameters:
- ref – Reference translations, one per utterance.
- hyp – Hypotheses, aligned with
ref.
Examples
>>> metric = BLEU()
>>> metric.update(ref=["hallo welt"], hyp=["hallo welt"])write_artifacts(test_dir: Path) → None
Write the scored passes to test_dir/bleu_detail.
- Parameters:test_dir – The test set’s output directory, already created.
