espnet3.systems.base.metric.measure

source

espnet3.systems.base.metric.measure(metrics_config: DictConfig, inference_config: DictConfig | None = None)

Compute metrics for each test set and write a metrics JSON file.

Test sets are resolved in the following order:

1. If metrics_config.dataset.test is defined, use the configured name fields as-is.
2. Otherwise, scan metrics_config.inference_dir and treat each non-hidden subdirectory as a test set.

Example

If inference_dir contains:

exp/my_run/inference/
  test-clean/
  test-other/

then measure() scores both test-clean and test-other when metrics_config.dataset.test is omitted.

A metric’s inputs are bound by its config’s inputs: (declared name -> source; see espnet3.contract.metrics.check_metric_inputs), required for every metric that declares one. A source is either a .scp file the infer stage wrote (binding hyp to text reads <test_name>/text.scp), or a column of the test set itself, named dataset:<column> (binding ref to dataset:text reads the transcript from the data and writes it to <test_name>/dataset/text.scp on first use). The reference is the data’s, so it comes from the data, not from what inference wrote. A column that is not a scalar - a reference waveform for an audio metric - is written as an artifact beside it, as a dataset_artifacts entry for that column (type: wav, sample_rate: ...) says, .npy by default.

• Parameters:
  • metrics_config – Omegaconf configuration with inference and metric settings.
  • inference_config – The inference config, needed for dataset:<column> inputs; the system passes its own.
• Returns: Nested dict keyed by metric class path and test set name.
• Raises:ValueError – If no test sets can be resolved from either metrics_config.dataset.test or metrics_config.inference_dir.

Example

s

>>> results = measure(metrics_config, inference_config)
>>> wer = results["espnet3.systems.esp2_asr.metrics.wer.WER"]
>>> wer["test-clean"]
{'WER': 4.3}