espnet2.aqa.metric_tokenizer.metric_tokenizer.MetricTokenizer

source

class espnet2.aqa.metric_tokenizer.metric_tokenizer.MetricTokenizer(tokenizer_config: str | Dict[str, Any], tokenize_metric: List[str] | None)

Bases: AbsMetricTokenizer

Initialize the MetricTokenizer with a configuration file path.

• Parameters:
  • tokenizer_config – The tokenizer configuration JSON file/Dictionary
  • tokenize_metric – Selected known metric names, or None for all metrics

add_offset(src_tokens: Iterable[int], metric_name: str) → List[int]

Add back the offset related to metrics.

• Parameters:
  • src_tokens – source tokens to be added
  • metric_name – the name of the metric
• Returns: Token with added offset

get_metric_meta_label(metric_name: str) → int

Get the meta label index for a given metric.

• Parameters:metric_name – Name of the metric
• Returns: Index of the meta label in the vocabulary

get_token_index(metric_name: str, value_index: int, reduce_offset: bool = False) → Tuple[int, int]

Get the token indices for a metric and its value.

• Parameters:
  • metric_name – Name of the metric
  • value_index – Index of the value for this metric
  • reduce_offset – Whether to reduce metric-related offset
• Returns: Tuple of (meta_label_index, value_index) in the vocabulary

get_token_list() → List[str]

Get the list of tokens in the vocabulary.

• Returns: List of tokens

metric2token(metrics: Dict[str, float | str | int | Tuple[int, int]], reduce_offset: bool = False) → Dict[str, Tuple[int, int]]

Convert metrics dictionary to token indices.

• Parameters:
  • metrics – Dictionary of metric names and their values. Unknown names raise ValueError; known metrics outside tokenize_metric are skipped.
  • reduce_offset – Whether to reduce metric-related offset
• Returns: Dictionary of token indices for each metric (meta_label, value)

token2metric(token: int, metric: str | None = None) → float | str

Convert a single token index back to its metric representation.

• Parameters:
  • token – Token index
  • metric – Optional metric name for non-meta_label tokens
• Returns: String representation of the metric

tokenseq2metric(tokens: Iterable[int], return_dict: bool = False) → str | Dict[str, List[float | int | str | Tuple[float, float]]]

Convert token indices back to a metric representation.

• Parameters:
  • tokens – Iterable of token indices
  • return_dict – If True, return a dictionary instead of a string
• Returns: String representation of the metrics