espnet2.sds.end_to_end.mini_omni_e2e.MiniOmniE2EModel

source

class espnet2.sds.end_to_end.mini_omni_e2e.MiniOmniE2EModel(device: str = 'cuda', dtype: str = 'float16', stream_stride: int = 4, max_tokens: int = 2048, temperature: float = 0.9, top_k: int | None = 1, top_p: float = 1.0)

Bases: AbsE2E

Mini-OMNI E2E

A class to initialize and manage the OmniInference client

for end-to-end dialogue systems.

• Parameters:
  • device (Literal [ "cuda" , "cpu" ] , optional) – The device to run the inference on. Defaults to “cuda”.
  • dtype (str , optional) – The dtype used to build the warmup input. Defaults to “float16”.
  • stream_stride (int , optional) – Number of SNAC frames decoded per streamed chunk. Defaults to 4.
  • max_tokens (int , optional) – Maximum number of tokens to generate. Defaults to 2048.
  • temperature (float , optional) – Sampling temperature. Defaults to 0.9.
  • top_k (int , optional) – Number of highest probability tokens to keep. Defaults to 1, which makes decoding greedy and therefore deterministic, regardless of temperature. Set to None to disable top-k filtering; this is the only place it can be disabled, since forward reads None as “keep the value set here”.
  • top_p (float , optional) – Nucleus sampling threshold. Defaults to 1.0, which disables it.
• Raises:ImportError – If required dependencies (Pydub, Huggingface Hub, or OmniInference) are not installed.

forward(array: ndarray, orig_sr: int, temperature: float | None = None, top_k: int | None = None, top_p: float | None = None) → Tuple[str, bytes]

Processes audio input to generate synthesized speech

and the corresponding text response.

• Parameters:
  • array (np.ndarray) – The input audio array to be processed.
  • orig_sr (int) – The sample rate of the input audio.
  • temperature (float , optional) – Overrides the temperature given to the constructor, for this call only. Defaults to None, which keeps the constructor value.
  • top_k (int , optional) – Overrides top_k for this call only. Defaults to None, which keeps the constructor value. Pass a value above 1 to draw several different responses for the same input. Note that None cannot disable top-k filtering here, because it means “keep the constructor value”; set top_k=None on the constructor instead.
  • top_p (float , optional) – Overrides top_p for this call only. Defaults to None, which keeps the constructor value.
• Returns: A tuple containing:
  • text_str (str): The generated text response.
  • audio_output (bytes): The synthesized speech as an MP3 byte stream.
• Return type: Tuple[str, bytes]

warmup()

Perform a single forward pass with dummy input to

pre-load and warm up the model.