Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1511,6 +1511,19 @@ audiocpp_add_model(hviske_asr
engine::models::hviske_asr::make_hviske_asr_loader
)

audiocpp_add_model(hviske_asr_v6
SOURCES
src/models/hviske_asr_v6/assets.cpp
src/models/hviske_asr_v6/frontend.cpp
src/models/hviske_asr_v6/encoder.cpp
src/models/hviske_asr_v6/decoder.cpp
src/models/hviske_asr_v6/session.cpp
INCLUDES
engine/models/hviske_asr_v6/session.h
LOADERS
engine::models::hviske_asr_v6::make_hviske_asr_v6_loader
)

audiocpp_add_model(irodori_tts
SOURCES
src/models/irodori_tts/assets.cpp
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ Model weights keep the license of their original release, which is separate from
| **fun_asr_nano** | ASR | auto, zh, en, ja | Fun-ASR-Nano-2512 | GGUF 16/Q8 |
| **higgs_audio_stt** | ASR | en | Higgs Audio v3 STT | GGUF 16/Q8, Stream |
| **hviske_asr** | ASR | da | Hviske v5.3 | GGUF Q8 |
| **[hviske_asr_v6](docs/models/hviske_asr_v6.md)** | ASR | da | Hviske v6 | GGUF BF16 / Q8 |
| **index_echo** | ASR, Translate | zh → en, es, ja | Index-Echo-S2TT 2B<br>Index-Echo-S2TT 9B | GGUF original/Q8_0/Q4_K |
| **marblenet_vad** | VAD | lang agnostic | MarbleNet VAD | Bundled |
| **pulsevad** | VAD | lang agnostic | PulseVAD 2.1K Student / 81K Teacher | GGUF F32 |
Expand Down
1 change: 1 addition & 0 deletions docs/model_licenses.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ Licenses change. If a row is wrong or out of date, please open a PR.
| `htdemucs` | [facebookresearch/demucs](https://github.com/facebookresearch/demucs) | MIT | Yes | | 2026-09-21 |
| `htdemucs_6stems` | [facebookresearch/demucs `5c90dfd2`](https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/5c90dfd2-34c22ccb.th) | MIT | Yes | | 2026-09-26 |
| `hviske_asr` | [syvai/hviske-v5.3](https://huggingface.co/syvai/hviske-v5.3) | CC-BY-NC-4.0 | No | | 2026-09-21 |
| `hviske_asr_v6` | [syvai/hviske-v6](https://huggingface.co/syvai/hviske-v6) | CC-BY-NC-4.0 | No | Retain upstream NOTICE and third-party license notices. | 2026-10-07 |
| `index_echo` | [IndexTeam/Index-Echo-S2TT-2B](https://huggingface.co/IndexTeam/Index-Echo-S2TT-2B)<br>[IndexTeam/Index-Echo-S2TT-9B](https://huggingface.co/IndexTeam/Index-Echo-S2TT-9B) | Apache-2.0 | Yes | S2TT checkpoints only. | 2026-10-01 |
| `index_tts2` | [IndexTeam/IndexTTS-2](https://huggingface.co/IndexTeam/IndexTTS-2)<br>[IndexTeam/IndexTTS-2.5](https://huggingface.co/IndexTeam/IndexTTS-2.5) | [bilibili Model Use License](https://huggingface.co/IndexTeam/IndexTTS-2.5/blob/main/LICENSE) | Conditional | A separate license is needed above RMB 1 billion yearly revenue or 100 million monthly users. The model and its outputs may not be used to improve other commercial AI models. | 2026-09-21 |
| `inflect_v2` | [owensong/Inflect-Micro-v2](https://huggingface.co/owensong/Inflect-Micro-v2)<br>[owensong/Inflect-Nano-v2](https://huggingface.co/owensong/Inflect-Nano-v2) | Apache-2.0 | Yes | | 2026-09-21 |
Expand Down
63 changes: 63 additions & 0 deletions docs/models/hviske_asr_v6.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Hviske v6

`hviske_asr_v6` is a separate family from the Conformer-based `hviske_asr`.
It uses a Whisper encoder with partial rotary positions, four-frame stacking,
an RMSNorm/linear projector, and a Qwen3 causal decoder.

Upstream: [syvai/hviske-v6](https://huggingface.co/syvai/hviske-v6).
The checkpoint is Danish-only and licensed under CC BY-NC 4.0; see the
upstream model card and notices for the underlying models' licenses.

## Usage

```bash
audiocpp_cli --task asr --family hviske_asr_v6 \
--model /path/to/hviske-v6-bf16.gguf --backend cuda \
--audio danish.wav --log
```

The model supports offline transcription, without timestamps, diarization, or
native streaming. Input is mixed to mono and resampled to 16 kHz. Recordings
longer than 30 seconds are split at silence boundaries into chunks of at most
28 seconds, following the upstream transcription wrapper.

## Request Options

| Option | Default | Meaning |
|---|---|---|
| `audio_chunk_mode` | `auto` | `auto`: silence splitting above 30 seconds; `silence`: silence splitting above the configured duration; `fixed`: fixed-duration cuts; `none`: no splitting, input must be at most 30 seconds. |
| `audio_chunk_duration_sec` | `28` | Maximum chunk duration (0.001-30 seconds). Does not change the 30-second trigger in `auto` mode. |
| `max_tokens` | `440` | Per-chunk generation limit, further capped to eight tokens per second plus twelve. |
| `num_beams` | `2` | Beam-search width; `1` selects greedy decoding. |
| `length_penalty` | `1.0` | Beam-score length-normalization exponent. |
| `cased` | Unset | Request cased or lowercase output. |
| `punctuated` | Unset | Request or suppress punctuation. |

Set both `cased` and `punctuated` to override automatic formatting. With either
option omitted, the model selects formatting from the audio, matching upstream.
These are model conditioning tokens, not post-processing guarantees.

```bash
--request-option cased=true --request-option punctuated=true
```

## Session Options

| Option | Default | Meaning |
|---|---|---|
| `hviske_asr_v6.weight_type` | `native` | Preserve checkpoint tensor types, or explicitly override weight storage. |

## Conversion

No model-specific checkpoint rewrite is required. Use the standard GGUF tool:

```bash
audiocpp_gguf --family hviske_asr_v6 \
--model-spec model_specs/hviske_asr_v6.json \
--input weights=/path/to/hviske-v6/model.safetensors \
--root /path/to/hviske-v6 --type orig \
--output /path/to/hviske-v6-bf16.gguf
```

The original checkpoint tensors are BF16. The converter embeds the config,
tokenizer, and model spec for standalone loading.
14 changes: 14 additions & 0 deletions include/engine/framework/audio/chunking.h
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ enum class AudioChunkMode {
QuietEnergy,
Vad,
None,
Silence, // Recording-relative RMS threshold and silence-gap midpoints.
};

struct AudioChunkSpec {
Expand Down Expand Up @@ -61,6 +62,19 @@ struct QuietEnergyAudioChunkOptions {
int64_t min_energy_window_samples = 0;
};

struct SilenceAudioChunkOptions {
int64_t max_chunk_samples = 0;
// Inputs at or below this length stay whole, even if above max_chunk_samples.
int64_t trigger_samples = 0;
};

// Librosa-style silence splitting: centered 2048-sample RMS, hop 512,
// 30 dB below peak RMS. Preserve all audio and cut at silence-gap midpoints,
// falling back to max_chunk_samples when no gap is available.
std::vector<runtime::TimeSpan> plan_silence_audio_chunks(
const std::vector<float> & mono_samples,
const SilenceAudioChunkOptions & options);

std::vector<AudioChunkSpan> plan_audio_chunks(int64_t input_samples, const AudioChunkSpec & spec);

AudioChunkMode parse_audio_chunk_mode(
Expand Down
55 changes: 55 additions & 0 deletions include/engine/models/hviske_asr_v6/assets.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#pragma once

#include "engine/framework/assets/resource_bundle.h"

#include <cstdint>
#include <filesystem>
#include <memory>

namespace engine::models::hviske_asr_v6 {

struct HviskeV6WhisperConfig {
int64_t mel_bins = 128;
int64_t hidden_size = 1280;
int64_t intermediate_size = 5120;
int64_t layers = 32;
int64_t heads = 20;
int64_t rotary_dim = 56;
float rope_theta = 10000.0f;
};

struct HviskeV6Qwen3Config {
int64_t vocab_size = 151936;
int64_t hidden_size = 1024;
int64_t intermediate_size = 3072;
int64_t layers = 28;
int64_t heads = 16;
int64_t kv_heads = 8;
int64_t head_dim = 128;
int64_t max_positions = 32768;
float rms_norm_eps = 1.0e-6f;
float rope_theta = 1000000.0f;
};

struct HviskeV6Config {
HviskeV6WhisperConfig encoder;
HviskeV6Qwen3Config decoder;
int64_t frame_stack = 4;
int32_t bos_token_id = 151644;
int32_t eos_token_id = 151645;
int32_t pad_token_id = 151643;
int32_t cased_token_id = 151670;
int32_t nocase_token_id = 151669;
int32_t punctuation_token_id = 151672;
int32_t no_punctuation_token_id = 151671;
};

struct HviskeV6Assets {
assets::ResourceBundle resources;
HviskeV6Config config;
std::shared_ptr<const assets::TensorSource> weights;
};

std::shared_ptr<const HviskeV6Assets> load_hviske_v6_assets(const std::filesystem::path & path);

} // namespace engine::models::hviske_asr_v6
30 changes: 30 additions & 0 deletions include/engine/models/hviske_asr_v6/decoder.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#pragma once

#include "engine/framework/core/execution_context.h"
#include "engine/models/hviske_asr_v6/assets.h"

#include <memory>
#include <vector>

namespace engine::models::hviske_asr_v6 {

struct HviskeV6DecodingOptions {
int64_t max_tokens = 440;
int64_t num_beams = 2;
float length_penalty = 1.0f;
std::vector<int32_t> control_tokens;
};

class HviskeV6Qwen3DecoderRuntime {
public:
HviskeV6Qwen3DecoderRuntime(std::shared_ptr<const HviskeV6Assets> assets,
core::ExecutionContext & execution, assets::TensorStorageType storage);
~HviskeV6Qwen3DecoderRuntime();
std::vector<int32_t> generate(const std::vector<float> & audio, const HviskeV6DecodingOptions & options);

private:
struct Impl;
std::unique_ptr<Impl> impl_;
};

} // namespace engine::models::hviske_asr_v6
24 changes: 24 additions & 0 deletions include/engine/models/hviske_asr_v6/encoder.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#pragma once

#include "engine/framework/core/execution_context.h"
#include "engine/models/hviske_asr_v6/assets.h"
#include "engine/models/hviske_asr_v6/frontend.h"

#include <memory>
#include <vector>

namespace engine::models::hviske_asr_v6 {

class HviskeV6WhisperRoPEEncoderRuntime {
public:
HviskeV6WhisperRoPEEncoderRuntime(std::shared_ptr<const HviskeV6Assets> assets,
core::ExecutionContext & execution, assets::TensorStorageType storage);
~HviskeV6WhisperRoPEEncoderRuntime();
std::vector<float> encode(const HviskeV6Features & features);

private:
struct Impl;
std::unique_ptr<Impl> impl_;
};

} // namespace engine::models::hviske_asr_v6
26 changes: 26 additions & 0 deletions include/engine/models/hviske_asr_v6/frontend.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#pragma once

#include "engine/framework/audio/mel_spectrogram_frontend.h"

#include <cstdint>
#include <memory>
#include <vector>

namespace engine::models::hviske_asr_v6 {

struct HviskeV6Features {
std::vector<float> values;
int64_t frames = 0;
int64_t valid_frames = 0;
};

class HviskeV6WhisperFrontend {
public:
explicit HviskeV6WhisperFrontend(int64_t mel_bins);
HviskeV6Features extract(const std::vector<float> & mono_16k, size_t threads) const;

private:
std::shared_ptr<const audio::MelSpectrogramFrontend> mel_;
};

} // namespace engine::models::hviske_asr_v6
7 changes: 7 additions & 0 deletions include/engine/models/hviske_asr_v6/session.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#pragma once

#include "engine/framework/runtime/model.h"

namespace engine::models::hviske_asr_v6 {
std::shared_ptr<runtime::IVoiceModelLoader> make_hviske_asr_v6_loader();
}
Loading
Loading