# Hyperframes Media > Audio y medios para composiciones de HyperFrames desde un motor compartido: TTS multiproveedor, música de fondo, efectos, transcripción con Whisper, eliminación de fondo y redacción de subtítulos. Fuente: https://skillsagentes.com/skills/calesthio/openmontage/hyperframes-media Markdown: https://skillsagentes.com/skills/calesthio/openmontage/hyperframes-media.md Repositorio: https://github.com/calesthio/OpenMontage Autor: calesthio Licencia: AGPL-3.0 Actualizado: el mes pasado Coste de contexto: 149 tok instalada, 2.9k tok al activarse, 357.4k tok con todos los archivos del bundle Bundle: 40 archivos, 1.4 MB Permisos que pide: ninguno declarado ## Instalación Un skill son archivos markdown: los mismos archivos valen para cualquier agente y lo único que cambia es el directorio de destino, es decir la bandera `--agent`. Añade `-g` para instalarlo en todos los proyectos de la máquina. ```bash # Claude Code npx -y skills add calesthio/OpenMontage --skill hyperframes-media --agent claude-code # Cursor npx -y skills add calesthio/OpenMontage --skill hyperframes-media --agent cursor # Codex npx -y skills add calesthio/OpenMontage --skill hyperframes-media --agent codex # Gemini CLI npx -y skills add calesthio/OpenMontage --skill hyperframes-media --agent gemini # Windsurf npx -y skills add calesthio/OpenMontage --skill hyperframes-media --agent windsurf # Cline npx -y skills add calesthio/OpenMontage --skill hyperframes-media --agent cline ``` ## Qué hace - Produce todo el audio de las composiciones desde un único motor compartido, `scripts/audio.mjs`, que muestra el estado de sesión antes de cualquier operación - TTS multiproveedor: HeyGen, ElevenLabs o Kokoro en local - Música de fondo y efectos: por defecto recupera de la biblioteca de audio de HeyGen, con Lyria o MusicGen en local y una librería SFX incluida como alternativa sin credenciales - Transcribe con Whisper y quita fondos - Redacta subtítulos, incluido karaoke y estilo por palabra ## Cuándo usarla - Voz en off o TTS, música de fondo o efectos de sonido - Transcripción, subtítulos, letras o karaoke con estilo por palabra - Elegir voz y proveedor, o escribir prompts de estado de ánimo musical ## Qué la activa - "Genera la voz en off de esta composición" - "Ponle música de fondo a este vídeo" - "Añade subtítulos karaoke palabra por palabra" - "Quita el fondo de este clip" ## Antes de instalar - El motor comprueba el estado de sesión antes de generar audio; la librería SFX incluida y Lyria/MusicGen en local son la ruta sin credenciales. - Necesita en el PATH: node, npx - Variables de entorno: BGM_PY_DEPS, ELEVENLABS_API_KEY, GEMINI_API_KEY, GOOGLE_API_KEY, HEYGEN_API_KEY, HEYGEN_BASE, HEYGEN_CONFIG_DIR, HYPERFRAMES_API_KEY, JSON - makes network requests - needs API credentials ## Archivos - SKILL.md — 11 KB - assets/sfx/CREDITS.md — 1 KB - assets/sfx/chime.mp3 — 27 KB - assets/sfx/click-soft.mp3 — 11 KB - assets/sfx/click.mp3 — 11 KB - assets/sfx/error.mp3 — 51 KB - assets/sfx/glitch-1.mp3 — 82 KB - assets/sfx/glitch-2.mp3 — 110 KB - assets/sfx/glitch-3.mp3 — 97 KB - assets/sfx/impact-bass-1.mp3 — 66 KB - assets/sfx/impact-bass-2.mp3 — 81 KB - assets/sfx/key-press.mp3 — 4 KB - assets/sfx/manifest.json — 4 KB - assets/sfx/notification.mp3 — 77 KB - assets/sfx/ping.mp3 — 26 KB - assets/sfx/pop.mp3 — 23 KB - assets/sfx/riser.mp3 — 314 KB - assets/sfx/sparkle.mp3 — 56 KB - assets/sfx/typing.mp3 — 26 KB - assets/sfx/whoosh-cinematic.mp3 — 173 KB - assets/sfx/whoosh-short.mp3 — 18 KB - assets/sfx/whoosh.mp3 — 18 KB - references/bgm.md — 7 KB - references/captions/authoring.md — 9 KB - references/captions/motion.md — 6 KB - references/captions/transcript-handling.md — 5 KB - references/remove-background.md — 8 KB - references/requirements.md — 4 KB - references/sfx.md — 3 KB - references/transcribe.md — 3 KB - references/tts-to-captions.md — 1 KB - references/tts.md — 8 KB - scripts/audio.mjs — 12 KB - scripts/heygen-tts.mjs — 4 KB - scripts/lib/bgm.mjs — 10 KB - scripts/lib/heygen.mjs — 6 KB - scripts/lib/sfx.mjs — 5 KB - scripts/lib/tts.mjs — 10 KB - scripts/lyria-recipe.py — 4 KB - scripts/wait-bgm.mjs — 5 KB ## SKILL.md Reproducido tal cual desde calesthio/OpenMontage bajo AGPL-3.0. Esta sección es el documento original y está en inglés. # HyperFrames Media Create the audio and media assets a composition needs — voiceover (TTS), background music + sound effects, transcription, captions, background removal — then consume and animate that data in HTML. For placing assets into compositions, see `hyperframes-core`. ## The audio engine — one source for TTS · BGM · SFX Workflows do NOT hand-roll audio or vendor a copy. There is one engine — **`scripts/audio.mjs`** — that takes a neutral `audio_request.json` and writes `audio_meta.json` (plus assets under `assets/voice|bgm|sfx`): ```bash # = this skill's directory node /scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json ``` All three capabilities degrade on **ONE switch** — whether a HeyGen credential is present (resolved from `$HEYGEN_API_KEY` / `$HYPERFRAMES_API_KEY` / `~/.heygen`, **not** the CLI): | Capability | HeyGen credential present | absent | | ---------- | -------------------------------------------------- | ---------------------------------------------------- | | TTS | HeyGen Starfish REST (native word timestamps) | → ElevenLabs → Kokoro (chain `transcribe` for words) | | BGM | HeyGen music **retrieval** | Lyria → MusicGen local **generation** (detached) | | SFX | HeyGen sound-effects **retrieval** (min_score 0.4) | bundled 21-file library (`assets/sfx/`) | - **Request** (`audio_request.json`): `{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }`. `id` joins each line back to the caller's model (a frame number, a scene id, …). `bgm.mode` = `retrieve | generate | none`; omit for auto (retrieve when credentialed, else generate). An **explicit** `retrieve` is strict — it skips rather than starting a detached generate (for callers with no `wait-bgm` step). - **Output** (`audio_meta.json`, id-keyed): `{ tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }`. - `--only tts,bgm,sfx` runs a subset and **merges** into an existing `--out` (e.g. TTS+BGM early, SFX once cues exist). - BGM generate is spawned **detached** (`bgm_pending: true`) — run `scripts/wait-bgm.mjs` before assembling. - `scripts/heygen-tts.mjs` is a single-shot CLI over the same code (one text → wav + words) for when you just need HeyGen TTS without a request file. Full flag list + the `audio_meta.json` schema live in the header of `scripts/audio.mjs`. The references below cover the provider details and edge cases behind each capability. ## Preflight — show sign-in status before any audio **Always run this before generating voice or BGM — inside a full workflow _or_ a one-off "generate me a BGM/voiceover" request.** No HeyGen credential is **not** a reason to silently fall back to local engines: first recommend signing in and let the user decide. Run the shared preflight and **relay its output verbatim** — don't improvise your own "missing key" prompt, and don't offer to write keys into a per-repo `.env`: ```bash npx hyperframes auth status ``` - **Signed in** → it prints the account; proceed. - **Not signed in** (`exit 1` is expected here — "not signed in" is a normal state, not a failure) → it prints registration-first guidance. Recommend signing in: `npx hyperframes auth login` is browser OAuth — it **signs in and creates an account** (always available through this repo's CLI). To use an existing HeyGen API key (from app.heygen.com/settings/api), run `npx hyperframes auth login --api-key` — it saves to the shared `~/.heygen` (no per-repo `.env`). The output also lists the local engines voice/BGM will fall back to and a `pip` hint when deps are missing. **Relay this output as-is — don't paraphrase it into your own wording.** Then **STOP and wait** for the user to choose — sign in, or say "go" / "local" to continue offline — **before generating anything.** This is a real decision point, not a passing note: don't fold it into another question, and don't proceed past it on your own. (Exception: in autonomous / non-interactive mode, note the status and continue offline.) - `npx hyperframes auth status --json` returns `{ configured, recommended_action, offline_engines }` for deterministic branching. - **If the CLI can't run** (not on PATH and `npx` can't fetch it) → still **recommend signing in** (`npx hyperframes auth login`) and **STOP for the user's choice** — don't treat "no credential" as a silent green light for local generation. Credential resolution, full key priority, and the local-dependency list are in `references/requirements.md`. ## Provider chains (the detail behind the engine) **TTS** — first available provider wins (the engine, or `npx hyperframes tts "..."`): | Order | Provider | Detected when | Word timestamps | | ----- | ----------------------------- | -------------------------------------------- | ---------------------------------------------------------------- | | 1 | HeyGen (Starfish) | `$HEYGEN_API_KEY` / `hyperframes auth login` | **Yes, native** — pass `--words narration.words.json` to capture | | 2 | ElevenLabs | `$ELEVENLABS_API_KEY` set | No — chain `transcribe` after | | 3 | Kokoro-82M (local, 54 voices) | always (no key required) | No — chain `transcribe` after | > The published `hyperframes tts` CLI is often the local-only build (its `--help` says "Kokoro-82M", no `--provider`/`--words`) and silently falls back to Kokoro even with `$HEYGEN_API_KEY` set. That is why the engine's HeyGen path is the self-contained `scripts/heygen-tts.mjs` (REST), NOT the CLI; the CLI is used only for the Kokoro path. See `references/tts.md`. **BGM & SFX** — by default **retrieved** from the HeyGen audio library (`/v3/audio/sounds`), same credential as HeyGen TTS, with the no-credential fallback from the switch above: | Asset | HeyGen `type` | Lands in | Fallback (no credential) | | ----- | ------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------- | | BGM | `music` | `assets/bgm/track.mp3` (retrieve) · `track.wav` (generate) | Lyria / MusicGen generation | | SFX | `sound_effects` (min_score 0.4) | `assets/sfx/.mp3` | bundled 21-file library (`assets/sfx/*` + `manifest.json`) | See `references/bgm.md` and `references/sfx.md`. ## Routing | Task | Read | | ------------------------------------------------------------------- | -------------------------------------------- | | The audio engine — request/meta schema, `--only`, the switch | `scripts/audio.mjs` (header comment) | | `npx hyperframes tts` / `heygen-tts.mjs` — providers, voices, words | `references/tts.md` | | BGM — HeyGen retrieval + local Lyria / MusicGen generation | `references/bgm.md` | | SFX — HeyGen retrieval (min_score 0.4) + bundled local library | `references/sfx.md` | | `npx hyperframes transcribe` — Whisper, model rules, output shape | `references/transcribe.md` | | `npx hyperframes remove-background` — transparent cutouts | `references/remove-background.md` | | TTS → transcription → captions (no recorded voiceover) | `references/tts-to-captions.md` | | Caption authoring — style detection, layout, word grouping, exit | `references/captions/authoring.md` | | Transcript handling — input formats, quality gates, cleanup, APIs | `references/captions/transcript-handling.md` | | Caption motion — karaoke, marker effects, audio-reactive | `references/captions/motion.md` | | Model caches, system dependencies, troubleshooting | `references/requirements.md` | ## Non-negotiable rules - **One engine, no vendored copies.** Produce audio via `scripts/audio.mjs` (or `heygen-tts.mjs` for one-shot HeyGen TTS). Don't re-implement TTS/BGM/SFX inside a workflow — write an `audio_request.json` adapter and call the engine. - **"HeyGen available" = a resolvable credential, not the CLI.** The whole switch keys off `heygenCredential()`; the published `hyperframes tts` may be Kokoro-only, and there is no `hyperframes bgm` / `hyperframes sfx` command at all. - **Voice IDs are provider-specific.** `am_michael` is Kokoro-only; HeyGen UUIDs don't work on Kokoro. If you pass `--voice`, also pin `--provider` to avoid silent provider drift when the user's env changes. - **Always pass `--model` to `transcribe`.** The CLI default `small.en` silently translates non-English audio. See `references/transcribe.md` → "Language Rule". - **HeyGen returns word timestamps; ElevenLabs / Kokoro do not.** The engine chains `transcribe` automatically for the latter two; standalone, pass `--words` to HeyGen or run `transcribe` against the audio file. - **Captions consume the flat word-array format** with `{ id, text, start, end }`. See `references/transcribe.md` → "Output Shape". - **`remove-background --background-output` is hole-cut, not inpainted.** For "scene without the person", a different tool is needed. See `references/remove-background.md` → "When NOT the right tool". - **BGM/SFX default to HeyGen retrieval; the no-credential fallback is generation (BGM) or the bundled library (SFX).** `/audio/sounds` ranks by a text query — name effects concretely (`glass shatter`, not `dramatic sound`); a no-match **skips**, never blocks the render. SFX sit at volume ~0.35 under voice + BGM. See `references/sfx.md` / `references/bgm.md`. - **Treat workflow caption HTML as generated output.** For preset-backed videos, the reusable skin source lives at `.hyperframes/caption-skin.html` and the workflow script writes `compositions/captions.html`; do not edit generated `compositions/captions.html` to fix the skin. Rebuild via the workflow's `captions.mjs`, or use that workflow's explicit overrides mechanism when present. ## Dónde encaja - Categoría: [Diseño y UI](https://skillsagentes.com/categorias/diseno-ui.md) — Sistemas de diseño, trabajo con componentes y acabado visual. - Creador: [calesthio](https://skillsagentes.com/creators/calesthio.md) — 0 skills en el directorio - [Todas las skills](https://skillsagentes.com/skills.md) - [Ranking de instalaciones](https://skillsagentes.com/ranking.md) ## Otras skills del mismo repositorio - [Seedance 2 5](https://skillsagentes.com/skills/calesthio/openmontage/seedance-2-5.md): Genera vídeo cinematográfico de 4-30 s con ByteDance Seedance 2.5 por fal.ai, Volcengine Ark, Runway o ComfyUI. Cubre el contrato de prompt 2.5, cortes duros, locks de continuidad y voz. - [Comfyui](https://skillsagentes.com/skills/calesthio/openmontage/comfyui.md): Úsalo al trabajar con workflows de ComfyUI en OpenMontage: comfyui_image/video/music, workflows propios, selección de output_node, modelos que faltan, LoRAs, poca VRAM e importación de workflows de la comunidad. - [Fish Audio Tts](https://skillsagentes.com/skills/calesthio/openmontage/fish-audio-tts.md): Genera narración expresiva y multilingüe con fish.audio (modelos S1 / S2) y reutiliza voces clonadas mediante reference_id. - [Minimax H3](https://skillsagentes.com/skills/calesthio/openmontage/minimax-h3.md): Genera vídeo con MiniMax H3 (Hailuo 3.0) por la API oficial v2, fal.ai, Runway, nodos partner de ComfyUI o pesos abiertos locales. Clips de 4-15s a 2K con animación de primer/último fotograma. - [Gemini Omni](https://skillsagentes.com/skills/calesthio/openmontage/gemini-omni.md): Genera y edita conversacionalmente vídeos cortos con Google Gemini Omni Flash: itera con ediciones en lenguaje natural, clips de 3-10s a 720p con audio y texto en pantalla, e imágenes de referencia por etiquetas. --- Skills Agentes · [Índice de páginas en markdown](https://skillsagentes.com/sitemap.md) · [Inicio](https://skillsagentes.com/index.md)