> ## Documentation Index
> Fetch the complete documentation index at: https://developer.audioshake.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Speaker Separation

> Separate a multi-speaker recording into per-speaker tracks, with diarization and confidence scores

Multi-Speaker Separation takes one recording of several people and returns a clean, time-aligned audio track per speaker — even where speakers overlap — plus a structured results file with the diarization timeline, speaker count, and confidence scores. Background audio is preserved in its own track.

It works on the full range of real-world audio: 8 kHz phone calls to 48 kHz studio masters, in any language, with the same API call.

> [card] [Hear it on Hugging Face](https://huggingface.co/spaces/audioshake/multispeaker2)
>
> Listen to separated speakers in our interactive demo.

> [card] [Read the announcement](https://www.audioshake.ai/post/multi-speaker-2-0-keep-every-voice-in-the-conversation)
>
> What's new in Multi-Speaker Separation and the benchmarks behind it.

## Create a Task

Set `residual: true` if you also want the background (non-speech) audio as its own track.

```python multi_speaker.py theme={null}
import requests

API_KEY = "your_api_key"
HEADERS = {"Content-Type": "application/json", "x-api-key": API_KEY}

response = requests.post(
    "https://api.audioshake.ai/tasks",
    headers=HEADERS,
    json={
        "assetId": "your_asset_id",
        "targets": [
            {"model": "multi_voice", "formats": ["wav"], "residual": True}
        ]
    }
)

task_id = response.json()["id"]
print(f"Task created: {task_id}")
```

```javascript multiSpeaker.js theme={null}
const API_KEY = "your_api_key";
const headers = { "Content-Type": "application/json", "x-api-key": API_KEY };

const createRes = await fetch("https://api.audioshake.ai/tasks", {
  method: "POST",
  headers,
  body: JSON.stringify({
    assetId: "your_asset_id",
    targets: [
      { model: "multi_voice", formats: ["wav"], residual: true }
    ]
  })
});

const { id: taskId } = await createRes.json();
console.log(`Task created: ${taskId}`);
```

```bash curl theme={null}
curl -X POST "https://api.audioshake.ai/tasks" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $AUDIOSHAKE_API_KEY" \
  -d '{
    "assetId": "your_asset_id",
    "targets": [
      { "model": "multi_voice", "formats": ["wav"], "residual": true }
    ]
  }'
```

The model detects the number of speakers automatically. If you already know how many speakers are in the recording, set `numSpeakers` to pin it:

```json theme={null}
{ "model": "multi_voice", "formats": ["wav"], "numSpeakers": 2 }
```

[Check Task status](https://developer.audioshake.ai/check-task-status.md) to monitor progress and download results, or use [webhooks](https://developer.audioshake.ai/api-reference/tasks/webhooks.md) to be notified when each target completes.

## Outputs

A completed `multi_voice` target returns one file per detected speaker, plus a results file:

| Output | Format | Contents |
| - | - | - |
| `speaker_01` … `speaker_NN` | audio | One track per detected speaker, containing only that speaker's isolated voice |
| `multi_voice_residual` | audio | Everything that isn't speech — room tone, music, effects. Only returned when `residual: true` |
| `results` | json | Diarization timeline, speaker count, and confidence scores |

All tracks are time-aligned with the input and with each other; regions where a speaker is silent are silence in their track. Speaker numbering matches the `SPEAKER_XX` labels in the results file, and is local to each task — `SPEAKER_01` in one recording is not the same person as `SPEAKER_01` in another.

## The results file

The `results` output ties the audio tracks to a timeline:

```json results.json theme={null}
{
  "results": {
    "diarization": {
      "segments": [
        { "speaker": "SPEAKER_01", "start": 0.333, "end": 4.633 },
        { "speaker": "SPEAKER_02", "start": 4.832, "end": 12.672 },
        { "speaker": "SPEAKER_01", "start": 11.921, "end": 19.861 }
      ],
      "num_speakers": 2,
      "scores": {
        "scores_per_frame": {
          "assignment_confidence": { "resolution": 0.02, "score": [0.798, 0.982, "..."] },
          "separation_confidence": { "resolution": 0.02, "score": [0.998, 0.998, "..."] }
        },
        "scores": {
          "assignment_confidence": 0.887,
          "separation_confidence": 0.786
        }
      }
    }
  }
}
```

All fields below are relative to `results.diarization`:

* `segments` — one entry per speech segment, with `speaker` (matching the `speaker_XX` audio outputs) and `start`/`end` in seconds. The timeline is overlap-aware: segments from different speakers can overlap, as `SPEAKER_01` and `SPEAKER_02` do between 11.921s and 12.672s above. Nothing gets dropped when people talk at once.
* `num_speakers` — number of distinct speakers detected.
* `scores.scores_per_frame.<metric>` — per-frame confidence values in \[0, 1]. `resolution` is the frame duration in seconds (0.02 = 20 ms); multiply a frame index by it to get the timestamp.
* `scores.scores.<metric>` — whole-file confidence values, averaged with weighting by speech activity so silence can't inflate them.

## Confidence scores

Every job returns two scores, at 20 ms resolution and for the whole file:

* **`separation_confidence`** — how confident the model is that it correctly pulled apart overlapping speech. Most informative where speakers actually overlap; on single-speaker frames it saturates near 1.0 unless there is bleed between tracks.
* **`assignment_confidence`** — how confident the model is that speech is attributed to the right speaker. Treat this as the diarization-quality signal.

The two scores catch different failure modes, so read a dip in each differently:

* **Only `separation_confidence` dips** — the labels are right, but the audio in that region likely carries bleed from another voice. Still fine for talk-time analytics; exclude it from per-speaker ASR, dubbing, or voice training data.
* **Only `assignment_confidence` dips** — the audio is clean, but it may be attributed to the wrong speaker. Fine for identity-independent uses like cleanup; risky for anything routed by speaker.
* **Both dip together** — usually overlapping speech the model couldn't untangle. Drop the region, or route it to review.
* **Both high** — trust the region. Note that silence also scores high, so check the region against `segments` before treating a high score as confirmed clean speech.

If you need a single number, take the minimum of the two.

## Use cases

* **Transcription and captioning** — transcribe each speaker's track independently so overlapping speech is never assigned to the wrong person or dropped. The diarization timeline is already aligned to the tracks, with no second pass to reconcile.
* **Training data for speech AI** — turn real multi-party recordings into speaker-isolated tracks with turn boundaries and quality scores attached, and filter a corpus by confidence instead of by ear.
* **Post-production** — get a track per person in unscripted TV, documentaries, and interviews, and edit one voice without touching another. The residual track keeps room tone and effects available.
* **Voice and media pipelines** — feed clean per-speaker audio into translation, dubbing, or analysis, on the phone-quality and far-field recordings those pipelines actually receive.

> [card] [Speech Recovery](https://developer.audioshake.ai/speech-recovery.md)
>
> Denoise and de-reverb individual speaker stems after separation.

> [card] [Dialogue Separation](https://developer.audioshake.ai/remove-dialogue-for-dubbing.md)
>
> Separate all speech from music and effects instead.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.