Clone a voice and set its settings

A voice clone is a reusable voice of the workspace, built from audio samples or from one speaker of a project. Creating one is free within the plan’s quota. You need the plan feature voice_clones and the scope voices:write. Uploading a sample also needs media:write, and using the clone needs projects:write to create a project or dubbing:write to dub one, each with credits:spend. The program at the end also reads the project, which needs projects:read.

1. Check the quota

const { data: voiceCloneQuota } = await rask.voiceClones.getVoiceCloneQuota();

voice_clone_quota = client.voice_clones.get_voice_clone_quota()

voiceCloneQuota, err := client.VoiceClones.GetVoiceCloneQuota(ctx)
if err != nil {
	return err
}

VoiceCloneQuota voiceCloneQuota = rask.voiceClones().getVoiceCloneQuota().data();

$voiceCloneQuota = $rask->voiceClones->getVoiceCloneQuota();

curl https://api.rask.ai/v3/voice-clones/quota \
  -H "Authorization: Bearer $RASK_API_KEY"

used counts the workspace’s clones that are processing or ready, and remaining is how many are left. A failed or deleted clone frees its place. With none left, createVoiceClone answers 403 voice_clone_limit_reached. Beside this quota, the payer has a daily cap on clone creations, which also counts deleted and instant clones (429 rate_limited, policy clone-create-day).

2. Upload a sample

Send purpose: voice_sample. A sample can be audio up to 11 MB, kept as uploaded, or video up to 500 MB, whose audio is extracted to MP3 on import. As stored, each sample must be 11 MB or less: an import whose extracted audio is larger fails with media_too_large.

const { data: upload } = await rask.uploads.createUpload({
  body: { filename: 'maria-sample.m4a', content_type: 'audio/x-m4a', size_bytes: 3145728, purpose: 'voice_sample' },
});

upload = client.uploads.create_upload(
    filename="maria-sample.m4a", content_type="audio/x-m4a", size_bytes=3145728, purpose="voice_sample"
)

upload, err := client.Uploads.CreateUpload(ctx, api.CreateUploadRequest{
	Filename:    "maria-sample.m4a",
	ContentType: api.UploadContentTypeAudioxM4a,
	SizeBytes:   3145728,
	Purpose:     rask.Ptr(api.MediaPurposeVoiceSample),
})
if err != nil {
	return err
}

Upload upload = rask.uploads().createUpload(CreateUploadRequest.builder()
        .filename("maria-sample.m4a")
        .contentType(UploadContentType.AUDIO_X_M4A)
        .sizeBytes(3145728)
        .purpose(MediaPurpose.VOICE_SAMPLE)
        .build()).data();

$upload = $rask->uploads->createUpload([
    'filename' => 'maria-sample.m4a',
    'content_type' => 'audio/x-m4a',
    'size_bytes' => 3145728,
    'purpose' => 'voice_sample',
]);

curl -X POST https://api.rask.ai/v3/uploads \
  -H "Authorization: Bearer $RASK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"filename": "maria-sample.m4a", "content_type": "audio/x-m4a", "size_bytes": 3145728, "purpose": "voice_sample"}'

Send the bytes and complete the upload as in the Quickstart, and keep result.media_id of the media_import job.

3. Create the clone

Give 1 to 10 samples, and set the two settings if you want other than the defaults. Both run from 0 to 1. rights_confirmed must be true: it attests that you are the speaker or have their permission, and it is recorded.

const { data: job } = await rask.voiceClones.createVoiceClone({
  body: {
    name: 'Maria (narrator)',
    source: { type: 'media', media_ids: ['MEDIA_ID'] },
    settings: { accent: 0.3, expressiveness: 0.35 },
    rights_confirmed: true,
  },
});

job = client.voice_clones.create_voice_clone(
    name="Maria (narrator)",
    source={"type": "media", "media_ids": ["MEDIA_ID"]},
    settings={"accent": 0.3, "expressiveness": 0.35},
    rights_confirmed=True,
)

var source api.VoiceCloneSourceInput
if err := source.FromMediaVoiceCloneSourceInput(api.MediaVoiceCloneSourceInput{
	Type:     api.MediaVoiceCloneSourceInputTypeMedia,
	MediaIds: []string{"MEDIA_ID"},
}); err != nil {
	return err
}
job, err := client.VoiceClones.CreateVoiceClone(ctx, api.CreateVoiceCloneRequest{
	Name:            "Maria (narrator)",
	Source:          source,
	Settings:        &api.VoiceCloneSettingsInput{Accent: rask.Ptr(0.3), Expressiveness: rask.Ptr(0.35)},
	RightsConfirmed: true,
})
if err != nil {
	return err
}

Job job = rask.voiceClones().createVoiceClone(CreateVoiceCloneRequest.builder()
        .name("Maria (narrator)")
        .source(MediaVoiceCloneSourceInput.builder().mediaIds(List.of("MEDIA_ID")).build())
        .settings(VoiceCloneSettingsInput.builder().accent(0.3).expressiveness(0.35).build())
        .rightsConfirmed(true)
        .build()).data();

$job = $rask->voiceClones->createVoiceClone([
    'name' => 'Maria (narrator)',
    'source' => ['type' => 'media', 'media_ids' => ['MEDIA_ID']],
    'settings' => ['accent' => 0.3, 'expressiveness' => 0.35],
    'rights_confirmed' => true,
]);

curl -X POST https://api.rask.ai/v3/voice-clones \
  -H "Authorization: Bearer $RASK_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"name": "Maria (narrator)", "source": {"type": "media", "media_ids": ["MEDIA_ID"]}, "settings": {"accent": 0.3, "expressiveness": 0.35}, "rights_confirmed": true}'

To clone a speaker of a project instead, use {"type": "project_speaker", "project_id": "PROJECT_ID", "speaker_id": "SPEAKER_01"} as the source. The answer is a voice_clone job. The clone exists at once with status: processing, and the job’s params.voice_clone_id names it. When the job succeeds, the clone is ready and result.voice_clone_id names it too. When the job fails, the clone is failed, with its failure, and no longer counts against the quota.

4. Read its voice ids

A clone has one voice per language. getVoiceClone lists them in languages, each with its voice_id and a sample. That voice_id is what you assign to a speaker.

const { data: voiceClone } = await rask.voiceClones.getVoiceClone({ path: { voice_clone_id: voiceCloneId } });

voice_clone = client.voice_clones.get_voice_clone(voice_clone_id)

voiceClone, err := client.VoiceClones.GetVoiceClone(ctx, voiceCloneID)
if err != nil {
	return err
}

VoiceClone voiceClone = rask.voiceClones().getVoiceClone(voiceCloneId).data();

$voiceClone = $rask->voiceClones->getVoiceClone($voiceCloneId);

curl "https://api.rask.ai/v3/voice-clones/$VOICE_CLONE_ID" \
  -H "Authorization: Bearer $RASK_API_KEY"

5. Change its settings

const { data: voiceClone } = await rask.voiceClones.updateVoiceClone({
  path: { voice_clone_id: voiceCloneId },
  body: { settings: { accent: 0.61 } },
});

voice_clone = client.voice_clones.update_voice_clone(voice_clone_id, settings={"accent": 0.61})

voiceClone, err := client.VoiceClones.UpdateVoiceClone(ctx, voiceCloneID, api.UpdateVoiceCloneRequest{
	Settings: &api.VoiceCloneSettingsInput{Accent: rask.Ptr(0.61)},
})
if err != nil {
	return err
}

VoiceClone voiceClone = rask.voiceClones().updateVoiceClone(voiceCloneId, UpdateVoiceCloneRequest.builder()
        .settings(VoiceCloneSettingsInput.builder().accent(0.61).build())
        .build()).data();

$voiceClone = $rask->voiceClones->updateVoiceClone($voiceCloneId, ['settings' => ['accent' => 0.61]]);

curl -X PATCH "https://api.rask.ai/v3/voice-clones/$VOICE_CLONE_ID" \
  -H "Authorization: Bearer $RASK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"settings": {"accent": 0.61}}'

Settings apply to the next segment generation or dub at once.
Segments whose speaker uses the clone get dub_state: outdated, so the next dub voices them again.
Each language’s sample is regenerated when the clone is next used in that language. Until then samples_status is regenerating and that language’s sample is null. If a regeneration fails, samples_status is stale and the next use in that language tries again; the settings still apply.
A clone still processing cannot be changed: 409 voice_clone_not_ready.
By default a clone takes 10 settings changes per hour (429 rate_limited, policy voice-settings-hour). A request that only renames the clone, or sends the settings it already has, does not count.
The source files cannot be replaced: create a new clone.

6. Use it on a project

On a new project, name the voice in speaker_voices, so the first dub already uses it. Speaker 1 is the first speaker that detection numbers. A label that never occurs is ignored with the notice speaker_voice_unused. If a voice cannot be applied once speakers are known, the speaker keeps the suggested voice, with the notice speaker_voice_not_applied.

const { data: job } = await rask.projects.createProject({
  body: {
    media_id: 'MEDIA_ID',
    target_language: 'es-es',
    speaker_voices: [{ speaker_label: 'Speaker 1', voice_id: 'VOICE_ID' }],
    expected_cost_minutes: 1,
  },
  idempotencyKey: createKey,
});

job = client.projects.create_project(
    media_id="MEDIA_ID",
    target_language="es-es",
    speaker_voices=[{"speaker_label": "Speaker 1", "voice_id": "VOICE_ID"}],
    expected_cost_minutes=1,
    idempotency_key=create_key,
)

job, err := client.Projects.CreateProject(ctx, api.CreateProjectRequest{
	MediaId:             "MEDIA_ID",
	TargetLanguage:      "es-es",
	SpeakerVoices:       &[]api.SpeakerVoiceInput{{SpeakerLabel: "Speaker 1", VoiceId: "VOICE_ID"}},
	ExpectedCostMinutes: rask.Ptr(int32(1)),
}, rask.WithIdempotencyKey(createKey))
if err != nil {
	return err
}

Job job = rask.projects().createProject(
        CreateProjectRequest.builder()
                .mediaId("MEDIA_ID")
                .targetLanguage("es-es")
                .speakerVoices(List.of(SpeakerVoiceInput.builder()
                        .speakerLabel("Speaker 1")
                        .voiceId("VOICE_ID")
                        .build()))
                .expectedCostMinutes(1)
                .build(),
        RequestOptions.builder().idempotencyKey(createKey).build()).data();

$job = $rask->projects->createProject(
    [
        'media_id' => 'MEDIA_ID',
        'target_language' => 'es-es',
        'speaker_voices' => [['speaker_label' => 'Speaker 1', 'voice_id' => 'VOICE_ID']],
        'expected_cost_minutes' => 1,
    ],
    ['idempotency_key' => $createKey],
);

CREATE_KEY=$(uuidgen)
curl -X POST https://api.rask.ai/v3/projects \
  -H "Authorization: Bearer $RASK_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $CREATE_KEY" \
  -d '{"media_id": "MEDIA_ID", "target_language": "es-es", "speaker_voices": [{"speaker_label": "Speaker 1", "voice_id": "VOICE_ID"}], "expected_cost_minutes": 1}'

On an existing project, read the speaker’s ETag and assign the voice. The speaker’s segments lose their audio and become outdated.

const { data: speaker } = await rask.speakers.getSpeaker({ path: { project_id: projectId, speaker_id: 'SPEAKER_00' } });

const { data: speakerVoiceUpdateResult } = await rask.speakers.updateSpeakerVoice({
  path: { project_id: projectId, speaker_id: 'SPEAKER_00' },
  body: { voice_id: 'VOICE_ID' },
  ifMatch: '"v-1a2b3c4d5e6f7a8b"',
});

speaker = client.speakers.get_speaker(project_id, "SPEAKER_00")

speaker_voice_update_result = client.speakers.update_speaker_voice(
    project_id, "SPEAKER_00", if_match='"v-1a2b3c4d5e6f7a8b"', voice_id="VOICE_ID"
)

speaker, err := client.Speakers.GetSpeaker(ctx, projectID, "SPEAKER_00")
if err != nil {
	return err
}

speakerVoiceUpdateResult, err := client.Speakers.UpdateSpeakerVoice(
	ctx,
	projectID,
	"SPEAKER_00",
	`"v-1a2b3c4d5e6f7a8b"`,
	api.UpdateSpeakerVoiceRequest{VoiceId: "VOICE_ID"},
)
if err != nil {
	return err
}

Speaker speaker = rask.speakers().getSpeaker(projectId, "SPEAKER_00").data();

SpeakerVoiceUpdateResult speakerVoiceUpdateResult = rask.speakers().updateSpeakerVoice(
        projectId,
        "SPEAKER_00",
        "\"v-1a2b3c4d5e6f7a8b\"",
        UpdateSpeakerVoiceRequest.builder().voiceId("VOICE_ID").build()).data();

$speaker = $rask->speakers->getSpeaker($projectId, 'SPEAKER_00');

$speakerVoiceUpdateResult = $rask->speakers->updateSpeakerVoice(
    $projectId,
    'SPEAKER_00',
    ['voice_id' => 'VOICE_ID'],
    ['if_match' => '"v-1a2b3c4d5e6f7a8b"'],
);

curl "https://api.rask.ai/v3/projects/$PROJECT_ID/speakers/SPEAKER_00" \
  -H "Authorization: Bearer $RASK_API_KEY"

curl -X PUT "https://api.rask.ai/v3/projects/$PROJECT_ID/speakers/SPEAKER_00/voice" \
  -H "Authorization: Bearer $RASK_API_KEY" \
  -H "Content-Type: application/json" \
  -H 'If-Match: "v-1a2b3c4d5e6f7a8b"' \
  -d '{"voice_id": "VOICE_ID"}'

The result’s effects.next_dub_mode says which mode the next dub applies. Then estimateDub and createDub with mode: changed_segments voice the change. On a project that has never been dubbed, that mode is refused with no_previous_dub (409 from createDub, a reason in the estimate): use mode: auto for its first dub.

Instant clones

A speaker can also use their own voice, an instant clone, without a library clone. Assign the instant_clone voice from listProjectVoices, with instant_clone_settings. It needs the plan feature instant_voice_clone, and a speaker it can work for: instant_clone.eligible and instant_clone.reason on the speaker say whether it can (enough speech, a supported language and voice provider). Otherwise the assignment answers 422 instant_clone_not_available. The first instant_clone_settings for a speaker create its instant clone, which counts against the daily cap; changing them later counts against voice-settings-hour.

const { data: speakerVoiceUpdateResult } = await rask.speakers.updateSpeakerVoice({
  path: { project_id: projectId, speaker_id: 'SPEAKER_00' },
  body: { voice_id: 'VOICE_ID', instant_clone_settings: { accent: 0.3, expressiveness: 0.65 } },
  ifMatch: '"v-1a2b3c4d5e6f7a8b"',
});

speaker_voice_update_result = client.speakers.update_speaker_voice(
    project_id,
    "SPEAKER_00",
    if_match='"v-1a2b3c4d5e6f7a8b"',
    voice_id="VOICE_ID",
    instant_clone_settings={"accent": 0.3, "expressiveness": 0.65},
)

speakerVoiceUpdateResult, err := client.Speakers.UpdateSpeakerVoice(
	ctx,
	projectID,
	"SPEAKER_00",
	`"v-1a2b3c4d5e6f7a8b"`,
	api.UpdateSpeakerVoiceRequest{
		VoiceId:              "VOICE_ID",
		InstantCloneSettings: &api.InstantCloneSettingsInput{Accent: 0.3, Expressiveness: 0.65},
	},
)
if err != nil {
	return err
}

SpeakerVoiceUpdateResult speakerVoiceUpdateResult = rask.speakers().updateSpeakerVoice(
        projectId,
        "SPEAKER_00",
        "\"v-1a2b3c4d5e6f7a8b\"",
        UpdateSpeakerVoiceRequest.builder()
                .voiceId("VOICE_ID")
                .instantCloneSettings(InstantCloneSettingsInput.builder().accent(0.3).expressiveness(0.65).build())
                .build()).data();

$speakerVoiceUpdateResult = $rask->speakers->updateSpeakerVoice(
    $projectId,
    'SPEAKER_00',
    ['voice_id' => 'VOICE_ID', 'instant_clone_settings' => ['accent' => 0.3, 'expressiveness' => 0.65]],
    ['if_match' => '"v-1a2b3c4d5e6f7a8b"'],
);

curl -X PUT "https://api.rask.ai/v3/projects/$PROJECT_ID/speakers/SPEAKER_00/voice" \
  -H "Authorization: Bearer $RASK_API_KEY" \
  -H "Content-Type: application/json" \
  -H 'If-Match: "v-1a2b3c4d5e6f7a8b"' \
  -d '{"voice_id": "VOICE_ID", "instant_clone_settings": {"accent": 0.3, "expressiveness": 0.65}}'

The whole program

clone_voice.py

"""Clone a voice from a sample and dub a project's first speaker with it: python clone_voice.py sample.m4a PROJECT_ID

Its keys are saved in clone-PROJECT_ID.json: run it again within 24 hours, and no paid work starts twice. Delete the file to
start anew.
"""
import sys

from rask_client import Progress, call, upload_file, wait_for_job

sample_path, project_id = sys.argv[1], sys.argv[2]
progress = Progress(f"clone-{project_id}.json")

quota = call("GET", "/voice-clones/quota")
if quota["remaining"] == 0:
    sys.exit("The plan's clone quota is used up.")

sample_id = progress.once("sample_id", lambda: upload_file(sample_path, "audio/x-m4a", purpose="voice_sample"))
job = call(
    "POST",
    "/voice-clones",
    {
        "name": "Maria (narrator)",
        "source": {"type": "media", "media_ids": [sample_id]},
        "settings": {"accent": 0.3, "expressiveness": 0.35},
        "rights_confirmed": True,
    },
    key=progress.key("clone"),
)
clone_id = wait_for_job(job["id"])["result"]["voice_clone_id"]

call("PATCH", f"/voice-clones/{clone_id}", {"settings": {"accent": 0.61}})

project = call("GET", f"/projects/{project_id}")
clone = call("GET", f"/voice-clones/{clone_id}")
voice_id = next(item["voice_id"] for item in clone["languages"] if item["language"] == project["target_language"])

speaker = call("GET", f"/projects/{project_id}/speakers/SPEAKER_00")
call(
    "PUT",
    f"/projects/{project_id}/speakers/SPEAKER_00/voice",
    {"voice_id": voice_id},
    if_match=speaker["etag"],
)

estimate = call("POST", f"/projects/{project_id}/dub/estimate", {"mode": "changed_segments"})
if not estimate["eligible"]:
    sys.exit(f"Cannot dub: {estimate['reasons']}")
dub = call(
    "POST",
    f"/projects/{project_id}/dub",
    {"mode": "changed_segments", "expected_cost_minutes": estimate["total_minutes"]},
    key=progress.key("dub"),
)
wait_for_job(dub["id"])

Get started

AI agents

Recipes

Concepts

Errors

Account

Usage

Catalogs

Uploads

Media

Projects

Exports

Glossaries

Jobs

Transcript

Segments

Versions

Speakers

Voices

Voice clones

Dubbing

Lipsync

Onscreen

Clone a voice and set its settings

1. Check the quota

2. Upload a sample

3. Create the clone

4. Read its voice ids

5. Change its settings

6. Use it on a project

Instant clones

The whole program

​1. Check the quota

​2. Upload a sample

​3. Create the clone

​4. Read its voice ids

​5. Change its settings

​6. Use it on a project

​Instant clones

​The whole program

1. Check the quota

2. Upload a sample

3. Create the clone

4. Read its voice ids

5. Change its settings

6. Use it on a project

Instant clones

The whole program