Validation samples — base model, SFT, ground truth
18 utterances from the held-out validation split, which was selected by
uid hash and never trained on. Every row plays the same prompt three ways: the reference
recording, the base model before this fine-tune, and the fine-tune. Same seed, same sampling
parameters for both models. The exact prompt string is under each card.
Validation loss4.7076 → 4.6314
Voice profiles4.4786 → 4.4030
Real speech4.8668 → 4.7902
Training256 GPUs · 1h53
Held-out reconstruction — base, fine-tune, ground truth
00
voice profilesDEreference audiogen_generalduration tag
Um diese lebenswichtigen Banken der Lech und Donau zu halten, wurde Elsass völlig ausgeplündert. Rheingraf Otto Ludwig kämpfte verzweifelt gegen die aufgebrachte Bauernschaft, nachdem Horn gegangen war.
emolia_c1073 · Aaron
· 26.4 s spoken / 32.0 s clip · 400 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
<|audio|>
- Instruction:
GENERAL: A voice very high in emphasis. The emphasis is applied with violent, explosive intensity, creating massive acoustic disparities between stressed and unstressed words. Highlighted syllables erupt with jarring volume and prolonged duration, resembling a high-pressure sales pitch or a furious argument.
- Tokens:
400
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
[26.4 seconds duration] Um diese lebenswichtigen Banken der Lech und Donau zu halten, wurde Elsass völlig ausgeplündert. Rheingraf Otto Ludwig kämpfte verzweifelt gegen die aufgebrachte Bauernschaft, nachdem Horn (contented sigh) gegangen war. (contented sigh) (contented sigh)
</user_inst>
01
voice profilesENspeaker namecaption_tpl
By varying relaxation times and using gadolinium as contrast, we can actually tell apart spinal tumors from other diseases. (scream) This usually clarifies whether a syringomyelia is communicating or not.
mediathek_0170 · Günther
· 8.1 s spoken / 9.1 s clip · 114 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
Speaker: Günther
- Instruction:
VOICE: adult masculine
DELIVERY: normally alert, normal-paced, neutral tension, fairly steady
TIMBRE: slightly warm, neutral-bright, fairly smooth, full
SPEECH: average clarity, some disfluency, moderate pitch range, light breath
AFFECT: mildly negative, slightly dominant, slightly guarded
EMOTION: fear, pain, helplessness
STYLE: casual, conversational
RECORDING: good recording, no background noise
GENUINENESS: 2.3/6
BLEND: 5.3/10
DURATION: 9.1s
LANG: EN
- Tokens:
114
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
(surprised gasp) By varying relaxation times and using gadolinium as contrast, we can actually tell apart spinal tumors from other diseases. (scream) This usually clarifies whether a syringomyelia is communicating or not.
</user_inst>
02
real speechDEno referencegen_scriptduration tagburst cues stripped
Werd' ich begleitet und alles andere ergibt sich von selbst, dann ist natürlich auch noch das Wetter schön, das Essen ist gut und dann, ja, geht's einem definitiv besser.
em:DE_WYhrg9a_TXQ
· 10.3 s spoken / 10.6 s clip · 131 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
SCRIPT:
Werd' ich begleitet und alles andere ergibt sich von selbst, dann ist natürlich auch noch das Wetter schön, das Essen ist gut und dann, ja, geht's einem definitiv besser.
- Tokens:
131
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
[10.3 seconds duration] Werd' ich begleitet und alles andere ergibt sich von selbst, dann ist natürlich auch noch das Wetter schön, das Essen ist gut und dann, ja, geht's einem definitiv besser.
</user_inst>
03
real speechDEreference audiogen_bothduration tag
Schwierig, difficult. Ich fand Mathematik immer sehr schwierig. I always thought math was very difficult.
em:DE_B00000_S05976
· 8.6 s spoken / 9.9 s clip · 124 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
<|audio|>
- Instruction:
GENERAL: A middle-aged somewhat feminine voice; delivery is very low-energy, measured, relaxed, moderately variable; timbre is neutral-toned, slightly dark, slightly rough, balanced body; average clarity, frequent disfluency, wide pitch range, normal breath; affect is mildly positive, slightly submissive, neutral openness; reads as relief, fatigue exhaustion, shame; style: whispered, conversational; good recording, quiet background; genuineness 2.0/6; vocal-burst blend 0.4/10; 9.9s, DE.
SCRIPT:
Schwierig, difficult. Ich fand Mathematik immer sehr schwierig. I always thought math was very difficult.
- Tokens:
124
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
[8.6 seconds duration] Schwierig, difficult. Ich fand Mathematik immer sehr schwierig. I always thought math was very difficult.
</user_inst>
04
real speechENno referencegen_scriptduration tag
I never lose hope. I never lose hope for anybody and one of the reasons why is because I'm a recovering drug addict and alcoholic and seven years ago, I had a 10% chance of living and I did not want to live. Like the doctors told me, they said, listen,
em:EN_VKoJgW4jld0_W000001
· 14.4 s spoken / 14.9 s clip · 186 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
SCRIPT:
I never lose hope. I never lose hope for anybody and one of the reasons why is because I'm a recovering drug addict and alcoholic and seven years ago, I had a 10% chance of living and I did not want to live. Like the doctors told me, they said, listen,
- Tokens:
186
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[14.4 seconds duration] I never lose hope. I never lose hope for anybody and one of the reasons why is because I'm a recovering drug addict and alcoholic and seven years ago, I had a 10% chance of living and I did not want to live. Like the doctors told me, they said, listen,
</user_inst>
05
real speechENno referencegen_bothduration tag
Interesting. Did you hear what he said? Like, you call me a beast, I didn't ask for this. Oh damn. That's a cool fucking... That's a cool character development kinda deal.
em:EN_B00044_S03705
· 9.6 s spoken / 9.9 s clip · 123 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: A young adult masculine voice; delivery is normally alert, normal-paced, neutral tension, moderately variable; timbre is neutral-toned, neutral-bright, slightly rough, balanced body; average clarity, some disfluency, wide pitch range, light breath; affect is mildly negative, slightly dominant, neutral openness; reads as astonishment surprise, awe, interest; style: casual, conversational; average recording, quiet background; mildly explicit content; genuineness 5.3/6; vocal-burst blend 4.1/10; 9.9s, EN.
SCRIPT:
Interesting. Did you hear what he said? Like, you call me a beast, I didn't ask for this. Oh damn. That's a cool fucking... That's a cool character development kinda deal.
- Tokens:
123
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[9.6 seconds duration] Interesting. Did you hear what he said? Like, you call me a beast, I didn't ask for this. Oh damn. That's a cool fucking... That's a cool character development kinda deal.
</user_inst>
06
voice profilesDEreference audiocaption_tpl
Hollister Jogginghosen, Leute! Die Freunde machen Moves wie die Stooges, ein kritisches Blackjack-Spiel entfaltet sich gerade!
emolia_c0421 · Eleanor
· 4.5 s spoken / 4.6 s clip · 58 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
<|audio|>
- Instruction:
Speak with a young adult feminine voice. Keep the delivery highly aroused, fast, slightly tense and moderately variable. Let the timbre sit slightly cool, slightly bright, fairly smooth and thin. Colour it with impatience and irritability, anger and contempt. The affect should read elated, very dominant and guarded.
- Tokens:
58
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
Hollister Jogginghosen, Leute! Die Freunde machen Moves wie die Stooges, ein kritisches Blackjack-Spiel entfaltet sich gerade!
</user_inst>
07
voice profilesENspeaker namecaption_tplduration tag
Do you happen to know which song is playing in Rurouni Kenshin during that moment between Okina and Shinomori? (gurgling) I'm trying to remember the exact title of the track.
emolia_c0155 · Daniel
· 5.6 s spoken / 5.7 s clip · 71 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
Speaker: Daniel
- Instruction:
[young adult masculine, normally alert, normal-paced; infatuation and longing]
- Tokens:
71
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[5.6 seconds duration] Do you happen to know which song is playing in Rurouni Kenshin during that moment between Okina and Shinomori? (gurgling) I'm trying to remember the exact title of the track.
</user_inst>
08
real speechDEno referencegen_both
Aber wir sind immer noch in einer Phase, in der wir Ausbrüche sehen, in der das Virus sich nicht irgendwie gleichmäßig durch die Bevölkerung verteilt hat, also die echte Pandemie. Dieser ersten Phase waren auch die Ptalsur-Kranknäuser schwer belastet mit Patienten.
em:DE_8wjMQiALN-A_W000004
· 18.7 s spoken / 18.8 s clip · 235 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: A middle-aged masculine voice; delivery is subdued, measured, slightly relaxed, steady; timbre is neutral-toned, slightly dark, rough, balanced body; somewhat unclear, frequent disfluency, fairly narrow pitch, normal breath; affect is neutral, neutral stance, slightly guarded; reads as helplessness, fear, contemplation; style: monologue; average recording, quiet background; genuineness 2.5/6; vocal-burst blend 0.1/10; 18.8s, DE.
SCRIPT:
Aber wir sind immer noch in einer Phase, in der wir Ausbrüche sehen, in der das Virus sich nicht irgendwie gleichmäßig durch die Bevölkerung verteilt hat, also die echte Pandemie. Dieser ersten Phase waren auch die Ptalsur-Kranknäuser schwer belastet mit Patienten.
- Tokens:
235
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
Aber wir sind immer noch in einer Phase, in der wir Ausbrüche sehen, in der das Virus sich nicht irgendwie gleichmäßig durch die Bevölkerung verteilt hat, also die echte Pandemie. Dieser ersten Phase waren auch die Ptalsur-Kranknäuser schwer belastet mit Patienten.
</user_inst>
09
real speechDEreference audiogen_bothduration tag
Ja, Pflanzchen. Ja, Pflanzchen. Ja, sehr gut. Ah, so, ah, wir haben hier eine Pflanze.
em:DE_B00000_S09700
· 5.5 s spoken / 7.1 s clip · 88 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
<|audio|>
- Instruction:
GENERAL: A young adult feminine voice; delivery is normally alert, normal-paced, slightly relaxed, fairly steady; timbre is neutral-toned, neutral-bright, fairly smooth, balanced body; average clarity, some disfluency, moderate pitch range, light breath; affect is mildly positive, slightly submissive, neutral openness; reads as elation; style: casual, conversational; average recording, quiet background; genuineness 3.5/6; vocal-burst blend 0.4/10; 7.1s, DE.
SCRIPT:
Ja, Pflanzchen. Ja, Pflanzchen. Ja, sehr gut. (ahem) Ah, so, (ahem) ah, wir haben hier eine Pflanze.
- Tokens:
88
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
[5.5 seconds duration] Ja, Pflanzchen. Ja, Pflanzchen. Ja, sehr gut. (ahem) Ah, so, (ahem) ah, wir haben hier eine Pflanze.
</user_inst>
10
real speechENno referencegen_bothduration tag
So what might be relevant to Germany might not be relevant to Ghana. So I think we need to really look at ourselves in terms of our mitigation. Of course we'll still mitigate.
em:EN_n34Ue0uu4zI_W000058
· 7.7 s spoken / 7.9 s clip · 98 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: An adult masculine voice; delivery is normally alert, normal-paced, slightly relaxed, fairly steady; timbre is neutral-toned, neutral-bright, fairly smooth, balanced body; somewhat unclear, some disfluency, moderate pitch range, light breath; affect is neutral, neutral stance, slightly guarded; reads as doubt, contemplation, infatuation; style: casual, monologue; average recording, quiet background; genuineness 5.4/6; vocal-burst blend 6.1/10; 7.9s, EN.
SCRIPT:
So what might be relevant to Germany might not be relevant to Ghana. So I think we need to really look at ourselves in terms of our mitigation. Of course we'll still mitigate.
- Tokens:
98
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[7.7 seconds duration] So what might be relevant to Germany might not be relevant to Ghana. So I think we need to really look at ourselves in terms of our mitigation. Of course we'll still mitigate.
</user_inst>
11
real speechENno referencegen_bothduration tag
So, you know, we had to plan for those continues. We had to be ready, but that was a strain that I certainly remember well. And I remember that conversation as I'm sure you do coming back and recognizing like.
em:EN_B00008_S00583
· 9.6 s spoken / 9.8 s clip · 122 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: An adult masculine voice; delivery is normally alert, normal-paced, neutral tension, fairly steady; timbre is neutral-toned, neutral-bright, fairly smooth, balanced body; somewhat unclear, some disfluency, moderate pitch range, light breath; affect is mildly negative, neutral stance, slightly guarded; reads as longing, contemplation; style: casual, conversational; good recording, no background noise; genuineness 4.2/6; vocal-burst blend 10.0/10; 9.8s, EN.
SCRIPT:
So, you know, we had to plan for those continues. We had to be ready, but that was a strain that I certainly remember well. And I remember that conversation as I'm sure you do coming back and recognizing like.
- Tokens:
122
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[9.6 seconds duration] So, you know, we had to plan for those continues. We had to be ready, but that was a strain that I certainly remember well. And I remember that conversation as I'm sure you do coming back and recognizing like.
</user_inst>
12
voice profilesDEspeaker namecaption_tplduration tag
Die NASA ist die Erkundungsagentur unseres Planeten. Wir wollen die Weite des Weltraums und die Möglichkeiten, die er für die Menschheit birgt, verstehen.
emolia_c0359 · Morgan
· 8.3 s spoken / 10.4 s clip · 130 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
Speaker: Morgan
- Instruction:
Reads as pride, elation and awe, in an adult masculine voice. Affect neutral, neutral stance and slightly guarded. Delivered normally alert, measured, slightly relaxed and fairly steady.
- Tokens:
130
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
[8.3 seconds duration] Die NASA ist die Erkundungsagentur unseres Planeten. Wir wollen die Weite des Weltraums und die Möglichkeiten, die er für die Menschheit birgt, verstehen.
</user_inst>
13
voice profilesENspeaker namegen_both
You have to quantify it somehow, right? Because love, it’s just a big thing, and you need a way to actually measure what that means.
emolia_c0749 · Iris
· 6.4 s spoken / 7.0 s clip · 87 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
Speaker: Iris
- Instruction:
GENERAL: A voice somewhat low in warmth. The vocal resonance is highly functional but distinctly neutral-cool, operating with a very dry acoustic profile that prioritizes hard information over comfort. The low-mid frequencies are present but distinctly suppressed, preventing the voice from sounding naturally inviting or physically relaxed.
SCRIPT:
(neutral, natural delivery, exactly as this voice normally speaks) "You have to quantify it somehow, right? Because love, it’s just a big thing, and you need a way to actually measure what that means."
- Tokens:
87
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
You (childlike giggle) have to quantify it somehow, right? Because love, it’s just a big thing, and you need a way to actually measure what that means.
</user_inst>
14
real speechDEno referencegen_bothduration tag
Aber da machen wir noch ein schöneres Bild. Also, der, das, das Foto kommt ihm nicht nahe.
unk:kartoffelphon
· 5.6 s spoken / 5.8 s clip · 72 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: A young adult masculine voice; delivery is normally alert, normal-paced, relaxed, fairly steady; timbre is neutral-toned, neutral-bright, slightly rough, balanced body; somewhat unclear, frequent disfluency, moderate pitch range, light breath; affect is neutral, slightly submissive, neutral openness; reads as confusion, intoxication altered states of consciousness, affection; style: casual, conversational; below-average recording, some background noise; genuineness 5.4/6; vocal-burst blend 1.6/10; 5.8s, DE.
SCRIPT:
Aber da machen wir noch ein schöneres Bild. Also, der, das, das Foto kommt ihm nicht nahe.
- Tokens:
72
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
[5.6 seconds duration] Aber da machen wir noch ein schöneres Bild. Also, der, das, das Foto kommt ihm nicht nahe.
</user_inst>
15
real speechDEreference audiogen_bothduration tag
Sie so mit dem bloßen Auge noch gut aussieht. Die Bakterien darauf seht ihr nicht. Die befinden sich aber auf dieser Zahnbürste und die vermehren sich mit der Zeit mit jeder Benutzung und jedem mal, wo sie feucht wird und wieder trocknet und so. Es werden einfach mit der Zeit immer mehr Bakterien. Die Bakterien sammeln sich auch insbesondere dort, wo die Borsten an dem Kopf befestigt sind, also um
em:DE__XdTQ3zPTrI
· 23.6 s spoken / 24.1 s clip · 301 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
<|audio|>
- Instruction:
GENERAL: A child feminine voice; delivery is normally alert, normal-paced, slightly relaxed, moderately variable; timbre is neutral-toned, neutral-bright, fairly smooth, balanced body; average clarity, some disfluency, moderate pitch range, light breath; affect is mildly positive, slightly dominant, neutral openness; reads as longing, sourness, affection; style: monologue, casual; average recording, quiet background; genuineness 3.7/6; vocal-burst blend 0.9/10; 24.1s, DE.
SCRIPT:
Sie so mit dem bloßen Auge noch gut aussieht. Die Bakterien darauf seht ihr nicht. Die befinden sich aber auf dieser Zahnbürste und die vermehren sich mit der Zeit mit jeder Benutzung und jedem mal, wo sie feucht wird und wieder trocknet und so. Es werden einfach mit der Zeit immer mehr Bakterien. Die Bakterien sammeln sich auch insbesondere dort, wo die Borsten an dem Kopf befestigt sind, also um
- Tokens:
301
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
German
- Text:
[23.6 seconds duration] Sie so mit dem bloßen Auge noch gut aussieht. Die Bakterien darauf seht ihr nicht. Die befinden sich aber auf dieser Zahnbürste und die vermehren sich mit der Zeit mit jeder Benutzung und jedem mal, wo sie feucht wird und wieder trocknet und so. Es werden einfach mit der Zeit immer mehr Bakterien. Die Bakterien sammeln sich auch insbesondere dort, wo die Borsten an dem Kopf befestigt sind, also um
</user_inst>
16
real speechENno referencegen_both
Have you heard Bill's podcast from last week? No, I haven't. Dude, that one's fucking good. I'm telling you, it's too funny.
em:EN_jONls6ocDW4_W000178
· 7.0 s spoken / 7.2 s clip · 89 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: A young adult masculine voice; delivery is normally alert, normal-paced, neutral tension, moderately variable; timbre is neutral-toned, neutral-bright, slightly rough, thin; somewhat unclear, some disfluency, moderate pitch range, normal breath; affect is mildly positive, slightly dominant, neutral openness; reads as intoxication altered states of consciousness, elation, pleasure ecstasy; style: casual, conversational; below-average recording, some background noise; mildly explicit content; genuineness 6.0/6; vocal-burst blend 8.2/10; 7.2s, EN.
SCRIPT:
Have you heard Bill's podcast from last week? No, I haven't. Dude, that one's fucking good. I'm telling you, it's too funny.
- Tokens:
89
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
Have you heard Bill's podcast from last week? No, I haven't. Dude, that one's fucking good. I'm telling you, it's too funny.
</user_inst>
17
real speechENno referencegen_bothduration tag
Like my first time, you know, hearing about you was when you was with the P Miller, uh, it was the all stars, right? Um, like how did, how did that relationship form? Like how did, how did that come about?
em:EN_B00037_S03792
· 12.8 s spoken / 12.9 s clip · 160 frames
Ground truthheld-out reference audio
Base modelvoice-acting-v2, before any of this
SFTfull fine-tune
SFT + DPO LoRArank-64 preference LoRA on the SFT model
SFT + DPO fullfull-parameter preference tuning of the SFT model
Exact prompt the model received
<user_inst>
- Reference(s):
None
- Instruction:
GENERAL: An adult masculine voice; delivery is normally alert, normal-paced, neutral tension, fairly steady; timbre is neutral-toned, neutral-bright, fairly smooth, balanced body; somewhat unclear, some disfluency, moderate pitch range, light breath; affect is neutral, slightly dominant, neutral openness; reads as longing, affection, infatuation; style: casual, conversational; good recording, no background noise; mildly explicit content; genuineness 4.3/6; vocal-burst blend 7.1/10; 12.9s, EN.
SCRIPT:
Like my first time, you know, hearing about you was when you was with the P Miller, (low mumble) uh, it was the all stars, right? (low mumble) Um, like how did, how did that relationship form? Like how did, how did that come about?
- Tokens:
160
- Quality:
None
- Sound Event:
None
- Ambient Sound:
None
- Language:
English
- Text:
[12.8 seconds duration] Like my first time, you know, hearing about you was when you was with the P Miller, (low mumble) uh, it was the all stars, right? (low mumble) Um, like how did, how did that relationship form? Like how did, how did that come about?
</user_inst>
Emotion control — all 40 heads, three prompts each
Two matched A/B pairs per emotion. Prompts 1 and 2 are the same short
sentence in the same voice, one without and one with a parenthesised inline direction;
prompts 3 and 4 are the same multi-sentence passage, one plain and one carrying the
emotion cue plus a vocal-burst cue placed mid-utterance. Exactly half of all prompts carry
inline direction, which is the format 59.3 % of training rows used. The older text below
describes the carrier idea:
— one English, one German — so the words are held fixed and only the emotion field of
the instruction changes. Scan down and the delivery should change while the sentence does not.
If it does not change, the model is ignoring that field, and this grid shows that rather than
hiding it. The third prompt is a line written for the emotion. There is no ground truth here:
these are new prompts, not held-out recordings.
affection
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (affection, warm, soft-edged, unhurried)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
I kept your chair by the window, the way you like it. Sit down, would you. I have so much to tell you and none of it is urgent.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
I kept your chair by the window, the way you like it. Sit down, would you. I have so much to tell you and none of it is urgent.
Complex passage — no inline direction caption onlyEnglish
Hold on, go back. You're saying it was sitting in the drawer this entire time? The whole time we were tearing the house apart, it was three feet away from us?
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
Hold on, go back. You're saying it was sitting in the drawer this entire time? The whole time we were tearing the house apart, it was three feet away from us?
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (awe, breathy, slow, open)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
I didn't expect it to be quiet. You look up and it just keeps going, and somewhere up there it stops being a number and starts being something else entirely.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
I didn't expect it to be quiet. You look up and it just keeps going, and somewhere up there it stops being a number and starts being something else entirely.
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (bitterness, dark, tight, clipped)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
Of course they gave it to him. Twelve years I carried that department, and they hand it to the one who learned the org chart before he learned the work.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
Of course they gave it to him. Twelve years I carried that department, and they hand it to the one who learned the org chart before he learned the work.
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (contemplation, slow, soft, low-arousal)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
I've been sitting with it for a week now. And I still can't tell whether I'm at peace with it or whether I've just gotten tired of arguing with myself.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
I've been sitting with it for a week now. And I still can't tell whether I'm at peace with it or whether I've just gotten tired of arguing with myself.
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (contentment, warm, relaxed, smooth)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
The bread's still warm and nobody needs anything from me until Monday. I could sit right here in this exact chair until the light goes, and that would be a good day.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
The bread's still warm and nobody needs anything from me until Monday. I could sit right here in this exact chair until the light goes, and that would be a good day.
Complex passage — no inline direction caption onlyEnglish
I really thought this was the one that was going to work. I had it in my head already, how I'd tell everyone, and now I have to go back and unsay all of it.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
I really thought this was the one that was going to work. I had it in my head already, how I'd tell everyone, and now I have to go back and unsay all of it.
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (distress, tense, unsteady, pressed)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
I don't know what to do. I've gone through it four times and every road ends the same way. Tell me there's something I haven't thought of, please, because I'm out.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
I don't know what to do. I've gone through it four times and every road ends the same way. Tell me there's something I haven't thought of, please, because I'm out.
Complex passage — no inline direction caption onlyEnglish
It's fine. It doesn't really land one way or the other. People keep asking how I'm taking it and I don't have an answer, because there isn't a feeling there to report.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
It's fine. It doesn't really land one way or the other. People keep asking how I'm taking it and I don't have an answer, because there isn't a feeling there to report.
Complex passage — no inline direction caption onlyEnglish
This could actually work. If the second trial comes back anything like the first one, then everything we've been building toward for four years is suddenly next year, not someday.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
This could actually work. If the second trial comes back anything like the first one, then everything we've been building toward for four years is suddenly next year, not someday.
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (infatuation, warm, breathy, animated)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
I can't stop thinking about her, it's ridiculous. She said one thing about the rain on Tuesday and I've been turning it over ever since like it meant something.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
I can't stop thinking about her, it's ridiculous. She said one thing about the rain on Tuesday and I've been turning it over ever since like it meant something.
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (intoxication altered states of consciousness, slurred, loose, drifting)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
Everything's gone kind of soft at the edges. Is the room doing that or is it me, because I feel like I'm arriving a half second after I get here.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
Everything's gone kind of soft at the edges. Is the room doing that or is it me, because I feel like I'm arriving a half second after I get here.
inline: (intoxication altered states of consciousness, slurred, loose, drifting) · (loose laugh) mid-utterance
Base
SFT
+DPO LoRA
+DPO full
jealousy and envy
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (jealousy and envy, tight, sour-edged, controlled)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
Must be nice. Walking into a room where everyone already decided you belong, never once having to prove the thing the rest of us prove every single day.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
Must be nice. Walking into a room where everyone already decided you belong, never once having to prove the thing the rest of us prove every single day.
Short carrier — no inline direction caption onlyEnglish
So this is how it turned out in the end.
Base
SFT
+DPO LoRA
+DPO full
Short carrier — WITH inline direction inline cueEnglish
So this is how it turned out in the end.
inline: (longing, warm, drawn-out, falling)
Base
SFT
+DPO LoRA
+DPO full
Complex passage — no inline direction caption onlyEnglish
I'd give anything for one more afternoon in that kitchen. Just the noise of it, everyone talking over each other, and me not knowing yet that it was the last time.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
I'd give anything for one more afternoon in that kitchen. Just the noise of it, everyone talking over each other, and me not knowing yet that it was the last time.
Complex passage — no inline direction caption onlyEnglish
No, don't warn him. Let him walk in there with that smile on and find out the way the rest of us did, and I want to be standing where I can see his face.
Base
SFT
+DPO LoRA
+DPO full
Complex passage — WITH inline directions inline cueEnglish
No, don't warn him. Let him walk in there with that smile on and find out the way the rest of us did, and I want to be standing where I can see his face.
Sampling: do_sample=True, temperature=1.0, top_p=0.95, top_k=50,
identical for base and SFT, seeded per sample. Ground truth is decoded from the stored
MOSS-Audio-Tokenizer-v2 codes of the held-out clip, so it carries the same codec loss as the
generated tracks and the comparison is fair. Validation loss measures next-token likelihood, not
audio quality — that is what these players are for.
Model: laion/moss-tts-local-transformer-4.55b-voice-acting-v2-sft.
CC-BY-4.0, Christoph Schuhmann and LAION.