Automatic Music Transcription (AMT) converts an audio recording into symbolic notes, usually MIDI. Single-instrument transcription already works reasonably well. However, transcribing a full multi-instrument mix stays difficult. Kyutai and Mirelo team now release MuScriptor to close that gap. It is an open-weight model trained on real, multi-instrument recordings across many genres.
This article explains how MuScriptor works, what the benchmarks show, and how to run it.
What is MuScriptor?
At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a short audio segment. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In effect, transcription becomes a language-modeling task, following the MT3 tokenization scheme.
The release ships three weight variants on Hugging Face. Their sizes are small (103M), medium (307M, default), and large (1.4B). The inference code uses the MIT license. The weights use CC BY-NC 4.0, so commercial use is restricted.
How the Three-Stage Pipeline Works
MuScriptor’s main idea is data, not architecture. Accordingly, training moves through three stages, and each builds on the last.
Pre-training uses D<sub>Synth</sub>, roughly 1.45M MIDI files. An on-the-fly pipeline synthesizes them during training. Augmentations include pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations.
Fine-tuning uses D<sub>Real</sub>, an internal set of 170,000 recordings. Together they total more than 11,000 hours with aligned note annotations. Most alignments come from audio-symbolic synchronization using interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a maximum time-dilation factor.
Reinforcement learning post-training uses D<sub>RL</sub>, 300 manually verified tracks. The team applies a GRPO-like method combining REINFORCE with group-relative advantage normalization. The reward sums three F-scores: onset, frame, and offset. As a result, the model learns to favor cleaner transcriptions.
Transcribe</button>
</div>
</div>
<div class=”rollwrap”>
<svg class=”roll” id=”roll” viewBox=”0 0 720 300″ preserveAspectRatio=”xMidYMid meet” role=”img” aria-label=”Piano roll transcription”></svg>
</div>
<div class=”legend”>
<span><i style=”background:#2563EB”></i>True positive (detected)</span>
<span><i style=”background:#16A34A”></i>False negative (missed)</span>
<span><i style=”background:#DC2626″></i>False positive (wrong)</span>
</div>
<div class=”grid”>
<div class=”metrics”>
<div class=”metric”>
<small>Onset F1</small>
<div class=”val” id=”mOnset”>—</div>
<div class=”baseline” id=”bOnset”></div>
<div class=”metricbar”><i id=”barOnset”></i></div>
</div>
<div class=”metric”>
<small>Multi F1</small>
<div class=”val” id=”mMulti”>—</div>
<div class=”baseline” id=”bMulti”></div>
<div class=”metricbar”><i id=”barMulti”></i></div>
</div>
</div>
<div class=”stream” id=”stream”>
<div class=”h”>Event stream · model.transcribe()</div>
<div class=”ln” style=”opacity:.6″>Press Transcribe to stream note events…</div>
</div>
</div>
<p class=”note” id=”stageNote”></p>
</div>
<div class=”foot”>
<span>Note pattern is illustrative. F1 scores are real, from the MuScriptor paper (1.3B model on D<sub>Test</sub>).</span>
<span>Built by <b>Marktechpost</b></span>
</div>
</div>
<script>
(function(){
var SVG=”http://www.w3.org/2000/svg”;
var roll=document.getElementById(“roll”);
var W=720,H=300,PADL=64,PADR=14,PADT=14,PADB=26;
var DUR=10; // seconds shown
var rows=[
{name:”Drums”,inst:”drums”,y:0},
{name:”E-Bass”,inst:”e_bass”,y:1},
{name:”Dist. E-Gtr”,inst:”distorted_e_guitar”,y:2},
{name:”Ac. Piano”,inst:”acoustic_piano”,y:3},
{name:”Voice”,inst:”voice”,y:4},
{name:”Strings”,inst:”string_ensemble”,y:5}
];
var NR=rows.length;
var rowH=(H-PADT-PADB)/NR;
function tx(t){return PADL+(t/DUR)*(W-PADL-PADR);}
function ry(r){return PADT+r*rowH;}
// Illustrative ground-truth notes: [row, start, dur, rank(0=easy..1=hard)]
var GT=[
[0,0.3,0.18,0.1],[0,0.8,0.18,0.15],[0,1.3,0.18,0.2],[0,1.8,0.18,0.1],[0,2.4,0.18,0.25],[0,3.0,0.18,0.2],[0,3.6,0.18,0.3],[0,4.2,0.18,0.2],[0,4.9,0.18,0.4],[0,5.5,0.18,0.3],[0,6.2,0.18,0.5],[0,6.9,0.18,0.4],[0,7.6,0.18,0.6],[0,8.3,0.18,0.5],[0,9.0,0.18,0.7],
[1,0.4,0.8,0.2],[1,1.6,0.8,0.25],[1,3.0,0.9,0.35],[1,4.4,0.8,0.4],[1,5.8,0.9,0.5],[1,7.2,0.9,0.6],[1,8.6,0.9,0.7],
[2,1.0,1.1,0.45],[2,2.6,1.0,0.55],[2,4.2,1.2,0.6],[2,6.0,1.1,0.7],[2,7.8,1.1,0.8],
[3,0.6,0.7,0.3],[3,1.7,0.6,0.35],[3,2.9,0.7,0.4],[3,4.1,0.6,0.5],[3,5.3,0.7,0.55],[3,6.6,0.6,0.65],[3,8.0,0.8,0.75],
[4,2.0,1.4,0.5],[4,4.0,1.5,0.6],[4,6.3,1.4,0.72],[4,8.4,1.2,0.85],
[5,1.2,1.8,0.55],[5,3.6,1.9,0.68],[5,6.0,1.9,0.8],[5,8.2,1.6,0.9]
];
// Illustrative false positives available: [row,start,dur,rank]
var FPS=[
[0,2.1,0.16,0.9],[0,5.9,0.16,0.7],[0,7.9,0.16,0.85],
[1,2.4,0.6,0.8],[1,6.6,0.7,0.9],
[2,3.7,0.9,0.85],[2,8.9,0.9,0.95],
[3,3.5,0.5,0.9],[3,7.3,0.6,0.8],
[4,3.5,0.9,0.9],[4,7.7,0.8,0.95],
[5,5.1,1.2,0.88],[5,9.2,0.9,0.97]
];
// Real numbers from the paper (1.3B, DTest). recall/fpRate are illustrative thresholds.
var STAGES={
synth:{onset:34.5,multi:16.2,onBase:26.1,recall:0.42,fp:0.85,
note:”Trained only on synthetic MIDI. It finds coarse pitch activity, but misses many onsets and mislabels instruments.”},
real:{onset:54.4,multi:41.6,onBase:52.5,recall:0.78,fp:0.35,
note:”Fine-tuning on 170k real recordings lifts every metric by roughly 20 points over synthetic-only training.”},
rl:{onset:60.4,multi:48.2,onBase:60.4,recall:0.90,fp:0.12,
note:”GRPO-style RL post-training on 300 verified tracks reduces false negatives and sharpens onset timing.”}
};
var current=”rl”, cond=true, playing=false, playhead=0, raf=null, streamed={};
function selectedRows(){
if(!cond) return rows.map(function(r){return r.y;});
// conditioning example: focus on the core rhythm section + piano
return [0,1,2,3];
}
function classify(stage){
var s=STAGES[stage];
var tp=[],fn=[],fp=[];
var sel=selectedRows();
GT.forEach(function(n){
if(cond && sel.indexOf(n[0])===-1) return; // hidden by conditioning
var boost=cond?0.08:0; // conditioning nudges recall up (illustrative)
if(n[3] <= s.recall+boost) tp.push(n); else fn.push(n);
});
FPS.forEach(function(n){
if(cond && sel.indexOf(n[0])===-1) return;
var fpr=cond?s.fp*0.7:s.fp; // conditioning trims spurious notes
if(n[3] > (1-fpr)) fp.push(n);
});
return {tp:tp,fn:fn,fp:fp};
}
function el(tag,attrs){var e=document.createElementNS(SVG,tag);for(var k in attrs)e.setAttribute(k,attrs[k]);return e;}
function drawGrid(){
while(roll.firstChild) roll.removeChild(roll.firstChild);
var sel=selectedRows();
for(var r=0;r<NR;r++){
var active=sel.indexOf(r)!==-1;
roll.appendChild(el(“rect”,{x:PADL,y:ry(r),width:W-PADL-PADR,height:rowH,
fill:(r%2? “#FAFBFD”:”#FFFFFF”),opacity:active?1:0.4}));
roll.appendChild(el(“line”,{x1:PADL,y1:ry(r),x2:W-PADR,y2:ry(r),stroke:”#EFF1F5″,”stroke-width”:1}));
var lbl=el(“text”,{x:PADL-8,y:ry(r)+rowH/2+4,”text-anchor”:”end”,”font-size”:11,
fill:active?”#14161C”:”#B6BCC7″,”font-weight”:active?600:500,”font-family”:”Inter,sans-serif”});
lbl.textContent=rows[r].name; roll.appendChild(lbl);
}
// time gridlines every 2s (segment markers)
for(var t=0;t<=DUR;t+=2){
roll.appendChild(el(“line”,{x1:tx(t),y1:PADT,x2:tx(t),y2:H-PADB,stroke:”#E7E9EE”,”stroke-width”:1,”stroke-dasharray”:”2 4″}));
var tl=el(“text”,{x:tx(t),y:H-9,”text-anchor”:”middle”,”font-size”:10,fill:”#9AA1AE”,”font-family”:”Inter,sans-serif”});
tl.textContent=t+”s”; roll.appendChild(tl);
}
roll.appendChild(el(“line”,{x1:PADL,y1:H-PADB,x2:W-PADR,y2:H-PADB,stroke:”#E7E9EE”,”stroke-width”:1}));
}
var noteLayer,headLine;
function render(reveal){
drawGrid();
var c=classify(current);
noteLayer=el(“g”,{}); roll.appendChild(noteLayer);
function bar(n,color,opacity){
var x=tx(n[1]),w=Math.max(6,tx(n[1]+n[2])-tx(n[1]));
var y=ry(n[0])+rowH*0.28,h=rowH*0.44;
var vis=(!reveal)|| (n[1]<=playhead);
var rect=el(“rect”,{x:x,y:y,width:w,height:h,rx:3,fill:color,opacity:vis?opacity:0});
rect.style.transition=”opacity .18s”;
noteLayer.appendChild(rect);
}
c.fn.forEach(function(n){bar(n,”#16A34A”,0.55);});
c.fp.forEach(function(n){bar(n,”#DC2626″,0.8);});
c.tp.forEach(function(n){bar(n,”#2563EB”,0.95);});
headLine=el(“line”,{x1:tx(playhead),y1:PADT,x2:tx(playhead),y2:H-PADB,stroke:”#5B9A00″,”stroke-width”:2,opacity:reveal?0.9:0});
roll.appendChild(headLine);
postSize();
}
function setMetrics(){
var s=STAGES[current];
animateNum(“mOnset”,s.onset); animateNum(“mMulti”,s.multi);
document.getElementById(“bOnset”).textContent=”baseline YourMT3+ 32.5″;
document.getElementById(“bMulti”).textContent=”baseline YourMT3+ 21.9″;
document.getElementById(“barOnset”).style.width=s.onset+”%”;
document.getElementById(“barMulti”).style.width=(s.multi/60*100)+”%”;
document.getElementById(“stageNote”).textContent=s.note;
}
function animateNum(id,to){
var e=document.getElementById(id),from=parseFloat(e.textContent)||0,st=null,dur=550;
function step(ts){if(!st)st=ts;var p=Math.min(1,(ts-st)/dur);
e.textContent=(from+(to-from)*(1-Math.pow(1-p,3))).toFixed(1);
if(p<1)requestAnimationFrame(step);}
requestAnimationFrame(step);
}
// event stream
var stream=document.getElementById(“stream”);
function resetStream(){stream.innerHTML='<div class=”h”>Event stream · model.transcribe()</div>’;streamed={};}
function pushEvent(n,kind){
var pitchNames=[“C2″,”G2″,”E3″,”C4″,”A4″,”D5”];
var line=document.createElement(“div”);line.className=”ln”;
var p=pitchNames[n[0]]||”C4″;var inst=rows[n[0]].inst;
if(kind===”start”){
line.innerHTML='<span class=”st”>NoteStart</span> t=<span class=”pi”>’+n[1].toFixed(2)+’s</span> pitch=<span class=”pi”>’+p+'</span> inst=<span class=”in”>’+inst+'</span>’;
}else{
line.innerHTML='<span class=”en”>NoteEnd</span> t=<span class=”pi”>’+(n[1]+n[2]).toFixed(2)+’s</span> (‘+inst+’)’;
}
stream.appendChild(line);stream.scrollTop=stream.scrollHeight;
}
function play(){
if(playing)return; playing=true;
var btn=document.getElementById(“playBtn”);btn.disabled=true;btn.textContent=”● Transcribing…”;
playhead=0; resetStream();
var c=classify(current);
var visible=c.tp.concat(c.fp).sort(function(a,b){return a[1]-b[1];});
render(true);
var t0=null,SPEED=DUR/4200; // ms mapping
function frame(ts){
if(!t0)t0=ts;
playhead=Math.min(DUR,(ts-t0)*SPEED);
// reveal notes + emit events
visible.forEach(function(n,i){
var key=”s”+i;
if(n[1]<=playhead && !streamed[key]){streamed[key]=1;
var rects=noteLayer.querySelectorAll(“rect”);
pushEvent(n,”start”);
setTimeout((function(nn){return function(){pushEvent(nn,”end”);};})(n),120);
}
});
// update opacities
Array.prototype.forEach.call(noteLayer.querySelectorAll(“rect”),function(r){});
render(true);
if(playhead<DUR){raf=requestAnimationFrame(frame);}
else{playing=false;btn.disabled=false;btn.textContent=” Transcribe again”;
headLine.setAttribute(“opacity”,”0″);postSize();}
}
raf=requestAnimationFrame(frame);
}
// controls
document.getElementById(“stages”).addEventListener(“click”,function(e){
var b=e.target.closest(“button”);if(!b)return;
Array.prototype.forEach.call(this.children,function(c){c.classList.remove(“on”);});
b.classList.add(“on”);current=b.getAttribute(“data-s”);
playhead=DUR;render(false);setMetrics();postSize();
});
document.getElementById(“condToggle”).addEventListener(“click”,function(){
cond=!cond;this.classList.toggle(“on”,cond);
playhead=DUR;render(false);postSize();
});
document.getElementById(“playBtn”).addEventListener(“click”,play);
// pipeline shimmer
var pnodes=document.querySelectorAll(“#pipe .node”),pi=0;
setInterval(function(){
pnodes.forEach(function(n){n.classList.remove(“hot”);});
pnodes[pi].classList.add(“hot”);pi=(pi+1)%pnodes.length;
},900);
// resize to parent (WordPress embed)
function postSize(){
try{var h=document.body.offsetHeight+40;
window.parent.postMessage({muscriptorHeight:h},”*”);}catch(e){}
}
window.addEventListener(“resize”,function(){render(false);postSize();});
// init
playhead=DUR;render(false);setMetrics();setTimeout(postSize,60);setTimeout(postSize,400);
})();
</script>
</body>
</html>
“>
Performance
For evaluation, the research team use D<sub>Test</sub>, 372 held-out tracks with accurate annotations. They report instrument-agnostic metrics from the mir_eval library. Among them, Multi F1 is strictest, since it also requires the correct instrument.
The table below traces each training stage against the YourMT3+ baseline, using the large (~1.3B) model.
Model (D<sub>Test</sub>)Onset F1Frame F1Offset F1Drums F1Multi F1YourMT3+ (baseline)32.545.517.841.421.9MuScriptor · D<sub>Synth</sub>34.548.916.121.016.2MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub>54.469.342.343.341.6MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> + D<sub>RL</sub>60.473.349.050.248.2
Clearly, every stage improves results, and real data matters most. Synthetic-only training reaches competitive frame F1 but weak onset and multi scores. Adding D<sub>Real</sub> then lifts all metrics by roughly 20 points. Finally, RL post-training reduces false negatives and sharpens onset timing.
Cross-dataset tests point the same way. For example, frame F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset stay lower on hard styles like chorals.
Getting Started
Installation takes one command, and inference streams note events directly.
# pip install muscriptor (or: uv add muscriptor)
from pathlib import Path
from muscriptor import TranscriptionModel
# Downloads the default “medium” variant (also accepts “small” / “large”)
model = TranscriptionModel.load_model()
# Stream note events; optionally condition on known instruments
for event in model.transcribe(“audio.wav”, instruments=[“acoustic_piano”, “drums”]):
print(event) # NoteStartEvent / NoteEndEvent / ProgressEvent
# Or write a MIDI file directly
Path(“out.mid”).write_bytes(model.transcribe_to_midi(“audio.wav”))
For the released models, keep cfg_coef at 1, since they are already RL post-trained. Additionally, uvx muscriptor serve launches a browser web UI with a live piano roll.
Use Cases with Examples
Because the output is standard MIDI, many workflows open up:
Producers can extract a MIDI bassline from a mix, then re-voice it in a DAW.
Musicologists can convert historical recordings into editable scores for analysis.
MIR researchers can feed transcriptions into chord or key recognition systems.
Educators can build practice tools showing a live piano roll during playback.
Developers can transcribe only drums by passing instrument conditioning.
Strengths and Weaknesses
Strengths:
Trained on 170k real recordings spanning classical to heavy metal.
Open weights plus MIT-licensed inference code, in three size variants.
Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on D<sub>Test</sub>.
Instrument conditioning customizes output and stabilizes cross-segment predictions.
A streaming API emits note events and MIDI, alongside a browser web UI.
Weaknesses:
Weights are CC BY-NC 4.0, so commercial deployment is restricted.
The tokenizer drops velocity and cannot represent overlapping same-pitch, same-instrument notes.
Onset and offset accuracy stay lower on chorals and similar styles.
The large model wants a GPU for practical speed.
The 5-second segment size limits long-range context and inference speed.
Check out the Paper, GitHub Repo and Model Weights. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI appeared first on MarkTechPost.