国家同步辐射光源II（NSLS-II）的语音光束线.pptx

1、Voice-Enabled Beamlines at NSLS-II,Shray Mathur,Esther Tsai,Kevin Yager,Goal,Make user operation easier and more efficient at the beamline using AI agents.,2,sam.xabs(15)sam.quick_align(),Move sample x to absolute 15 mm and use quick align.,High-level speech commands,Beamline executable code,Challen

2、ges,Out of the box pretrained Speech-to-Text(STT)models not familiar with beamline specific terminology(SAXS,WAXS,GISAXS,GIWAXS,etc)or commands.,3,Further Challenges,Solution:Fine-tuning,4,Data Requirements:Require high-quality audio-text pairs to learn effectively,Data Availability:Where do we find

3、 reliable beamline-specific audio-text pairs?,Solution:Utilize Text-to-Speech(TTS)models to generate synthetic audio from beamline proposal documents,Fine-tuning pipeline:TTS+LoRA,5,Proposal Documents,Pre-trained STT,Fine-tuned STT,TTS,LoRA,Chunk,TTS model,Synthetic Audio-Text Pairs,LoRA FT,Fine-tun

4、ed Model,6,Fine-tuned STT,This process uses feedback from in-situ SAXS/WAXS measurements,Pre-trained STT,This process uses feedback from in-situ sacks wax measurements,Key Takeaways,Fine-tuning pipeline is:SimpleEffectiveScalableRequires about 8-10 mins of audio-text pairs to teach STT model a new wordWork part of a larger project-Exocortex!,7,Yager,Kevin G.Towards a Science Exocortex.Digital Discovery(2024).,