Research Engineer

Commotion; New York (NY); Hybrid; USD 125,000 - 190,000; Full time

Job summary

Commotion is seeking an AI/ML Engineer (Research) in New York to push the boundaries of speech-driven AI. You will work on LLM-backed speech systems, focusing on direct speech-to-speech modelling, VAD, and expressive speech synthesis within an autonomous research framework.

You will collaborate with the Research Director to shape long-term roadmaps, explore ART and memory-augmented architectures, and contribute to SOTA multimodal research.

Qualifications

• 2+ years of applied or academic experience in speech, multimodal, or LLM research.
• Bachelor's or Master's in Computer Science, AI, or a related field.
• Strong in Python and scientific computing; experience in JupyterHub environments.

Responsibilities

• Research and develop direct speech-to-speech modelling pipelines leveraging LLM backbones and audio encoders/decoders.
• Model and evaluate turn-taking, latency handling, and voice activity detection (VAD) for real-time conversational AI.
• Explore Agentic Reinforcement Training (ART) and self-learning loops for continual improvement.

Skills

LLM research

Speech research

Python programming

Experiment design

Education

Bachelor's or Master's in CS/AI

Tools

PyTorch

HuggingFace

LangChain

Job description

We're looking for an AI/ML Engineer (Research) with a strong background in LLM fine-tuning, Audio Language models and speech-to-speech pipelines. This is a pure research role focused on exploring new architectures, training methodologies, and agentic learning behaviours. You'll work on LLM-backed speech systems that can handle full conversational turn-taking, perform voice activity detection (VAD), maintain natural prosody, and drive end-to-end speech generation with minimal text intermediaries. Your work will help shape the cognitive and auditory intelligence of our AI Workers, making them expressive, contextually aware, and capable of continual self-improvement.

Responsibilities:

• Research and develop direct speech-to-speech modelling pipelines leveraging LLM backbones (e. g., Qwen) and audio encoders/decoders (e. g., Whisper).
• Model and evaluate turn-taking, latency handling, and voice activity detection (VAD) mechanisms for real-time conversational AI.
• Explore Agentic Reinforcement Training (ART) and self-learning loops for continual improvement of speech models.
• Design and experiment with memory-augmented multimodal architectures that allow persistent recall and contextual understanding across interactions.
• Develop novel approaches for expressive speech generation, emotion conditioning, and speaker identity preservation.
• Collaborate with the Research Director to define and drive long-term AI research roadmaps around autonomy, speech cognition, and agentic intelligence.
• Conduct internal benchmarks and contribute to SOTA research in multimodal learning, audio-language alignment, and agent reasoning.
• Work closely with MLEs for model training and evaluation, while focusing entirely on research design, datasets, and experimentation.

Requirements:

• Experience: 2+ years of applied or academic experience in speech, multimodal, or LLM research.
• Education: Bachelor's or Master's in Computer Science, AI, or a related field.
• Programming: Strong in Python and scientific computing; experience in JupyterHub environments.

Core Skills:

• Deep understanding of LLM architectures, transformers, and multimodal embeddings.
• Experience with speech modelling pipelines: ASR, TTS, speech-to-speech, or audio-language models.
• Understanding of turn-taking systems, VAD, prosody modelling, and real-time voice synthesis.
• Familiarity with self-supervised learning, contrastive representation learning, and agentic reinforcement (ART).
• Strong background in dataset curation, experimental design, and model evaluation.

Tools and Ecosystem:

• Comfortable using Agno, Pipecat, HuggingFace, and Pytorch.
• Familiarity with LangChain, vector stores, and memory systems for agentic research.
• Excellent written communication and ability to clearly articulate research insights.

Mindset and Traits:

• Deep research curiosity, constantly exploring new methods beyond published papers.
• Strong ownership mindset: design, experiment, analyse, iterate independently.
• Excited about building cognitive speech systems, not just fine-tuning existing ones.
• Commitment to mission: passionate about shaping the future of autonomous AI Workers.