AI / ML Engineer

Commotion; New York (NY); Hybrid; USD 120,000 - 190,000; Full time

Job summary

Commotion invites an AI/ML Engineer to advance LLM-backed speech systems, focusing on fine-tuning, audio-language models, and speech-to-speech pipelines. The role targets research into new architectures, training methodologies, and agentic learning behaviors, in a fast-paced environment in New York.

The candidate will work on turn-taking, VAD, and real-time voice synthesis, collaborating with the Research Director to chart long-term AI research roadmaps.

Qualifications

• 2+ years in speech, multimodal, or LLM research.
• Bachelor's or master's degree in CS/AI or related field.
• Strong Python and scientific computing skills; experience in JupyterHub environments.
• Deep understanding of LLM architectures, transformers, and multimodal embeddings.
• Experience with speech modeling pipelines (ASR/TTS/speech-to-speech).
• Knowledge of turn-taking, VAD, prosody modeling, and real-time voice synthesis.
• Familiarity with self-supervised learning, contrastive learning, and ART.
• Dataset curation, experimental design, and model evaluation.

Responsibilities

• Research and develop direct speech-to-speech pipelines using LLM backbones and audio encoders/decoders.
• Model and evaluate turn-taking, latency, and VAD for real-time conversational AI.
• Explore ART and self-learning loops for continual improvement of speech models.
• Design memory-augmented multimodal architectures for persistent recall and context understanding.
• Collaborate with Research Director to define long-term AI research roadmaps.

Skills

Python

LLM research

Speech modeling

Transformers

Multimodal embeddings

JupyterHub

Agentic Reinforcement

Data curation

Education

BSc/MS in CS/AI

Tools

Hugging Face

PyTorch

LangChain

Pipecat

Agno

Job description

We are looking for an AI/ML Engineer with a strong background in LLM fine-tuning, audio language models, and speech-to-speech pipelines. This is a research-focused role exploring new architectures, training methodologies, and agentic learning behaviors. You will work on LLM-backed speech systems to handle full conversational turn-taking, voice activity detection (VAD), and natural prosody.

Responsibilities:

• Research and develop direct speech-to-speech modeling pipelines using LLM backbones (e. g., Qwen) and audio encoders/decoders (e. g., Whisper).
• Model and evaluate turn-taking, latency handling, and VAD for real-time conversational AI.
• Explore Agentic Reinforcement Training (ART) and self-learning loops for continual improvement of speech models.
• Design memory-augmented multimodal architectures for persistent recall and contextual understanding.
• Collaborate with the Research Director to define long-term AI research roadmaps.

Requirements:

• 2+ years of applied or academic experience in speech, multimodal, or LLM research.
• Bachelor's or master's degree in computer science, AI, or a related field.
• Strong Python and scientific computing skills, with experience in JupyterHub environments.
• Deep understanding of LLM architectures, transformers, and multimodal embeddings.
• Experience with speech modeling pipelines, including ASR, TTS, speech-to-speech, or audio-language models.
• Understanding of turn-taking systems, VAD, prosody modeling, and real-time voice synthesis.
• Familiarity with self-supervised learning, contrastive representation learning, and agentic reinforcement training (ART).
• Strong background in dataset curation, experimental design, and model evaluation.
• Comfortable using Agno, Pipecat, Hugging Face, and PyTorch.
• Familiarity with LangChain, vector stores, and memory systems for agentic research.
• Excellent written communication skills and the ability to clearly articulate research insights.