Applied AI Engineer

Sarah Smith Fund; San Francisco, Northern (CA, KY); Hybrid; USD 150,000 - 200,000; Full time

Benefits offered by this job

Remote-first culture

Direct access to founders

Professional development budget

Job summary

Soulside AI in San Francisco (US On-Site) is seeking an Applied AI Engineer to own the model layer that makes our clinical documentation trustworthy. You will fine-tune open‑source models, stand up the infrastructure to serve them, and build evaluation sets that show measurable improvements in quality and safety.

This hands‑on role sits at the intersection of applied ML and product, requiring collaboration with clinical teams and rapid iteration.

Qualifications

• 3+ years in applied ML / AI engineering, or a Master's degree in a related field, with hands-on experience taking LLM-based systems into production.
• Practical experience with post-training / fine-tuning open-source models (e.g., Llama, Qwen, Mistral) using SFT, LoRA/PEFT, or preference-based methods.
• Experience serving or fine-tuning models on managed platforms such as Fireworks AI, Baseten, or Together AI (or comparable inference/training infra).
• Demonstrated ability to build evaluation frameworks for LLM tasks with measurable quality.
• Strong Python and familiarity with PyTorch and Hugging Face.
• Solid grounding in prompt engineering and structured-output validation.

Responsibilities

• Build post-training pipelines on open-source models—supervised fine-tuning, preference optimization, LoRA/adapters, and distillation—for domain-specific clinical tasks.
• Fine-tune, deploy, and serve models across managed inference and fine-tuning platforms such as Fireworks AI, Baseten, and Together AI.
• Design and maintain rigorous evaluation sets for high-stakes tasks like clinical reasoning and AI note generation.
• Turn eval results into a fast, trustworthy iteration loop and quantify model changes.
• Optimize the full LLM pipeline—prompting, retrieval, validation, latency, and cost.
• Collaborate with clinical experts to translate documentation and compliance requirements into model behavior and eval criteria.
• Monitor models in production for quality, drift, and failure modes.

Skills

Applied ML experience

Python

LLM deployment

Remote startup vibe

Education

Master's degree in a related field

Tools

PyTorch

Hugging Face

Fireworks AI

Baseten

Together AI

Job description

Applied AI Engineer

Soulside AI · US On-Site · Reports to the CTO

About Soulside

Soulside AI is the specialist AI platform for behavioral health documentation and compliance. We generate audit-ready clinical documentation across individual and group sessions, virtual and in-person care, admissions, and treatment planning—and we embed real-time chart audits and payer-aligned compliance checks into everyday workflows. The result is immediate and measurable: higher-quality charts, stronger medical necessity, and hours given back to clinicians every week.

We're backed by Counterpart Ventures, GreyMatter Capital, and One Mind, and we're a UCSF Rosenman Institute and One Mind Accelerator company. We've reached strong product-market fit and are scaling fast.

The Role

We're looking for an Applied AI Engineer to own the model layer that makes Soulside's documentation trustworthy. In behavioral health, a note isn't just text—it has to be clinically sound, defensible for medical necessity, and safe. Your job is to build the post-training pipelines and evaluation systems that get our models there, and keep them there as we scale.

This is a hands-on role for someone who lives at the intersection of applied ML and product. You'll fine-tune and adapt open-source models, stand up the infrastructure to serve them, and build the rigorous evaluation sets that tell us - objectively - whether a change made the product better or worse.

Why This Role Matters

• Accuracy Isn't Optional: In behavioral health, a wrong or unsupported note has real clinical and financial consequences. The pipelines and evals you build are what let us ship model changes with confidence.

• Own the Model Layer: You'll define how we post-train, evaluate, and deploy models end-to-end—not inherit someone else's stack.

• Direct Clinical Impact: Every improvement in clinical reasoning or note quality directly reduces documentation burden and strengthens the charts clinicians and payers rely on.

What You'll Do

• Build post-training pipelines on open-source models—supervised fine-tuning, preference optimization (DPO/RLHF), LoRA/adapters, and distillation—for domain-specific clinical tasks.

• Fine-tune, deploy, and serve models across managed inference and fine-tuning platforms such as Fireworks AI, Baseten, and Together AI, and make pragmatic build-vs-buy calls on where each workload should run.

• Design and maintain rigorous evaluation sets for high-stakes tasks like clinical reasoning and AI note generation—defining metrics, curating gold-standard data, and building automated and human-in-the-loop eval harnesses.

• Turn eval results into a fast, trustworthy iteration loop: catch regressions before they ship, and quantify the impact of every model or prompt change.

• Optimize the full LLM pipeline—prompting, retrieval, structured output validation, latency, and cost.

• Partner with clinical experts to translate documentation and compliance requirements into model behavior and evaluation criteria.

• Monitor models in production for quality, drift, and failure modes, and close the loop back into training data and evals.

What We're Looking For

• 3+ years in applied ML / AI engineering, or a Master's degree in a related field, with hands-on experience taking LLM-based systems into production.

• Practical experience with post-training / fine-tuning open-source models (e.g., Llama, Qwen, Mistral) using SFT, LoRA/PEFT, or preference-based methods.

• Experience serving or fine-tuning models on managed platforms such as Fireworks AI, Baseten, or Together AI (or comparable inference/training infra).

• Demonstrated ability to build evaluation frameworks for LLM tasks—you think in terms of measurable quality, not vibes.

• Strong Python and familiarity with the modern ML tooling ecosystem (PyTorch, Hugging Face, etc.).

• Solid grounding in prompt engineering and structured-output validation.

• Ability to thrive in a fast-paced, remote startup and communicate clearly with technical and clinical teammates.

• We're willing to sponsor visas, including H-1B and O-1, for the right candidate.

Bonus Points

• Experience with healthcare, clinical NLP, or other high-stakes / regulated domains.

• Familiarity with HIPAA and handling sensitive clinical data.

• RAG systems, retrieval quality tuning, or long-context document workflows.

• Experience with LLM observability, monitoring, and drift detection in production.

• Data pipeline and labeling workflow experience for curating high-quality training and eval sets.

• Open-source contributions in the ML/LLM ecosystem.

What We Offer

• Salary range of $150,000-$200,000, plus equity with significant upside potential as a founding team member

• Comprehensive health, dental, and vision insurance

• Flexible, remote-first culture

• Direct access to founders and influence on technical direction

• Professional development budget and conference attendance

• The chance to build AI that measurably improves mental health care at scale