Founding ML Engineer - Applied AI, LLMs & Compliance

SourceIn; San Francisco (CA); Hybrid; USD 180,000 - 250,000; Full time

Job summary

SourceIn is building an AI monitoring and assurance platform that evaluates customer interactions against regulations and business requirements. You’ll own the intelligence from compliance detection to production ML observability, shaping how enterprises measure and demonstrate accountability.

As a founding ML engineer, you will work closely with the CTO, guide evaluation practices, and mentor the engineering team while staying hands-on with Python and modern ML systems in a rapidly evolving

Qualifications

• Production ML experience with NLP/LLM systems in real-world settings.
• Strong evaluation pipelines and ground-truthing from noisy labels.
• Experience in early-stage startups with ambiguity and ownership.

Responsibilities

• Own compliance detection, hallucination monitoring, scoring architecture.
• Design evaluation frameworks and data flywheel.
• Build detectors for compliance, grounding, risk, and interaction quality.
• Collaborate with CTO and QA/compliance teams; mentor engineers.
• Ensure production observability and monitoring for regressions.

Skills

Python

LLMs

NLP

Machine Learning

AWS

Leadership

Evaluation pipelines

Compliance knowledge

Tools

LLM-based judges

Rules & classifiers

Job description

Founding ML Engineer | Applied AI, LLMs & Compliance

All United States United States - San Francisco United States - New York City 23 Oct

• San Francisco Bay Area or New York City Area
• Python | LLMs | NLP | Machine Learning | AWS
• Pre-seed AI startup | Founding engineering team

  Build the intelligence layer that makes AI customer conversations accountable

  Are you a Staff or Principal-level ML engineer who wants to turn cutting-edge AI into reliable, measurable, production-grade systems?

The opportunity

Enterprises are increasingly deploying AI agents to handle customer conversations across voice, chat, email, and SMS. These systems are already dealing with billing disputes, account changes, and other sensitive interactions.

But how can businesses be confident that their AI agents are following the rules, providing grounded answers, and handling customers appropriately?

Manual quality assurance cannot keep pace with millions of interactions. AI vendors focus on building capable agents, while enterprises need an independent way to measure their performance, identify failures, and demonstrate compliance.

The company is building an AI monitoring and assurance platform that evaluates customer interactions against applicable regulations and business requirements, helping enterprises detect issues early, reduce risk, and provide evidence of accountability.

You’ll own the intelligence that makes this possible – from compliance detection and hallucination monitoring to evaluation frameworks, scoring models, and production ML observability.

What you’ll own

Compliance, hallucination & scoring architecture

• Translate customer requirements and regulatory frameworks into measurable detectors for compliance, grounding, hallucinations, risk, escalation, and interaction quality.
• Choose the right combination of rules, classifiers, LLM-based judges, and fallback mechanisms.
• Establish confidence thresholds and promotion criteria based on explicit accuracy, latency, cost, and risk trade-offs.

Evaluation frameworks & the data flywheel

• Own annotation guidelines, ground‑truth datasets, adjudication processes, and offline and online evaluation.
• Build regression gates that prevent ungrounded outputs, unsupported compliance claims, and unmeasured accuracy from reaching production.
• Turn QA reviews, customer disputes, and human‑versus‑model disagreements into high‑quality training and evaluation data.
• Use synthetic data, weak supervision, and active learning where they demonstrably improve performance.

Customer rubric onboarding

• Build the tooling, prompts, datasets, and evaluation workflows needed to translate each customer’s framework into reliable production detectors.
• Reduce the time required to onboard new customers without compromising measurement quality.
• Develop reusable approaches while accommodating customer‑specific requirements.

Speech & conversation intelligence

• Make noisy transcripts reliably scorable using robust NLP and LLM techniques.
• Evaluate transcription quality and its impact on downstream compliance and scoring.
• Help design the next generation of speech‑processing capabilities, including automatic speech recognition (ASR), speaker diarisation, and streaming transcription.

Production ML economics & observability

• Own inference cost and latency per interaction, model routing, and the trade‑offs between model capability and operational efficiency.
• Monitor model drift and changes in compliance detection, grounding, hallucination detection, and scoring accuracy.
• Build production monitoring and alerting that surface regressions before they become customer problems.

Technical leadership & team building

• Partner directly with the CTO and design partners’ QA and compliance teams to understand real‑world model failures.
• Contribute to enterprise security and AI governance reviews.
• Establish the ML engineering standards, evaluation practices, and technical direction for future hires.
• Remain hands‑on while mentoring engineers and building the team.

What we’re looking for

• Staff or Principal‑level applied ML expertise: You’ve owned production ML, NLP, or LLM systems that other teams or customers depend on, and you’ve been accountable for their performance.
• Rigorous evaluation experience: You’ve built evaluation pipelines, developed ground truth from inconsistent human labels, and made explicit decisions about acceptable error rates.
• Early‑stage startup experience: You’ve shipped products in an environment defined by ambiguity, broad ownership, and limited supporting infrastructure. This is essential.
• Strong model‑selection judgement: You’re comfortable with prompting, retrieval, fine‑tuning, distillation, classical classifiers — and recognising when a simple rule or regex is the better solution.
• Hands‑on Python engineering: You’ve trained or debugged models yourself, built and maintained production services, and worked with messy, unstructured data.
• Production ML judgement: You understand inference economics, latency, model reliability, sensitive data handling, and the security requirements of enterprise customers.
• Technical leadership: You can set the direction for an ML function, establish high engineering standards, and recruit the engineers who will help scale it.
• Pragmatic AI‑assisted development: You’re comfortable using coding agents to accelerate development while maintaining rigorous standards for testing, security, and maintainability.

Bonus points

Experience in any of the following would be particularly valuable:

• Production speech and audio systems, including ASR, diarisation, streaming transcription, or real‑time latency optimisation.
• Custom NLP or scoring‑model development, including LLM‑as‑judge systems validated against human ground truth.
• Weak supervision, programmatic labelling, or active learning in production.
• Trust and safety, fraud and risk, contact‑centre quality assurance, regulatory compliance, or governance, risk and compliance (GRC).

Why consider this opportunity?

• Founding‑level impact: Work directly with the CTO and define the company’s applied ML strategy from the ground up.
• Real ownership: Take responsibility for the intelligence layer, evaluation methodology, and production performance.
• A meaningful problem: Help enterprises adopt AI customer support with greater confidence, transparency, and accountability.
• Real‑world ML challenges: Solve difficult problems involving noisy data, model reliability, measurable accuracy, and operational constraints.
• Build the team: Establish the technical standards and culture that future ML engineers will inherit.
• Shape the product: Work directly with design partners and use real customer feedback to guide the platform’s evolution.

Interested?

If you’re a hands‑on ML leader who enjoys solving difficult applied AI problems, building reliable systems from first principles, and taking ownership beyond the model itself, we’d love to hear from you.