Machine Learning Engineer | Applied AI, LLMs & Compliance

SourceIn

New York (NY)

On-site

USD 170,000 - 200,000

Full time

36 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

SourceIn is seeking a hands-on Machine Learning Engineer to build the intelligence and compliance layer of an early-stage AI startup. You will work with the CTO and engineering team to develop models, evaluation frameworks, and production ML capabilities powering AI monitoring and assurance.

This role emphasizes production readiness, model evaluation, and handling real-world data to improve accuracy, reliability, and operational efficiency across voice, chat, email, and SMS interactions.

Qualifications

  • Applied ML experience building or maintaining ML/NLP/LLM-powered applications.
  • Strong production-quality Python coding and data pipelines.
  • Hands-on experience with modern LLMs, prompting, evaluation, and model integrations.
  • Understanding of model accuracy, precision, recall, ground-truth datasets, and evaluation methodologies.
  • Production ML experience deploying or maintaining ML-powered applications.
  • Familiarity with processing unstructured/noisy data and human-labelled information.
  • Experience with AWS or similar cloud environments.

Responsibilities

  • Develop AI-powered systems for compliance, hallucination detection, and scoring.
  • Create evaluation datasets, testing frameworks, and ground-truth datasets.
  • Build prompts, datasets, and workflows for customer-specific scoring.
  • Improve model accuracy, reliability, and confidence thresholds.
  • Develop production ML services and monitoring for latency and cost optimization.
  • Collaborate with CTO and engineering teams to evolve ML architecture.
  • Translate customer requirements and regulatory frameworks into measurable evaluation criteria.
  • Work on speech and conversation intelligence including ASR and diarisation.
  • Ensure measurable AI performance across customers.

Skills

Applied ML
Python
LLM experience
Model evaluation
ML fundamentals
Production ML
Data handling
AWS / cloud
Engineering mindset
Startup mindset
LLM-as-judge
Weak supervision
Regulatory compliance

Tools

ASR
Diarisation
Streaming transcription

Job description

San Francisco Bay Area or New York City Area

$170K–$200K USD base salary + equity

Visa sponsorship available for candidates already in the US who require a visa transfer

Pre-seed AI startup | Early engineering team

Build the intelligence layer that makes AI customer conversations accountable

Are you a hands-on Machine Learning Engineer who enjoys turning modern AI technologies into reliable, measurable, production-grade systems?

We're looking for a Machine Learning Engineer to help build the intelligence and compliance layer of an early-stage AI startup tackling one of the most important challenges in enterprise AI: ensuring customer-facing AI agents are accurate, safe, compliant, and accountable.

You’ll work closely with the CTO and engineering team, developing the models, evaluation frameworks, and production ML capabilities that power the company's AI monitoring and assurance platform.

This isn’t a research-only position. You’ll work with real-world data, evaluate model performance, build production solutions, and help solve challenging problems involving accuracy, reliability, and operational efficiency.

The opportunity

Enterprises are increasingly deploying AI agents to handle customer conversations across voice, chat, email, and SMS.

However, businesses need an independent way to measure their performance, identify failures, detect hallucinations, and demonstrate compliance.

The company is building an AI monitoring and assurance platform that evaluates customer interactions against applicable regulations and business requirements.

You’ll contribute to the intelligence that makes this possible — from compliance detection and hallucination monitoring to evaluation frameworks, scoring models, and production ML observability.

What you’ll work on

Compliance, hallucination, and scoring

Develop AI-powered systems to detect compliance issues, hallucinations, ungrounded answers, and interaction-quality problems.

Work with rules, classifiers, LLM-based judges, and other evaluation approaches.

Help improve model accuracy, reliability, and confidence thresholds.

Evaluation frameworks and data

Develop evaluation datasets and testing frameworks.

Work with human-labelled data and ground-truth datasets.

Build regression tests to identify model performance issues.

Explore synthetic data, weak supervision, and active learning where appropriate.

Help translate customer requirements and regulatory frameworks into measurable evaluation criteria.

Develop prompts, datasets, and workflows for customer-specific scoring.

Improve the consistency and reliability of AI evaluation across different customers.

Speech and conversation intelligence

Work with noisy transcripts and unstructured conversational data.

Explore NLP and LLM techniques for conversation analysis.

Contribute to speech-processing capabilities, including ASR, speaker diarisation, and streaming transcription.

Production ML and performance

Develop and maintain production ML services using Python.

Work with model inference, routing, latency, and cost optimisation.

Monitor model performance and identify potential regressions.

Help build reliable ML workflows and monitoring capabilities.

Collaboration and engineering

Work closely with the CTO and engineering team.

Collaborate with customer-facing teams to understand real-world AI performance challenges.

Contribute to technical discussions and the evolution of the ML architecture.

Help establish effective ML engineering and evaluation practices.

What we’re looking for

Applied ML experience: Practical experience developing or maintaining machine learning, NLP, or LLM-powered applications.

Strong Python skills: Comfortable writing production-quality Python code and working with data processing pipelines.

LLM experience: Hands-on experience with modern LLMs, prompting, evaluation, retrieval, and model integrations.

Model evaluation: Understanding of model accuracy, precision, recall, ground-truth datasets, and evaluation methodologies.

Machine learning fundamentals: Knowledge of classification, NLP, model selection, and practical ML techniques.

Production experience: Some experience deploying or maintaining ML-powered applications in production environments.

Data handling: Comfortable working with unstructured data, noisy datasets, and human-labelled information.

Cloud familiarity: Experience with AWS or similar cloud environments.

Practical engineering mindset: Able to recognise when a simple rule or traditional ML approach is more appropriate than an LLM.

Startup mindset: Comfortable working in a small engineering team where flexibility, initiative, and continuous learning are important.

AI-assisted development: Familiarity with coding agents and AI-assisted development tools is beneficial.

Production speech and audio systems, including ASR, diarisation, and streaming transcription.

LLM-as-judge systems and evaluation against human ground truth.

Weak supervision, programmatic labelling, or active learning.

Trust and safety, fraud detection, risk management, or regulatory compliance.

Contact-centre quality assurance or workforce optimisation.

Why consider this opportunity?

Join an early-stage AI company working on a meaningful enterprise problem.

Work directly with the CTO and engineering team.

Gain exposure to real-world applied AI and LLM challenges.

Build production ML solutions rather than focusing exclusively on research.

Contribute to the evolution of an AI platform from its early stages.

Work on problems involving model reliability, measurable accuracy, and AI accountability.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Founding ML Engineer - Applied AI, LLMs & Compliance
Founding ML Engineer - Applied AI, LLMs & Compliance

SourceIn • San Francisco (CA)

Hybrid
USD 180,000 - 250,000
Founding ML Engineer Senior Level $200,000-$250,000 View →
Founding ML Engineer Senior Level $200,000-$250,000 View →

Nguyening Strategies Group • San Francisco (CA)

Remote
USD 200,000 - 250,000
Founding Machine Learning Engineer
Founding Machine Learning Engineer

Strativ Group • San Francisco (CA)

On-site
USD 150,000 - 190,000
AI Engineer, Multimodal LLMs
AI Engineer, Multimodal LLMs

Eloquent AI • San Francisco (CA)

On-site
USD 150,000 - 210,000
Staff AI Engineer
Staff AI Engineer

Meeru AI Inc • Pleasanton (CA)

On-site
USD 180,000 - 260,000
Staff AI Engineer
Staff AI Engineer

Harnham • San Francisco (CA)

On-site
USD 180,000 - 240,000
Application Engineer – LLM
Application Engineer – LLM

Salt • United States

On-site
USD 120,000 - 180,000
Platform Engineer | AI, Distributed Systems & SaaS
Platform Engineer | AI, Distributed Systems & SaaS

SourceIn • New York (NY)

Hybrid
USD 160,000 - 190,000
Equity
Visa sponsorship
Machine Learning Engineer, Applied AI
Machine Learning Engineer, Applied AI

Brain Co. • San Francisco (CA)

On-site
USD 140,000 - 210,000
Senior Machine Learning Engineer
Senior Machine Learning Engineer

Arionkoder • United States

Remote
USD 150,000 - 230,000
Monthly USD Service Fee
Paid Agreed Time Off
Caregiver Time Off
+2