ML Engineer – AI Accountability & Compliance (Equity & Visa)

SourceIn

New York (NY)

On-site

USD 170,000 - 200,000

Full time

5 days ago
Be an early applicant

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

SourceIn is seeking a hands-on Machine Learning Engineer to build the intelligence and compliance layer of an early-stage AI startup. You will work with the CTO and engineering team to develop models, evaluation frameworks, and production ML capabilities powering AI monitoring and assurance.

This role emphasizes production readiness, model evaluation, and handling real-world data to improve accuracy, reliability, and operational efficiency across voice, chat, email, and SMS interactions.

Qualifications

  • Applied ML experience building or maintaining ML/NLP/LLM-powered applications.
  • Strong production-quality Python coding and data pipelines.
  • Hands-on experience with modern LLMs, prompting, evaluation, and model integrations.
  • Understanding of model accuracy, precision, recall, ground-truth datasets, and evaluation methodologies.
  • Production ML experience deploying or maintaining ML-powered applications.
  • Familiarity with processing unstructured/noisy data and human-labelled information.
  • Experience with AWS or similar cloud environments.

Responsibilities

  • Develop AI-powered systems for compliance, hallucination detection, and scoring.
  • Create evaluation datasets, testing frameworks, and ground-truth datasets.
  • Build prompts, datasets, and workflows for customer-specific scoring.
  • Improve model accuracy, reliability, and confidence thresholds.
  • Develop production ML services and monitoring for latency and cost optimization.
  • Collaborate with CTO and engineering teams to evolve ML architecture.
  • Translate customer requirements and regulatory frameworks into measurable evaluation criteria.
  • Work on speech and conversation intelligence including ASR and diarisation.
  • Ensure measurable AI performance across customers.

Skills

Applied ML
Python
LLM experience
Model evaluation
ML fundamentals
Production ML
Data handling
AWS / cloud
Engineering mindset
Startup mindset
LLM-as-judge
Weak supervision
Regulatory compliance

Tools

ASR
Diarisation
Streaming transcription

Job description

SourceIn is seeking a hands-on Machine Learning Engineer to build the intelligence and compliance layer of an early-stage AI startup. You will work with the CTO and engineering team to develop models, evaluation frameworks, and production ML capabilities powering AI monitoring and assurance.

This role emphasizes production readiness, model evaluation, and handling real-world data to improve accuracy, reliability, and operational efficiency across voice, chat, email, and SMS interactions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Founding ML Engineer: AI Compliance & Trust Platform
Founding ML Engineer: AI Compliance & Trust Platform

SourceIn • San Francisco (CA)

Hybrid
USD 180,000 - 250,000
Machine Learning Engineer | Applied AI, LLMs & Compliance
Machine Learning Engineer | Applied AI, LLMs & Compliance

SourceIn • New York (NY)

On-site
USD 170,000 - 200,000
AI engineer
AI engineer

Quantum Technologies LLC • Atlanta (GA)

On-site
USD 140,000 - 190,000
AI Engineer
AI Engineer

Falcon Smart IT (FalconSmartIT) • Los Angeles (CA)

On-site
USD 190,000 - 260,000
Platform Engineer | AI, Distributed Systems & SaaS
Platform Engineer | AI, Distributed Systems & SaaS

SourceIn • New York (NY)

Hybrid
USD 160,000 - 190,000
Equity
Visa sponsorship
ML Engineer: NLP, Fraud Detection & Recommendations
ML Engineer: NLP, Fraud Detection & Recommendations

Validion • United States

Remote
USD 150,000 - 210,000
LLMs experience
Fraud detection background
Publications in ML/NLP conferences
+1
ML Engineering Manager, Advertiser Experience & AI
ML Engineering Manager, Advertiser Experience & AI

SNAP, Inc. • Palo Alto (CA)

On-site
USD 229,000 - 343,000
Founding ML Engineer - Applied AI, LLMs & Compliance
Founding ML Engineer - Applied AI, LLMs & Compliance

SourceIn • San Francisco (CA)

Hybrid
USD 180,000 - 250,000
Senior ML Engineer & Tech Lead — End-to-End AI Systems
Senior ML Engineer & Tech Lead — End-to-End AI Systems

Kount • Alpharetta (GA)

On-site
USD 140,000 - 190,000
Healthcare packages
401k matching
Paid time off
+1
ML Engineer
ML Engineer

Catalyst Labs • New Jersey

On-site
USD 100,000 - 150,000