Remote MTS: End-to-End ML Evaluation & Data Systems

Pro Integrate LLC; New York (NY); Remote; USD 240,000 - 350,000; Full time

Benefits offered by this job

Equity compensation

Health insurance reimbursement

401(k) with company match

Paid time off

Remote-first workforce

Job summary

Pro Integrate LLC is seeking a Member of Technical Staff (MTS) to serve as a technical owner at the intersection of research, data, and real-world AI systems. This hands-on role emphasizes rigorous evaluation, failure analysis, and iterative development to improve deployed systems.

You'll collaborate with researchers, domain experts, and operators to translate experimental work into defensible signals and measurable performance gains, while acting as a quality gate and driving end-to-end

Qualifications

• Strong judgment around research signal quality and externalization readiness.
• Experience designing ML datasets, evaluation frameworks, and QA processes.
• Ability to translate messy, real-world system behavior into structured research and evaluation opportunities.
• Clear written and verbal communication for technical and non-technical stakeholders.

Responsibilities

• Own research and evaluation initiatives end-to-end: problem framing, data design, quality calibration, and signal validation.
• Design ML-oriented data systems, including task definitions, annotation schemas, rubrics, incentives, and pipelines optimized for downstream model performance.
• Analyze model and system failures to identify root causes, edge cases, and opportunities for improvement.
• Translate ambiguous, real-world behavior into structured evaluation frameworks and new data categories.
• Work closely with researchers and domain experts to calibrate quality early and continuously raise the signal bar.
• Iterate rapidly on evaluations, datasets, and feedback loops to improve system performance.
• Act as a quality gate: block claims, pause work, or force scope changes when signal strength or data integrity is insufficient.
• Partner with cross-functional and client-facing teams to translate research progress into clear, credible narratives grounded in evidence.

Skills

Research signal quality

ML evaluation design

Data QA processes

System-level mindset

Stakeholder communication

Tools

RL environments

Job description

Pro Integrate LLC is seeking a Member of Technical Staff (MTS) to serve as a technical owner at the intersection of research, data, and real-world AI systems. This hands-on role emphasizes rigorous evaluation, failure analysis, and iterative development to improve deployed systems.

You'll collaborate with researchers, domain experts, and operators to translate experimental work into defensible signals and measurable performance gains, while acting as a quality gate and driving end-to-end