Senior ML Evaluation Engineer - AI & Aerospace

VTI Aerospace; Seattle (WA); On-site; USD 140,000 - 200,000; Full time

Benefits offered by this job

Health, dental, and vision insurance

Retirement plan (401(k)

Paid time off and holidays

Professional development opportunities

Job summary

VTI Aerospace, headquartered with Seattle and Bozeman offices, seeks a Senior Software Engineer – Evaluation to design and implement rigorous evaluation systems for computer vision, ASR, and language models. You will define methodologies, track KPIs, and collaborate with ML/data teams to enhance data and model performance.

You will build Python pipelines for benchmarking, monitor model drift in production, and create dashboards to visualize results, shaping the reliability of AI tools in

Qualifications

• 5+ years of experience developing production software systems in Python.
• Strong experience designing and implementing data processing or analysis pipelines.
• Experience building systems for machine learning evaluation, experimentation, or benchmarking.
• Experience working with large datasets and building scalable data workflows.
• Familiarity with statistical methods, experimental design, and model performance analysis.
• Experience building automated pipelines using tools such as Airflow or similar orchestration frameworks.
• Experience working with evaluation or experiment tracking tools such as MLflow or similar systems.
• Experience designing data models and working with relational databases and time-series data.

Responsibilities

• Define and implement evaluation methodologies for computer vision, ASR, and language systems.
• Identify and track key performance indicators (KPIs) that measure system and model effectiveness.
• Perform dataset coverage analysis to understand strengths, gaps, and biases in training and evaluation data.
• Identify model deficiencies and collaborate with ML engineers to improve training data and model performance.
• Build scalable model evaluation pipelines in Python for automated benchmarking and regression testing.
• Design and maintain systems for monitoring model performance and drift in production environments.
• Architect data models and storage systems for evaluation results using relational and time-series databases.
• Build dashboards and reporting tools to visualize model performance and evaluation metrics.
• Collaborate with ML and data engineering teams to improve evaluation workflows and data pipelines.
• Contribute to ground-truth data pipelines and processes that support reliable evaluation.

Skills

Python

Data pipelines

ML evaluation

Airflow

MLflow

Relational DBs

Time-series data

Collaboration

Tools

Grafana

PyTorch

Job description

VTI Aerospace, headquartered with Seattle and Bozeman offices, seeks a Senior Software Engineer – Evaluation to design and implement rigorous evaluation systems for computer vision, ASR, and language models. You will define methodologies, track KPIs, and collaborate with ML/data teams to enhance data and model performance.

You will build Python pipelines for benchmarking, monitor model drift in production, and create dashboards to visualize results, shaping the reliability of AI tools in