Role Overview
We are seeking a Senior Engineer to lead data operations and model evaluation activities supporting AI/ML development in a medical device environment. This role is responsible for managing the collection, organization, quality, governance, and distribution of clinical and field-derived datasets (e.g., TEE, CT, MRI, ICE imaging and AF mapping) used for AI model training and validation.
Title: R&D Engineer Sr.
Location: St. Paul, MN
Duration: 12 Months
Work Arrangement: 100% Onsite
The engineer will establish and maintain ground truth datasets, evaluate model performance against reference standards, generate metrics and performance reports, and partner closely with AI/ML engineers to continuously improve model quality. The role also serves as a key liaison between clinical partners, data providers, AI development teams, Cybersecurity, Privacy, Legal, and external vendors to ensure high-quality data pipelines and compliant handling of sensitive healthcare data.
What You Will Do
AI Data Management & Operations
- Manage the lifecycle of training, validation, and testing datasets used for AI/ML development.
- Coordinate collection, ingestion, organization, reviews, and maintenance of clinical and field-derived datasets, including imaging, mapping, and procedural data.
- Develop and maintain data inventories, metadata standards, lineage documentation, and tracking systems.
- Maintain auditable records of dataset provenance, versioning, approvals, and usage to support development, quality, and regulatory activities.
- Coordinate secure transfer and compliant handling and sharing of sensitive data and approved datasets with internal teams and external collaborators.
- Monitor data quality, completeness, consistency, and readiness for AI development activities.
- Support annotation and labeling workflows, including vendor management and quality review processes.
- Collaborate with AI/ML, software, systems, clinical, quality, regulatory, cybersecurity, privacy, legal, and OEC teams.
- Drive continuous improvement in data operations infrastructure and AI evaluation methodologies.
- Define and monitor data quality KPIs including completeness, consistency, annotation quality, and dataset representativeness.
- Identify data gaps and drive remediation plans with data providers and development teams.
Model Evaluation & Validation
- Establish and maintain ground truth datasets for model training and performance assessment.
- Own the governance, quality, and traceability of ground truth datasets used for training, validation, and verification activities.
- Design and execute model evaluation workflows to compare new AI models against established performance baselines.
- Coordinate with cross-functional teams to establish application-specific measures to evaluate and communicate model performance.
- Analyze failure modes, data quality issues, and performance gaps to identify improvement opportunities.
- Create dashboards, reports, and visualizations to communicate model performance trends to technical and non-technical stakeholders.
- Collaborate with AI/ML engineers to recommend improvements to data pipelines, labeling quality, and model development processes.
- Support statistical analysis and performance characterization for research, product development, and regulatory activities.
Required Qualifications
- Bachelor’s degree (or equivalent experience) in a related field (Computer Science, Engineering, Data/Analytics, Health Informatics, Biomedical, or similar).
- 5+ years of relevant industry experience
- Experience supporting AI/ML development, data engineering, model validation, analytics, or algorithm development.
- Demonstrated experience designing and executing model evaluation and performance frameworks.
- Experience with Power BI, Tableau, Azure Data Platform, Databricks, or cloud-based analytics environments.
- Demonstrated experience coordinating data transfers, documentation, or data pipeline handoffs across multiple stakeholders.
- Demonstrated ability to independently lead data operations programs with minimal supervision.
- Excellent communication skills with customers, data owners, compliance, and legal.
- Strong organizational and execution skills: tracking, follow-up, issue triage, status reporting, and escalation management.
- Demonstrated ability to lead cross-functional initiatives and influence technical decisions.
Preferred Qualifications
- Experience with medical imaging datasets including DICOM, CT, MRI, Ultrasound, TEE, ICE, or electrophysiology mapping systems.
- Experience with AI/ML model development lifecycles and MLOps workflows.
- Familiarity with machine learning evaluation methodologies and statistical validation techniques.
- Experience supporting regulated medical device software or AI-enabled products.
- Familiarity with FDA, MDR, or other regulatory expectations related to AI/ML validation.
Consultants Eligible Benefits Upon Waiting Period
- Medical and Prescription Drug Plans
- Dental Plan
- Vision Plan
- Health Savings Account (for High-Deductible Health Plans)
- Flexible Spending Accounts (Health, Limited Purpose, Dependent Care, Commuter Parking and Commuter Transit)
- Supplemental Life Insurance
- Short Term Disability (coverage varies by state)
- Long Term Disability
- Critical Illness, Hospital coverage, Accident Insurance
- MetLife Legal, MetLife ID Fraud, and MetLife Pet Insurance
- 401(k)