Senior SDET - AI/ML Testing & LLMs with MLOps

TechDigital Group; Minnesota; Hybride; USD 120 000 - 170 000; Plein temps

Résumé du poste

TechDigital Group in Minnesota seeks a seasoned SDET/QA lead for AI/ML testing of LLM/RAG systems. You will design test automation solutions, evaluate bias, drift, and fairness, and implement CI/CD pipelines with Azure DevOps.

You’ll work on performance metrics, MLflow, synthetic data, and explainability tools, collaborating with data and engineering teams to improve test coverage and reliability. Experience with Databricks, Azure Cloud services, and integration testing across streaming jobs and

Qualifications

• Design LLM/RAG test automation solutions.
• Test bias, drift and fairness.
• Familiar with performance metrics (precision, recall, F1, ROC-AUC).
• Knowledge of MLflow MLOps framework
• Knowledge of synthetic data generation and boundary testing
• KnowAzure DevOps for CI/CD pipeline development
• Experience with RAG/pipeline evaluation frameworks
• Exposure to explainability frameworks (SHAP, LIME, Captum)
• Hands-on API and database testing
• Databricks and Azure Cloud services experience
• Proficiency with Azure DevOps pipelines YAML templates
• Experience implementing AIML testing practices (test generation, anomaly detection, log analysis)
• Familiar with Cucumber (BDD) and Allure for test reporting
• Strong integration testing across Databricks streaming jobs and microservices
• Experience with performance testing tools (JMeter, LoadRunner)
• Experience with Pester for PowerShell scripts
• Solid Agile/Scrum knowledge with JIRA
• Excellent problem-solving and communication skills

Responsabilités

• Design and implement LLM/RAG test automation solutions.
• Evaluate bias, drift and fairness in AI systems.
• Develop and maintain CI/CD pipelines using Azure DevOps.
• Perform API and database testing across data platforms.
• Conduct performance testing and monitor drift with tools like Evidently.
• Collaborate with data science and engineering teams to improve test coverage.
• Apply explainability methods (SHAP/LIME) to test results.
• Support integration testing across Databricks streaming jobs and microservices.
• Document test plans and generate reports with Allure.
• Utilize synthetic data and boundary testing to strengthen tests.

Connaissances

SDET skills

Azure DevOps

Python

pytest framework

AI/ML testing

RAG testing

LLMs testing

Outils

DeepEval

TruLens

RAGAS

Eleuther

Promptfoo

Evidently

SHAP

LIME

Captum

Databricks

Azure DevOps pipelines

Description du poste

TechDigital Group in Minnesota seeks a seasoned SDET/QA lead for AI/ML testing of LLM/RAG systems. You will design test automation solutions, evaluate bias, drift, and fairness, and implement CI/CD pipelines with Azure DevOps.

You’ll work on performance metrics, MLflow, synthetic data, and explainability tools, collaborating with data and engineering teams to improve test coverage and reliability. Experience with Databricks, Azure Cloud services, and integration testing across streaming jobs and