SDET - AI/ML

TechDigital Group; Minnesota; Híbrido; USD 120.000 - 170.000; Jornada completa

Descripción de la vacante

TechDigital Group in Minnesota seeks a seasoned SDET/QA lead for AI/ML testing of LLM/RAG systems. You will design test automation solutions, evaluate bias, drift, and fairness, and implement CI/CD pipelines with Azure DevOps.

You’ll work on performance metrics, MLflow, synthetic data, and explainability tools, collaborating with data and engineering teams to improve test coverage and reliability. Experience with Databricks, Azure Cloud services, and integration testing across streaming jobs and

Formación

• Design LLM/RAG test automation solutions.
• Test bias, drift and fairness.
• Familiar with performance metrics (precision, recall, F1, ROC-AUC).
• Knowledge of MLflow MLOps framework
• Knowledge of synthetic data generation and boundary testing
• KnowAzure DevOps for CI/CD pipeline development
• Experience with RAG/pipeline evaluation frameworks
• Exposure to explainability frameworks (SHAP, LIME, Captum)
• Hands-on API and database testing
• Databricks and Azure Cloud services experience
• Proficiency with Azure DevOps pipelines YAML templates
• Experience implementing AIML testing practices (test generation, anomaly detection, log analysis)
• Familiar with Cucumber (BDD) and Allure for test reporting
• Strong integration testing across Databricks streaming jobs and microservices
• Experience with performance testing tools (JMeter, LoadRunner)
• Experience with Pester for PowerShell scripts
• Solid Agile/Scrum knowledge with JIRA
• Excellent problem-solving and communication skills

Responsabilidades

• Design and implement LLM/RAG test automation solutions.
• Evaluate bias, drift and fairness in AI systems.
• Develop and maintain CI/CD pipelines using Azure DevOps.
• Perform API and database testing across data platforms.
• Conduct performance testing and monitor drift with tools like Evidently.
• Collaborate with data science and engineering teams to improve test coverage.
• Apply explainability methods (SHAP/LIME) to test results.
• Support integration testing across Databricks streaming jobs and microservices.
• Document test plans and generate reports with Allure.
• Utilize synthetic data and boundary testing to strengthen tests.

Conocimientos

SDET skills

Azure DevOps

Python

pytest framework

AI/ML testing

RAG testing

LLMs testing

Herramientas

DeepEval

TruLens

RAGAS

Eleuther

Promptfoo

Evidently

SHAP

LIME

Captum

Databricks

Azure DevOps pipelines

Descripción del empleo

Experience level: 10 + years

Must have skills: SDET skills, Azure devops, python, pytest test automation and Framework, AI ML practices, RAGAS, LLMs

Brief JD:

• Experience in designing LLM/RAG test automation solutions
• Experience in testing for bias, drift, and fairness.
• Familiarity with performance metrics (precision, recall, F1, ROC-AUC).
• Knowledge of MLflow MLOps framework
• Knowledge of tools for synthetic data generation and boundary testing
• Knowledge of Azure DevOps for CI/CD pipeline development
• Experience in RAG/pipeline evaluation frameworks
  • pytest: Test automation framework
  • DeepEval or TruLens: LLM test assertions
  • RAGAS: RAG-specific metrics
  • Eleuther: LLM evaluation harness
  • Garak or Promptfoo: LLM red-teaming
  • Evidently: Drift/performance monitoring
• Exposure to explainability frameworks (SHAP, LIME, Captum)

• Handson experience in API and Database testing.
• Practical knowledge of Databricks, Azure Cloud services land distributed data validation.
• Proficiency with Azure DevOps pipelines YAML templates, agent pools, CICD workflows.
• Experience in implementing AIML practices in testing e.g., test generation, anomaly detection, log analysis to improve test efficiency and coverage.
• Familiarity with Cucumber (BDD) and test reporting frameworks (e.g., Allure).
• Strong understanding of integration testing across Databricks streaming jobs, applications and microservices.
• Experience with performance testing tools (JMeter, LoadRunner, or equivalent).
• Experience with Pester framework for validating PowerShell scripts and infrastructure automation.
• Solid understanding of Agile/Scrum methodology and handson usage of JIRA.
• Excellent problem-solving, debugging, and communication skills, with the ability to advise development teams on testing best practices.