Senior AI Tooling Engineer - Testing & Validation

Socket.dev; California (MO); On-site; USD 147,000 - 238,000; Full time

Job summary

Palo Alto Networks in the United States is seeking a senior QA/ML engineer to design and develop testing frameworks for AI/ML models and LLM applications. You will build automated pipelines for model validation, regression testing, and benchmarking, and create evaluation datasets and test scenarios for edge cases.

The role requires 8+ years in software engineering or ML QA, strong Python skills, experience with PyTest/unittest, APIs, and CI/CD.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, AI, Machine Learning, or related field.
• 8+ years of experience in software engineering, QA automation, or ML engineering.
• Strong programming skills in Python (preferred) or similar languages.
• Experience with testing frameworks (e.g., PyTest, unittest).
• Familiarity with machine learning concepts and model evaluation techniques.
• Experience working with APIs, distributed systems, and CI/CD pipelines.

Responsibilities

• Design and develop testing frameworks for AI/ML models and LLM applications.
• Build automated pipelines for model validation, regression testing, and benchmarking.
• Create evaluation datasets, synthetic data, and test scenarios for edge cases.
• Implement metrics to assess accuracy, robustness, latency, and safety.
• Develop tools for prompt testing, output validation, and hallucination detection.
• Collaborate with engineers and product teams to define test strategies.
• Monitor model performance in production and build alerting systems.
• Ensure compliance with ethical AI standards, fairness, and bias testing.
• Debug model behavior and identify root causes of failures.
• Analytical thinking and problem-solving.
• Attention to detail and quality-focused mindset.
• Strong collaboration and communication skills.
• Ability to design scalable and reusable testing systems.

Skills

Python

QA automation

ML engineering

APIs

CI/CD

Distributed systems

Education

Bachelor's or Master’s in CS/AI/ML

Tools

PyTest

unittest

LangChain

OpenAI Evals

AWS

GCP

Azure

Job description

Palo Alto Networks in the United States is seeking a senior QA/ML engineer to design and develop testing frameworks for AI/ML models and LLM applications. You will build automated pipelines for model validation, regression testing, and benchmarking, and create evaluation datasets and test scenarios for edge cases.

The role requires 8+ years in software engineering or ML QA, strong Python skills, experience with PyTest/unittest, APIs, and CI/CD.