Senior AI Tooling & Testing Engineer

Palo Alto Networks, Inc.; Santa Clara (CA); On-site; USD 147,000 - 237,500; Full time

Job summary

Palo Alto Networks, Inc. is seeking a highly experienced testing engineer to design and develop testing frameworks for AI/ML models and LLM applications. You will build automated pipelines for model validation, regression testing, and benchmarking and collaborate with engineers and product teams to define test strategies.

The role emphasizes evaluating datasets, synthetic data, and safety metrics, plus monitoring model performance in production and ensuring adherence to ethical AI standards.

Qualifications

• 8+ years of experience in software engineering, QA automation, or ML engineering.
• Strong programming skills in Python (preferred).
• Experience with testing frameworks (PyTest, unittest).
• Familiarity with APIs, distributed systems, and CI/CD pipelines.
• Knowledge of data structures, algorithms, and software design principles.

Responsibilities

• Design and develop testing frameworks for AI/ML models and LLM applications.
• Build automated pipelines for model validation, regression testing, and benchmarking.
• Create evaluation datasets, synthetic data, and test scenarios for edge cases.
• Implement metrics to assess accuracy, robustness, latency, and safety.
• Develop tools for prompt testing, output validation, and hallucination detection.
• Collaborate with engineers and product teams to define test strategies.
• Monitor model performance in production and build alerting systems.
• Ensure compliance with ethical AI standards, fairness, and bias testing.
• Debug model behavior and identify root causes of failures.
• Demonstrate analytical thinking and problem-solving.

Skills

Python

Testing frameworks

APIs

CI/CD

Distributed systems

ML concepts

Model evaluation

Communication skills

Education

Bachelor’s or Master’s in CS/AI/ML

Tools

LangChain

OpenAI Evals

CI/CD tooling

Job description

Palo Alto Networks, Inc. is seeking a highly experienced testing engineer to design and develop testing frameworks for AI/ML models and LLM applications. You will build automated pipelines for model validation, regression testing, and benchmarking and collaborate with engineers and product teams to define test strategies.

The role emphasizes evaluating datasets, synthetic data, and safety metrics, plus monitoring model performance in production and ensuring adherence to ethical AI standards.