AI/ML Testing Engineer – Quality & Automation

Robots and Pencils; United States; Remote; USD 13,000 - 26,000; Full time

Job summary

Robots & Pencils is an applied AI engineering firm building AI co-workers for enterprise operations. We’re seeking a Testing Engineer to own testing across automation and AI workflows, contributing to quality strategy from day one.

You will join a cross-functional team to write automated tests, validate AI/ML pipelines, and integrate quality practices into delivery. You’ll collaborate with developers, ML engineers, and data scientists, driving test coverage and quality through CI/CD integration

Qualifications

• 2+ years of experience in test automation or AI/ML testing.
• Proficiency in Python, JavaScript, Java, or similar.
• Experience with test automation frameworks for UI, API, or ML workflows.
• Understanding of AI/ML concepts and the ML lifecycle.
• Experience with CI/CD pipelines and version control.
• Working knowledge of API testing and data validation.
• Solid understanding of SDLC/STLC and Agile methodologies.
• Proficiency with bug-tracking and test management tools.
• Familiarity with testing AI/ML systems, including LLM evaluation and RAG retrieval quality.
• Demonstrable usage of AI-forward tools like Claude and Cursor.
• Strong problem-solving skills and ownership mindset.
• Experience with mobile automation, performance testing, or MLOps is a plus.

Responsibilities

• Write and maintain automated test scripts for UI, API, and integration testing.
• Validate ML model outputs against defined criteria.
• Test data pipelines and AI workflows with data quality checks.
• Execute automated test suites and document defects in tracker systems.
• Integrate automated tests into CI/CD pipelines.
• Support model evaluation processes and test AI workflows end-to-end.
• Test AI/ML systems including LLM evaluation and safety testing.

Skills

Test automation

AI/ML testing

Python

JavaScript

Java

Tools

Selenium

PyTest

Playwright

GitHub Actions

CI/CD Pipelines

Job description

Robots & Pencils is an applied AI engineering firm building AI co-workers for enterprise operations. We’re seeking a Testing Engineer to own testing across automation and AI workflows, contributing to quality strategy from day one.

You will join a cross-functional team to write automated tests, validate AI/ML pipelines, and integrate quality practices into delivery. You’ll collaborate with developers, ML engineers, and data scientists, driving test coverage and quality through CI/CD integration