AI/ML Testing Engineer — Validate & Ship QA Automation

Robots-&-Pencils-; Puerto Bogotá; On-site; COP 190,967,000 - 286,451,000; Full time

Job summary

Robots & Pencils is an applied AI engineering firm building AI co-workers that integrate into enterprise operations. This role focuses on owning testing duties across automation and AI, shaping quality strategy, and taking real responsibility from day one.

You will write automated tests, validate AI/ML pipelines, and integrate quality practices into delivery with cross-functional engineers.

Qualifications

• 2+ years of experience in test automation or AI/ML testing.
• Experience with test automation frameworks for UI, API, or ML workflows.
• Understanding of AI/ML concepts and the ML lifecycle, with the ability to validate model outputs.
• Experience with CI/CD pipelines and version control.
• Working knowledge of API testing and data validation.
• SDLC/STLC and Agile methodologies knowledge.
• Bug-tracking and test management tools proficiency.
• Familiarity with AI/ML testing: LLM evaluation, RAG, model drift, safety.
• Hands-on use of AI-forward tools like Claude and Cursor.
• Strong problem-solving skills and ambiguity handling.

Responsibilities

• Write and maintain automated test scripts for UI, API, and integration testing.
• Validate ML model outputs for accuracy and consistency against criteria.
• Test data pipelines and AI workflows with data quality checks.
• Run automated test suites, analyze results, and log defects in Jira/TestRail.
• Integrate tests into CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins).
• Support model evaluation and end-to-end AI workflow testing.
• Test AI/ML systems including LLM evaluation, RAG, model drift, and safety.
• Collaborate with developers, ML engineers, and data scientists on quality goals.
• Participate in sprint ceremonies and quality strategy discussions.
• Provide clear testing reports and communicate status to the team.

Skills

Test automation

AI/ML testing

CI/CD

API testing

Data validation

Bug-tracking tools

Agile methodologies

Mobile automation

MLOps pipelines

Claude/Cursor usage

Tools

Selenium

Cypress

Playwright

PyTest

Jira

TestRail

GitHub Actions

GitLab CI

Jenkins

Claude

Cursor

Job description

Robots & Pencils is an applied AI engineering firm building AI co-workers that integrate into enterprise operations. This role focuses on owning testing duties across automation and AI, shaping quality strategy, and taking real responsibility from day one.

You will write automated tests, validate AI/ML pipelines, and integrate quality practices into delivery with cross-functional engineers.