Principal Software Engineer, Test Automation & AI

Roca Alliances; Medellín; On-site; COP 360,000,000 - 480,000,000; Full time

Job summary

Roca Alliances is seeking a Principal Software Engineer in Test to own product quality for a global SaaS field service platform. You will define test strategy, automation architecture, and AI agents to execute it.

This role is a deeply technical IC, closer to platform engineering than QA, with no people-management duties. You will own the quality signal, automate critical customer workflows end-to-end, and drive a release gate based on realistic data.

Qualifications

• Deep experience designing test automation for a large, long-lived web application.
• Hands-on experience with LLMs, agent loops, and evaluation harnesses in CI.
• Strong SQL and relational data skills for test data in complex schemas.
• CI/CD ownership: pipelines, gating, flake management and run-time budgets.
• Advanced English.
• Experience with AI-driven testing approaches is a plus.

Responsibilities

• Own test strategy across web, mobile, and API layers and data.
• Design and operate automation architecture and data strategy for testing.
• Develop AI agents that derive test cases from tickets and code changes and review PRs.
• Maintain engineering practices for quality, including standards and guardrails.
• Deliver a clear quality signal to engineering and executives.

Skills

Test automation design

LLM agent loops

SQL

CI/CD ownership

English proficiency

AI in testing

Tools

Jenkins

Bitbucket

Jira

Confluence

Terraform

Datadog

Grafana

Job description

Principal Software Engineer, Test Automation & AI

Medellin, Colombia

At Roca Alliances Our mission is to inspire nearshore outsourcing to Latin America, as our expertise is providing customized services to help companies from the U.S., Canada, and Europe expand their operations through guided entry with our consolidated experience in the Latin American region. We are an expert recruitment firm, connecting top talent with leading global companies. We deliver tailored hiring solutions to ensure the perfect match for each role. With deep insight into industry trends, we drive targeted, efficient recruitment that builds long-lasting partnerships and supports business growth. Our mission is to inspire and help businesses thrive by delivering the right professionals to meet their needs.

Job Description

About the client

Our client is a global SaaS company building field service management software for trades businesses. Its platform runs their customers' jobs, quotes, invoices, scheduling and stock. It is part of a group of software brands with teams in Australia, the United Kingdom and North America, and it has a strong AI-first engineering strategy.

About the role

We are looking for a Principal Software Engineer in Test to own product quality: the test strategy, the automation architecture behind it, and the AI agents that carry it out. You will be the most senior technical voice on quality and the person accountable for whether a release is safe to ship. This is a deeply technical individual contributor role at principal level, closer to platform engineering than to traditional QA. It is not a manual testing or people-management position.

What you'll own

• Test strategy: what gets tested, at which layer and against what data, and what evidence a release must produce before it ships. You define the release health gate.
• Automation architecture: suite design across web, mobile and API, plus test data strategy, environments and fixtures, flake budgets, run times and the CI setup that keeps the gate credible.
• AI agents that test the product: you design, build and operate them. That includes agents that derive test cases from tickets and code changes, explore the application, turn real customer workflows into regression scenarios and review pull requests. You also own how they are evaluated.
• Engineering practice: how developers test what they write, including test standards, guardrails against recurring defects, static checks and review rules.
• The quality signal: defect rate, escape rate and early detection of critical issues, reported clearly to engineering and the executive team.

What success looks like in your first year

• A test and release strategy agreed with engineering leadership.
• The critical customer workflows automated end to end and running as a release gate on realistic data.
• At least one AI testing agent in production, finding defects the existing suites miss.
• Regression suites that cover boundary, volume and multi-record scenarios, plus feature-flag-aware verification.
• A quality dashboard the executive team can read without translation.

Tech environment

• Product: PHP / Symfony, Angular (including legacy layers), Ionic / Capacitor mobile apps, PostgreSQL, AWS
• Delivery and observability: Jenkins, Bitbucket, Jira, Confluence, Terraform, Datadog, Grafana

Requirements

Required

• You write production-quality code and would be hired as a senior software engineer on merit.
• Deep experience designing test automation for a large, long-lived web application.
• Hands-on experience building with LLMs beyond prompting: agent loops, tool / MCP integration, structured outputs and evaluation harnesses, with a view on cost and non-determinism in CI.
• Strong SQL and relational data skills for building and reasoning about test data in a complex schema.
• CI/CD ownership: pipelines, gating, flake management and run-time budgets.
• The ability to hold a position on quality with engineers, product and executives, and to change your mind when the evidence says so.
• Advanced English.

Nice to have

• PHP, Symfony or Angular experience, or comfort working with legacy layers you did not choose.
• Experience in field service, ERP or other domains where the product runs a customer's working day.
• Experience with multi-tenant products.
• Observability-driven testing: building scenarios from production telemetry.
• Published writing, talks or open-source work on testing or applied AI.
• Permanent contract (contrato a término indefinido) with all legal benefits
• Private health insurance (medicina prepagada)