Principal Software Quality Assurance Engineer

Roca Alliances; Medellín; On-site; COP 180,000,000 - 240,000,000; Full time

Benefits offered by this job

Permanent contract

Private health insurance

Job summary

Roca Alliances is seeking a Principal Software Engineer in Test to own product quality across web, mobile, and API. You will lead the test strategy, automation architecture, and AI agents that execute tests, driving release safety and quality at scale.

The role demands deep experience in test automation for large SaaS platforms, hands‑on work with LLMs, strong SQL skills, and the ability to influence engineers, product, and executives. This is a senior IC role with platform focus.

Qualifications

• Experience designing test automation for large, long‑lived web applications.
• Hands‑on experience building with LLMs, agent loops, tool integration and evaluation harnesses.
• Strong SQL and relational data skills for test data in complex schemas.
• Proven CI/CD ownership: pipelines, gating, flake management and run‑time budgets.
• Ability to align quality strategy with engineers, product and executives, and adapt when evidence demands.

Responsibilities

• Own test strategy: decide what to test, data to use, and evidence required before release.
• Design and maintain automation architecture across web, mobile, and API layers, including data strategy and CI fixtures.
• Develop and operate AI agents that derive tests from tickets, explore the app, and generate regression scenarios.
• Set engineering testing standards, guardrails, and review rules to improve product quality.
• Provide clear quality signals: track defect and escape rates, and communicate outcomes to leadership.

Skills

Test automation design

LLM-based agent loops

SQL & data modeling

CI/CD ownership

Cross-functional influence

Advanced English

Tools

Jenkins

Bitbucket

Jira

Confluence

Terraform

Datadog

Grafana

Job description

Our client is a global SaaS company building field service management software for trades businesses. Its platform runs their customers' jobs, quotes, invoices, scheduling and stock. It is part of a group of software brands with teams in Australia, the United Kingdom and North America, and it has a strong AI-first engineering strategy.

About the role

We are looking for a Principal Software Engineer in Test to own product quality: the test strategy, the automation architecture behind it, and the AI agents that carry it out. You will be the most senior technical voice on quality and the person accountable for whether a release is safe to ship. This is a deeply technical individual contributor role at principal level, closer to platform engineering than to traditional QA. It is not a manual testing or people-management position.

What you'll own

• Test strategy: what gets tested, at which layer and against what data, and what evidence a release must produce before it ships. You define the release health gate.
• Automation architecture: suite design across web, mobile and API, plus test data strategy, environments and fixtures, flake budgets, run times and the CI setup that keeps the gate credible.
• AI agents that test the product: you design, build and operate them. That includes agents that derive test cases from tickets and code changes, explore the application, turn real customer workflows into regression scenarios and review pull requests. You also own how they are evaluated.
• Engineering practice: how developers test what they write, including test standards, guardrails against recurring defects, static checks and review rules.
• The quality signal: defect rate, escape rate and early detection of critical issues, reported clearly to engineering and the executive team.

What success looks like in your first year

• A test and release strategy agreed with engineering leadership.
• The critical customer workflows automated end to end and running as a release gate on realistic data.
• At least one AI testing agent in production, finding defects the existing suites miss.
• Regression suites that cover boundary, volume and multi-record scenarios, plus feature-flag‑aware verification.
• A quality dashboard the executive team can read without translation.

Tech environment

• Product: PHP / Symfony, Angular (including legacy layers), Ionic / Capacitor mobile apps, PostgreSQL, AWS
• Delivery and observability: Jenkins, Bitbucket, Jira, Confluence, Terraform, Datadog, Grafana

Required

• You write production-quality code and would be hired as a senior software engineer on merit.
• Deep experience designing test automation for a large, long‑lived web application.
• Hands‑on experience building with LLMs beyond prompting: agent loops, tool / MCP integration, structured outputs and evaluation harnesses, with a view on cost and non‑determinism in CI.
• Strong SQL and relational data skills for building and reasoning about test data in a complex schema.
• CI/CD ownership: pipelines, gating, flake management and run‑time budgets.
• The ability to hold a position on quality with engineers, product and executives, and to change your mind when the evidence says so.
• Advanced English.

Nice to have

• PHP, Symfony or Angular experience, or comfort working with legacy layers you did not choose.
• Experience in field service, ERP or other domains where the product runs a customer's working day.
• Experience with multi‑tenant products.
• Observability‑driven testing: building scenarios from production telemetry.
• Published writing, talks or open‑source work on testing or applied AI.
• Permanent contract (contrato a término indefinido) with all legal benefits
• Private health insurance (medicina prepagada)