Senior QA Engineer - Automation

Recruitment Intelligence; Islamabad; On-site; PKR 1,800,000 - 2,400,000; Full time

Benefits offered by this job

Great compensation package

Medical benefit for you and yourfamily

Free lunch

Annual performance-tied increments & w

Performance recognition awards

Lean and agile work culture

Job summary

Convo is seeking a QA Automation Engineer with 5+ years of experience to build robust automated testing frameworks for APIs, agent workflows, tool calls, and critical UI experiences.

The candidate should have strong Python, pytest, API testing, Playwright, CI/CD, contract testing, and asynchronous workflows, plus experience creating reusable fixtures, mocks, and simulators for reliable, scalable product quality.

Qualifications

• 5+ years in test automation
• Strong Python, pytest and API-testing experience
• CI pipeline integration and test-result reporting
• Contract/schema validation for REST and event-driven interfaces
• Ability to test asynchronous, non-deterministic and retrying workflows
• Experience designing reusable test fixtures, mocks and simulators

Responsibilities

• Automate regression testing for agent tool calls, planner outputs, APIs and the operator-facing agentic console.
• Create API and workflow regression suites for orchestrator, planner and tool-registry behavior.
• Build deterministic assertions around schemas, tool-call sequences, constraints and expected side effects.
• Implement pytest-based service tests and Playwright tests for relevant console workflows.
• Create mocks, simulators and fixtures for tools, model responses and external dependencies.
• Integrate suites into CI, publish actionable results and manage flaky-test remediation.
• Maintain traceability from technical requirements and defects to automated coverage.

Skills

Python

pytest

API testing

CI/CD

contract testing

asynchronous workflows

test automation

Tools

Playwright

Job description

Convo is looking for QA Automation Engineers with 5+ years of experience in test automation to build robust automated testing frameworks for APIs, agent workflows, tool calls, and critical UI experiences. The ideal candidate should have strong expertise in Python, pytest, API testing, Playwright, CI/CD, contract testing, and asynchronous workflows, with experience creating reusable test fixtures, mocks, and simulators to ensure reliable and scalable product quality.

Technical mission

Automate regression testing for agent tool calls, planner outputs, APIs and the operator-facing agentic console.

Key responsibilities

Create API and workflow regression suites for orchestrator, planner and tool-registry behavior.

Build deterministic assertions around schemas, tool-call sequences, constraints and expected side effects.

Implement pytest-based service tests and Playwright tests for relevant console workflows.

Create mocks, simulators and fixtures for tools, model responses and external dependencies.

Integrate suites into CI, publish actionable results and manage flaky-test remediation.

Maintain traceability from technical requirements and defects to automated coverage.

Required technical capabilities

Minimum experience: 4+ years in test automation, including 2+ years testing APIs, asynchronous workflows or distributed services.

Strong Python, pytest and API-testing experience.

CI pipeline integration and test-result reporting.

Contract/schema validation for REST and event-driven interfaces.

Ability to test asynchronous, non-deterministic and retrying workflows without masking defects.

Experience designing reusable test fixtures, mocks and service simulators.

Preferred experience

Agent/LLM workflow evaluation and tool-call trace analysis.

Performance, resilience and fault-injection testing.

OpenTelemetry or distributed-trace-based test diagnostics.

Automated regression suites for planner, tool calls, APIs and critical UI paths.

CI integration with quality thresholds and readable failure evidence.

Reusable mocks, fixtures and simulators.

Coverage and defect-leakage reporting with maintained documentation.

Primary interfaces

Works with Agent Runtime, DevOps, AI Test & Dataset Engineering and product teams consuming the shared framework.

What We Have For You

• Great compensation package
• medical benefit for you and your family
• free lunch
• annual performance-tied increments & performance recognition awards
• a great lean and agile work culture!

Convo endorses a culture of diversity in all aspects and aims to build a diverse team of amazing individuals!