Senior QA Engineer - Automation

Convo; Islamabad; On-site; PKR 1,200,000 - 2,000,000; Full time

Benefits offered by this job

Medical benefits for you and family

Free lunch

Annual performance-tied increments

Performance recognition awards

Agile lean culture

Job summary

Convo is seeking QA Automation Engineers with 5+ years of test automation experience to build robust frameworks for APIs, agent workflows, tool calls, and critical UI experiences. You will implement reusable fixtures, mocks, and simulators to ensure reliable quality across distributed services.

The role emphasizes Python, pytest, Playwright, CI/CD, and contract testing, with a focus on asynchronous workflows and scalable regression suites integrated into our CI pipeline.

Qualifications

• 5+ years of test automation experience.
• Strong Python, pytest and API-testing experience.
• Experience with Playwright for UI workflows.
• CI pipeline integration and test-result reporting.
• Contract/schema validation for REST and event-driven interfaces.

Responsibilities

• Create API and workflow regression suites for orchestrator, planner and tool-registry behavior.
• Build deterministic assertions around schemas, tool-call sequences and expected side effects.
• Implement pytest-based service tests and Playwright tests for console workflows.
• Create mocks, simulators and fixtures for tools, model responses and external dependencies.
• Integrate suites into CI, publish actionable results and manage flaky-test remediation.
• Maintain traceability from technical requirements and defects to automated coverage.

Skills

Python

pytest

API testing

Playwright

CI/CD

Contract testing

Asynchronous workflows

Tools

Playwright

Job description

Convo is looking for QA Automation Engineers with 5+ years of experience in test automation to build robust automated testing frameworks for APIs, agent workflows, tool calls, and critical UI experiences. The ideal candidate should have strong expertise in Python, pytest, API testing, Playwright, CI/CD, contract testing, and asynchronous workflows, with experience creating reusable test fixtures, mocks, and simulators to ensure reliable and scalable product quality.

Technical mission

Automate regression testing for agent tool calls, planner outputs, APIs and the operator-facing agentic console.

Key Responsibilities

• Create API and workflow regression suites for orchestrator, planner and tool-registry behavior.
• Build deterministic assertions around schemas, tool-call sequences, constraints and expected side effects.
• Implement pytest-based service tests and Playwright tests for relevant console workflows.
• Create mocks, simulators and fixtures for tools, model responses and external dependencies.
• Integrate suites into CI, publish actionable results and manage flaky-test remediation.
• Maintain traceability from technical requirements and defects to automated coverage.

Required Technical Capabilities

• Minimum experience: 4+ years in test automation, including 2+ years testing APIs, asynchronous workflows or distributed services.
• Strong Python, pytest and API-testing experience.
• Playwright or comparable browser automation for critical UI workflows.
• CI pipeline integration and test-result reporting.
• Contract/schema validation for REST and event-driven interfaces.
• Ability to test asynchronous, non-deterministic and retrying workflows without masking defects.
• Experience designing reusable test fixtures, mocks and service simulators.

Preferred Experience

• Agent/LLM workflow evaluation and tool-call trace analysis.
• Performance, resilience and fault-injection testing.
• OpenTelemetry or distributed-trace-based test diagnostics.

Expected deliverables / acceptance evidence

• Automated regression suites for planner, tool calls, APIs and critical UI paths.
• CI integration with quality thresholds and readable failure evidence.
• Reusable mocks, fixtures and simulators.
• Coverage and defect-leakage reporting with maintained documentation.

Primary Interfaces

Works with Agent Runtime, DevOps, AI Test & Dataset Engineering and product teams consuming the shared framework.

What We Have For You

• Great compensation package, medical benefit for you and your family, free lunch, annual performance-tied increments & performance recognition awards and a great lean and agile work culture!
• Convo endorses a culture of diversity in all aspects and aims to build a diverse team of amazing individuals!