Software Quality Engineer

Jobtailor; Greater London; Hybrid; GBP 65,000 - 95,000; Full time

Job summary

Jobtailor is seeking a senior Quality Assurance Engineer in London to lead QA strategies for AI/LLM-powered applications.

You will design automated test frameworks and translate complex AI features into robust test cases, collaborating with ML engineers and data scientists.

Qualifications

• 5+ years of hands-on software quality assurance or engineering, focused on distributed AI/ML systems.
• 5+ years of experience with GitHub Actions and CI/CD pipelines.
• Proficiency in Python or TypeScript for test automation.
• Experience with modern testing frameworks such as Playwright.

Responsibilities

• Define and implement QA strategies and test plans for AI agents and LLM-powered apps.
• Design and develop scalable automated test frameworks.
• Translate AI features into test cases and validation criteria.
• Integrate automated checks into CI/CD pipelines.
• Manage defect lifecycle with logs and user feedback.
• Monitor production AI model performance and post-release validation.
• Promote quality practices across engineering teams.
• Design and manage test data strategies and mock services.
• Report to the director of engineering.

Skills

Quality Assurance

Test Automation

CI/CD

Python/TypeScript

Playwright

AI/ML Testing

Tools

GitHub Actions

AWS

Azure

GCP

Docker

Kubernetes

Job description

Job Responsibilities

• Define and implement comprehensive quality assurance strategies and test plans for AI agents and LLM-powered applications
• Design and develop robust, scalable, and maintainable automated test frameworks
• Collaborate with product managers, machine learning engineers, and data scientists to translate AI features and model behaviors into test cases and validation criteria
• Improve testing processes and infrastructure, integrating automated checks into CI/CD pipelines
• Identify, document, track, and analyze software defects and inconsistencies
• Monitor production systems and AI model performance and contribute to post-release quality validation
• Champion quality best practices across engineering teams
• Design, manage, and maintain test data strategies and mock services
• Manage the full defect lifecycle using customer feedback and debugging logs
• Report directly to the director of engineering

Requirements

• 5+ years of hands-on experience in software quality assurance or engineering, with a strong focus on testing complex distributed systems or AI/ML applications
• 5+ years of experience working with GitHub Actions
• Proficiency in Python or TypeScript for test automation
• Experience with modern testing frameworks such as Playwright
• Solid understanding of AI/ML concepts, including model evaluation metrics, data pipelines, and the unique challenges of testing generative AI outputs
• Experience designing, developing, or integrating agentic AI systems, AI skills, and the Model Context Protocol (MCP)
• Experience with AWS, Azure, or GCP and Docker or Kubernetes in a CI/CD environment is a big plus
• Exceptional analytical skills and keen attention to detail
• Demonstrated ability to drive initiatives independently and thrive in a fast-paced, evolving startup environment
• Must be located within 50 miles of one of the office hubs or be willing to relocate
• Must be able and willing to work in person in the office 3 days per week
• Must be legally authorized to work in the country in which the job is located

Core Competencies

Demonstrates expertise in quality assurance strategies for AI and LLM applications, with a strong focus on automated testing frameworks and CI/CD integration. Proficient in Python or TypeScript, with a solid understanding of AI/ML concepts and testing methodologies.

Highest-signal resume keywords

• Quality Assurance Strategies
• Automated Test Frameworks
• Python or TypeScript
• AI/ML Concepts
• GitHub Actions

Hard Skills

• Quality Assurance
• Test Automation
• Automated Testing Frameworks
• AI/ML Testing
• Defect Lifecycle Management
• Model Evaluation Metrics
• Data Pipelines
• Agentic AI Systems
• Playwright
• CI/CD Integration

Soft Skills

• Analytical Skills
• Attention to Detail
• Initiative
• Adaptability
• Collaboration

Industry Keywords

• AI Agents
• LLM-Powered Applications
• Testing Complex Distributed Systems
• Mock Services
• Post-Release Quality Validation

Tools & Technologies

• GitHub Actions
• AWS
• Azure
• GCP
• Docker
• Kubernetes