Senior Automation Tester (Python) – Agentic AI & LLM Systems (Hybrid)

Resillion; Birmingham; Hybrid; GBP 50,000 - 70,000; Full time

Job summary

Resillion is seeking a Senior Automation Tester (Python) to develop and maintain automation frameworks for validating AI systems, including chatbots and agentic workflows. Applicants should have strong Python skills, experience in test automation, and familiarity with CI/CD pipelines. This role involves close collaboration with cross-functional teams to deliver high-quality AI products. The position offers a hybrid work model ideal for passionate individuals interested in automation engineering and AI technologies.

Qualifications

• 3+ years of experience in automation engineering and testing AI/LLM systems.
• Strong knowledge of building automation frameworks.
• Experience with CI/CD tools and practices.

Responsibilities

• Develop and maintain a Python-based automation framework for validating AI agents.
• Design and execute automated tests for LLM responses and conversational flows.
• Integrate automated tests into CI/CD pipelines.

Skills

Python development

Test automation

AI/LLM systems testing

CI/CD pipeline implementation

Source code management (Git)

Tools

Docker

Kubernetes

Jenkins

GitHub Actions

Job description

At Resillion, our culture is built on a mindset of “if you see something, say something” — taking responsibility and speaking up. We expect everyone to adapt and embrace change as the company grows. We recognise the value of every employee and invest in developing their skills so that, as an organisation, we continue to lead our industry.

Above all, it’s a culture driven by passion for what we do and commitment to the bigger picture of the company.

If you want to be part of our journey, this role may be the right fit for you.

Job Description

Senior Automation Tester (Python) – Agentic AI & LLM Systems

Location: Glasgow (Hybrid preferred but not essential)

Start Date: Immediate

Right to Work: Must have full UK working rights (no sponsorship available due to start dates)

Additional Requirement: Must have 3 years of UK address history due to immediate onboarding

Programme: AI / Agentic AI

Seniority: Senior Automation Engineer

Overview

We are seeking a Senior Automation Engineer / SDET with strong Python skills and hands‑on experience testing AI systems, including LLM‑driven applications and agentic AI workflows. You will design and build a scalable automation framework, implement modern AI testing strategies, and ensure the quality and reliability of next‑generation conversational and agent‑based AI products.

This role is ideal for someone who is passionate about automation engineering and has a solid understanding of how AI/LLM systems behave in real‑world scenarios.

Key Responsibilities

• Develop and maintain a Python‑based automation framework for validating AI agents (chatbots, voicebots, agentic workflows).
• Design and execute automated tests for:
  • LLM responses
  • Conversational flows
  • Retrieval‑augmented generation (RAG)
  • End‑to‑end agentic scenarios
• Apply modern AI testing strategies and evaluation methods (e.g., DeepEval, RAGAS — beneficial but not essential).
• Integrate automated tests into CI/CD pipelines (Jenkins, GitHub Actions, Azure Repos).
• Build and manage batch and scheduled executions using Docker, Kubernetes, cron jobs, or similar tooling.
• Manage code repositories using Git, including branching strategies, version control, and collaborative workflows.
• Collaborate closely with engineering, AI/ML, and product teams to ensure high‑quality delivery of AI‑powered features.

Required Skills & Experience

• Strong Python development skills, particularly for automation (PyTest preferred).
• Solid background in test automation and framework design.
• Experience testing or validating AI/LLM‑based systems (agentic workflows, conversational AI, RAG pipelines, etc.).
• Understanding of AI evaluation concepts — familiarity with tools like DeepEval or RAGAS is a bonus, not a requirement.
• Experience implementing CI/CD pipelines and automated batch executions.
• Strong understanding of source code management, Git workflows, and collaborative engineering practices.
• Ability to design structured, scalable, and maintainable automation solutions.

Preferred Qualities

• Quick learner with a proactive, ownership‑driven mindset.
• Strong problem‑solving skills and ability to work in fast‑moving AI environments.
• Comfortable collaborating with cross‑functional teams (AI/ML, engineering, product).
• Passion for AI technologies and modern testing methodologies.