Test Lead

Zinkworks; Athlone; Vor Ort; EUR 85.000 - 120.000; Vollzeit

Zusammenfassung

Zinkworks is seeking a hands-on QA Lead to own quality for Network Digital Twin and agentic AI use cases in telecom and broadband. Define how we test AI agents that reason, call tools and act on live data, and build the testing framework and evaluation harnesses.

You will lead QA strategy, write test scenarios, and work with AI engineers, developers and architects on cloud-native systems running on GCP. Strong background in software testing and agentic testing is required.

Qualifikationen

• 7+ years in software testing with team leadership experience.
• Experience testing distributed microservices, REST APIs and event-driven systems.
• Hands-on testing of LLM-based or agentic applications.
• Knowledge of evaluation methods: golden datasets, LLM-as-judge, trajectory and tool-call evaluation.
• Experience with evaluation/observability frameworks such as LangSmith, Langfuse, DeepEval, promptfoo or Ragas.
• Strong Python (Pytest) for test automation and CI/CD integration.
• Understanding AI risks: hallucination, prompt injection, data leakage and unsafe actions.

Aufgaben

• Define the QA strategy for agentic software systems.
• Own end-to-end QA for agentic apps, including agents, tools, prompts, and RAG pipelines.
• Define quality gates for non-deterministic systems and establish acceptance criteria.
• Set offline evaluation, pre-release regression, and production monitoring of agent behavior.
• Validate Digital Twin fidelity against real network and use as a safe sandbox for testing.
• Build and run testing, and maintain evaluation harnesses and golden datasets.
• Test REST and event-driven microservices interactions for correctness and permissions.
• Design adversarial and safety tests: prompt injections, jailbreaks, data leakage, guardrail bypass.
• Integrate automated tests into CI/CD for backend and agent interactions.
• Lead QA across the team and mentor engineers on AI testing practices.
• Track quality metrics and regressions to leadership.
• Ensure agents are testable by design with observability hook points.
• Participate in agile ceremonies and drive testing discipline.

Kenntnisse

Distributed testing

REST APIs

Event-driven systems

Agentic testing

Python/Pytest

Observability tools

Tools

LangSmith

Langfuse

DeepEval

promptfoo

Ragas

Jobbeschreibung

About Zinkworks

Zinkworks partners with leading Telecommunications and Financial Services organizations to modernize legacy systems, migrate mission-critical platforms to the cloud, and engineer AI-driven automation. From OSS transformation to rApp development and network intelligence, our teams simplify complexity and turn it into competitive advantage. Based in Ireland and operating across the EU, UK, and US, Zinkworks combines deep domain expertise with delivery excellence to help clients modernize faster and operate smarter.

About the Role

We are looking for a hands-on QA Lead to own quality for our Network Digital Twin and agentic AI use cases in the telecoms and broadband domain. You will define how we test AI agents that reason, call tools and act on live network data, and you will build and run that testing yourself.

This is not a strategy-only role. You will set the QA strategy for agentic applications, then write the evaluation harnesses, automation and test scenarios that put it into practice. You will work alongside AI engineers, data scientists, developers and architects on cloud-native systems running on GCP.

You will bring a solid software testing background and extend it into the newer discipline of agentic testing: non-deterministic outputs, LLM evaluation, tool-calling correctness, guardrails and safety. Familiarity with telecom/PON is a strong bonus.

What You'll Do

• Define the QA strategy for agentic Software systems

• Own the end-to-end QA strategy for agentic applications, covering agents, tools, prompts, RAG pipelines, multi-agent workflows and the Digital Twin services they act on.

• Define quality gates and acceptance criteria for non-deterministic systems: task success rate, tool-call accuracy, groundedness, hallucination rate, latency and cost per task.

• Set the approach for offline evaluation, pre-release regression and production monitoring of agent behaviour.

• Define how the Digital Twin is validated for fidelity against the real network, and how it is used as a safe sandbox to test agents before they touch live systems.

• Build and run the testing, hands-on

• Build and maintain evaluation harnesses and golden datasets for agent scenarios, including LLM-as-judge, rubric-based and deterministic assertions.

• Test tool and API calls made by agents (including MCP and function calling) for correctness, sequencing, error handling and permission boundaries.

• Design adversarial and safety tests: prompt injection, jailbreaks, data leakage, unsafe or out-of-scope actions, and guardrail bypass.

• Write automated tests for REST and event-driven microservices behind the Digital Twin and integrate them into CI/CD pipelines.

• Run exploratory and scenario-based testing of agent workflows, including human-in-the-loop approvals and failure recovery.

• Debug agent behaviour using traces, logs and observability tools (e.g. LangSmith, Langfuse, OpenTelemetry, Cloud Logging) to find the root cause of failures.

• Lead quality across the team

• Lead and mentor QA engineers, and coach developers and AI engineers on testing practices for AI systems.

• Track and report quality metrics and model/prompt regressions to engineering and product leadership.

• Work with product owners and architects to make agents testable by design, with clear specifications, observability and evaluation hooks.

• Take part in agile ceremonies and drive continuous improvement in how we test AI.

What You'll Bring

• 7+ years in software testing, with at least 2 years leading QA for a team or product area.

• A strong foundation in testing distributed microservices, REST APIs and event-driven systems (e.g. Kafka, Pub/Sub).

• Hands‑on experience testing LLM-based or agentic applications, including evaluation of non-deterministic outputs.

• Practical knowledge of agent evaluation methods: golden datasets, LLM-as-judge, trajectory and tool‑call evaluation, and regression across model or prompt changes.

• Experience with at least one evaluation or observability framework such as DeepEval, Ragas, promptfoo, LangSmith or Langfuse.

• Solid coding skills in Python (or similar) for building test automation and evaluation pipelines, with tools such as Pytest.

• Understanding of AI risks and how to test for them: hallucination, prompt injection, bias, data leakage and unsafe actions.

• Experience embedding automated tests and quality gates into CI/CD pipelines.

• Ability to debug backend and agent interactions using logs, traces and metrics.

• Clear communication, with the ability to explain AI quality risks to technical and non-technical stakeholders.

Nice to Have

• Experience testing Digital Twins, simulators or network emulation environments.

• Familiarity with the telecom/PON domain (Passive Optical Networks) and network operations use cases.

• Experience on Google Cloud Platform, including Vertex AI, GKE, Pub/Sub or Cloud Run.

• Knowledge of agent frameworks such as LangGraph, Google ADK, CrewAI or the Model Context Protocol (MCP).

• Performance and load testing of AI services, including token cost and latency budgets.

Inclusion & Diversity at Zinkworks

At Zinkworks, we are deeply committed to fostering a culture of diversity, inclusion, and belonging. We believe that our strength lies in the unique backgrounds, perspectives, and experiences of our team members. By embracing an inclusive environment, we empower innovation and collaboration across all levels of our organisation. Based in Ireland, we are proud to support and engage with our local communities through meaningful initiatives that promote equity and opportunity. Our commitment extends beyond the workplace, as we actively contribute to creating a more inclusive and connected society for all.