Senior QA Automation Engineer (AI-First)

dowjones; Dublin; On-site; EUR 90,000 - 120,000; Full time

Job summary

Storyful is seeking a hands-on Senior QA Automation Engineer to raise the quality bar while accelerating delivery with AI-augmented tooling. You will design automated tests, build scalable frameworks, and work with engineers, data scientists, product managers, and SRE to embed quality throughout the lifecycle.

You will leverage Playwright/TypeScript, AI coding tools, and evaluation frameworks to scale testing, ensure reliability, and manage non-deterministic AI features with measurable quality

Qualifications

• Proven experience as a Senior QA Automation Engineer / SDET in production environments.
• Hands-on with Playwright using TypeScript in large-scale systems.
• Familiarity with AI coding tools and LLM-based quality evaluation.
• Strong understanding of automation frameworks and end-to-end test strategies.

Responsibilities

• Design and build AI-augmented automation systems across web UI, APIs, and data pipelines.
• Develop scalable automation frameworks and reusable testing patterns.
• Translate acceptance criteria into executable tests and maintain test suites.
• Reduce flakiness via observability, traces, and AI-assisted failure triage.
• Embed testing early in the CI/CD pipeline and collaborate with engineers.

Skills

Playwright

TypeScript

AI tooling

Automation frameworks

Testing strategies

CI/CD

Observability

Debugging

Security testing

Software engineering

Job description

Job Description

Founded by pioneering journalists as the first social media newswire, Storyful was created out of the need to break the news faster and utilize social content to add context to reporting. Acquired by News Corp in 2013, Storyful has evolved into a premium service for media, business leaders and investors.

Our goal is to help our partners understand the attitudes, behaviors and emotions shaping the world. Powered by a unique ability to gather and streamline data from all corners of an increasingly complex information environment, our teams deliver clarity in a world of confusion. Our mission is to dig deeper into the nuance inherent in digital media to establish context, verify the truth and help our partners make sense of the world.

Storyful is building AI-native products on top of complex, high-volume, multi-source content. We are looking for a hands-on Senior QA Automation Engineer to raise the quality bar while increasing delivery speed through AI-augmented engineering. This is a senior individual contributor role for someone who treats AI as a core part of their craft - using coding agents, LLMs, and evaluation frameworks to scale quality faster than headcount. You will design and build the automation, evaluation, and quality foundations that enable engineers to own quality, while ensuring our products remain reliable, explainable, and trusted - even as they become increasingly probabilistic. You will work closely with software engineers, data scientists, product managers, and SRE to embed quality into every stage of the development lifecycle.

What you will do

• Design and build AI-augmented automation systems
• Build and evolve scalable automation frameworks across web UI, APIs, and data pipelines (Playwright / TypeScript)
• Use AI coding agents (e.g. Claude Code) to generate, refactor, and maintain test suites at scale
• Translate natural-language acceptance criteria into executable, maintainable tests
• Define reusable testing patterns that enable engineers to own automation within their domains
• Continuously reduce flakiness using trace analysis, observability, and AI-assisted failure triage
• Enable shift-left, engineering-led quality
• Partner with engineers to embed testing early in the development lifecycle
• Define pragmatic, risk-based test strategies across unit, integration, contract, and E2E layers
• Establish clear definitions of done, quality gates, and release readiness standards
• Build tooling and frameworks that allow teams to move fast without relying on central QA
• Build QA practices for AI/LLM-powered features
• Design evaluation pipelines for LLM-driven features using:
  • Golden datasets and regression harnesses
  • Semantic similarity scoring
  • LLM-as-a-judge evaluation patterns
• Detect and prevent issues such as prompt drift, hallucinations, retrieval failures, and schema regressions
• Test agentic workflows end-to-end (tool use, multi-step reasoning, failure modes)
• Partner with data science to define measurable quality signals (precision, recall, grounding, drift)
• Integrate quality into CI/CD and delivery
• Embed automation and evaluation into CI/CD pipelines for fast, reliable feedback
• Define and tune meaningful quality gates based on real metrics (not ceremony)
• Support release processes with high-confidence signals and rollback readiness
• Drive debugging, observability, and defect excellence
• Investigate complex issues across distributed systems using logs, traces, and metrics
• Improve observability, including LLM trace visibility (e.g. Langfuse)
• Establish strong defect triage, categorisation, and reporting practices
• Use data (leakage, flake rate, MTTR, escape rate) to continuously improve quality
• Own non-functional and AI-specific risk testing
• Embed security testing into automation (OWASP, API security, auth flows)
• Test AI-specific risks (prompt injection, jailbreaks, unsafe tool use, data leakage)
• Collaborate on performance, resilience, and scalability testing where it matters most

What good looks like in this role

Engineers own quality by default, supported by strong frameworks and AI tooling

Automation is fast, reliable, and trusted as a release signal

LLM-powered features ship with measurable, repeatable quality evaluation

Regression risk is proactively managed across both deterministic and probabilistic systems

Delivery speed increases without compromising trust or stability

Quality becomes a competitive advantage, not a bottleneck

What we're looking for

• Proven experience as a Senior QA Automation Engineer / SDET in production environments
• Strong hands-on experience with Playwright (or equivalent) using TypeScript
• Demonstrated use of AI coding tools (e.g. Claude Code) as part of daily workflow
• Strong software engineering fundamentals and ability to write maintainable production-grade code
• Experience designing automation frameworks and test strategies end-to-end