AI Quality & Observability Engineer

Plan.Net CreativeNest South Africa

Wes-Kaap

On-site

ZAR 900,000 - 1,300,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Plan.Net CreativeNest South Africa, part of the Serviceplan Group, is expanding its AI Engineering team. We seek an AI Quality & Observability Engineer to own the quality, reliability and performance of our AI systems used by international clients.

You will build end-to-end observability, develop automated evaluation frameworks and integrate quality checks into CI/CD pipelines to prevent production issues. Excellent English communication and cross‑functional collaboration across regions are

Qualifications

  • Several years of professional experience in Python development and backend engineering, ideally within production environments.
  • Hands-on experience building and maintaining evaluation frameworks for production LLM applications.
  • Strong understanding of LLM evaluation methodologies, automated scoring and human validation.
  • Experience with AWS, particularly CloudWatch and related monitoring services.
  • Practical knowledge of observability, including tracing, logging, metrics and alerting, ideally using OpenTelemetry.
  • Experience integrating automated quality checks into CI/CD pipelines.
  • Strong analytical and problem-solving skills with a proactive approach to quality improvement.
  • Excellent written and spoken English communication skills.
  • Experience working in international, cross-functional teams, ideally collaborating with colleagues and stakeholders across different countries and time zones.
  • Ability to communicate complex technical concepts and quality considerations to non-technical stakeholders.

Responsibilities

  • AI Observability: Implement end-to-end tracing across AI agents, tool calls, model interactions, token usage and latency.
  • Quality Assurance: Develop and maintain regression test sets to ensure consistent performance across AI applications.
  • LLM Evaluation: Build automated evaluation frameworks, including LLM-as-judge and code-based evaluators, calibrated against human assessments.
  • CI/CD Integration: Integrate quality checks into deployment pipelines to identify regressions before production releases.
  • Performance Monitoring: Establish dashboards and alerts to monitor system performance, operational costs and latency.
  • Incident Management: Investigate quality issues, identify root causes and implement improvements to prevent recurring problems.
  • Data Security & Compliance: Develop audit logging and monitoring processes for systems handling confidential or sensitive information.
  • Engineering Standards: Define best practices and collaborate with engineering teams to continuously improve the quality of our AI solutions.

Skills

Python development
Backend engineering
LLM evaluation
CI/CD pipelines
AWS CloudWatch
Observability
OpenTelemetry
Cross-functional teams
English communication

Tools

LangGraph
LangChain
LangSmith
Arize Phoenix

Job description

At Plan.Net CreativeNest, we are a growing creative and technology hub based in South Africa and part of Serviceplan Group, one of Europe's largest independent agency groups.

We bring together creative talent, technology specialists, strategists and engineers to deliver innovative solutions for international clients across the Serviceplan network.

Our AI Engineering team develops and operates advanced multi-agent and Large Language Model (LLM) systems that solve real business challenges. Working closely with our international colleagues, particularly in Germany, we build scalable, reliable AI solutions that are used in production by our clients.

As we continue to grow our AI capabilities, we are looking for an AI Quality & Observability Engineer to take ownership of the quality, reliability and performance of our AI systems.

Your Role

You will be responsible for establishing and maintaining quality and observability standards across our AI engineering projects.

Working alongside Backend Engineers, DevOps specialists and international project teams, you will ensure our AI systems are reliable, measurable, cost-efficient and secure.

You will play a key role in defining how we evaluate AI performance, identify potential issues and continuously improve the systems we deliver to our clients.

What You Will Do
  • AI Observability: Implement end-to-end tracing across AI agents, tool calls, model interactions, token usage and latency.
  • Quality Assurance: Develop and maintain regression test sets to ensure consistent performance across AI applications.
  • LLM Evaluation: Build automated evaluation frameworks, including LLM-as-judge and code-based evaluators, calibrated against human assessments.
  • CI/CD Integration: Integrate quality checks into deployment pipelines to identify regressions before production releases.
  • Performance Monitoring: Establish dashboards and alerts to monitor system performance, operational costs and latency.
  • Incident Management: Investigate quality issues, identify root causes and implement improvements to prevent recurring problems.
  • Data Security & Compliance: Develop audit logging and monitoring processes for systems handling confidential or sensitive information.
  • Engineering Standards: Define best practices and collaborate with engineering teams to continuously improve the quality of our AI solutions.
What We're Looking For
  • Several years of professional experience in Python development and backend engineering, ideally within production environments.
  • Hands‑on experience building and maintaining evaluation frameworks for production LLM applications.
  • Strong understanding of LLM evaluation methodologies, automated scoring and human validation.
  • Experience with AWS, particularly CloudWatch and related monitoring services.
  • Practical knowledge of observability, including tracing, logging, metrics and alerting, ideally using OpenTelemetry.
  • Experience integrating automated quality checks into CI/CD pipelines.
  • Strong analytical and problem-solving skills with a proactive approach to quality improvement.
  • Excellent written and spoken English communication skills.
  • Experience working in international, cross‑functional teams, ideally collaborating with colleagues and stakeholders across different countries and time zones.
  • Ability to communicate complex technical concepts and quality considerations to non‑technical stakeholders.
Nice to Have
  • Experience with LangGraph, LangChain or similar agent frameworks.
  • Familiarity with Langfuse, LangSmith, Arize Phoenix, Braintrust or comparable evaluation platforms.
  • Experience working with AI applications that process confidential or regulated data.
  • Knowledge of data protection requirements such as POPIA or GDPR.
  • Experience in an agency, consultancy or international technology environment.
Why Join CreativeNest?

At CreativeNest, we combine the agility of a growing South African team with the opportunities of an international agency network.

You will have the chance to help shape our AI engineering capabilities, influence technical decisions and contribute to solutions that are deployed in real client environments.

We value people who take ownership, challenge existing approaches and bring new ideas to the table.

Join us and help build the next generation of reliable, scalable and impactful AI solutions at CreativeNest.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Global AI Quality & Observability Engineer
Global AI Quality & Observability Engineer

Plan.Net CreativeNest South Africa • Wes-Kaap

On-site
ZAR 900,000 - 1,300,000
Ai Developer
Ai Developer

Liquid Thought • Cape Town

On-site
ZAR 600,000 - 900,000
AI Engineer
AI Engineer

Platform45 • Johannesburg

On-site
ZAR 900,000 - 1,300,000
Job Title:Electrician
Job Title:Electrician

Manpowergroup • Cape Town

On-site
ZAR 700,000 - 1,000,000
Senior Python AI Engineer
Senior Python AI Engineer

Sabenza IT & Recruitment • Pretoria

On-site
ZAR 750,000 - 1,100,000
AI Solutions Architect - Natural Language Processing
AI Solutions Architect - Natural Language Processing

Placements24 • Cape Town

Hybrid
ZAR 1,200,000 - 2,100,000
Performance bonuses
Medical, dental, and vision coverage
Flexible working hours
+5
Ai Tooling Engineer
Ai Tooling Engineer

Jobgether • South Africa

Remote
ZAR 900,000 - 1,300,000
Fully remote work
Multicultural team
End-to-end ownership
Lead AI Engineer
Lead AI Engineer

Placements24 • Cape Town

Remote
ZAR 1,200,000 - 1,800,000
Annual bonus
Home office setup
Health care (medical, dental, vision)
Lead Ai Solutions Architect - Innovation Hub
Lead Ai Solutions Architect - Innovation Hub

Placements24 • Cape Town

Hybrid
ZAR 1,200,000 - 1,800,000
Stock options
Health insurance
Professional development budget
+1
AI Solutions Architect - Cloud & Automation
AI Solutions Architect - Cloud & Automation

Placements24 • Rustenburg

On-site
ZAR 1,500,000 - 1,900,000
Healthcare benefits
Retirement plan
Development allowance
+2