Global AI Quality & Observability Engineer

Plan.Net CreativeNest South Africa

Wes-Kaap

On-site

ZAR 900,000 - 1,300,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Plan.Net CreativeNest South Africa, part of the Serviceplan Group, is expanding its AI Engineering team. We seek an AI Quality & Observability Engineer to own the quality, reliability and performance of our AI systems used by international clients.

You will build end-to-end observability, develop automated evaluation frameworks and integrate quality checks into CI/CD pipelines to prevent production issues. Excellent English communication and cross‑functional collaboration across regions are

Qualifications

  • Several years of professional experience in Python development and backend engineering, ideally within production environments.
  • Hands-on experience building and maintaining evaluation frameworks for production LLM applications.
  • Strong understanding of LLM evaluation methodologies, automated scoring and human validation.
  • Experience with AWS, particularly CloudWatch and related monitoring services.
  • Practical knowledge of observability, including tracing, logging, metrics and alerting, ideally using OpenTelemetry.
  • Experience integrating automated quality checks into CI/CD pipelines.
  • Strong analytical and problem-solving skills with a proactive approach to quality improvement.
  • Excellent written and spoken English communication skills.
  • Experience working in international, cross-functional teams, ideally collaborating with colleagues and stakeholders across different countries and time zones.
  • Ability to communicate complex technical concepts and quality considerations to non-technical stakeholders.

Responsibilities

  • AI Observability: Implement end-to-end tracing across AI agents, tool calls, model interactions, token usage and latency.
  • Quality Assurance: Develop and maintain regression test sets to ensure consistent performance across AI applications.
  • LLM Evaluation: Build automated evaluation frameworks, including LLM-as-judge and code-based evaluators, calibrated against human assessments.
  • CI/CD Integration: Integrate quality checks into deployment pipelines to identify regressions before production releases.
  • Performance Monitoring: Establish dashboards and alerts to monitor system performance, operational costs and latency.
  • Incident Management: Investigate quality issues, identify root causes and implement improvements to prevent recurring problems.
  • Data Security & Compliance: Develop audit logging and monitoring processes for systems handling confidential or sensitive information.
  • Engineering Standards: Define best practices and collaborate with engineering teams to continuously improve the quality of our AI solutions.

Skills

Python development
Backend engineering
LLM evaluation
CI/CD pipelines
AWS CloudWatch
Observability
OpenTelemetry
Cross-functional teams
English communication

Tools

LangGraph
LangChain
LangSmith
Arize Phoenix

Job description

Plan.Net CreativeNest South Africa, part of the Serviceplan Group, is expanding its AI Engineering team. We seek an AI Quality & Observability Engineer to own the quality, reliability and performance of our AI systems used by international clients.

You will build end-to-end observability, develop automated evaluation frameworks and integrate quality checks into CI/CD pipelines to prevent production issues. Excellent English communication and cross‑functional collaboration across regions are

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Quality & Observability Engineer
AI Quality & Observability Engineer

Plan.Net CreativeNest South Africa • Wes-Kaap

On-site
ZAR 900,000 - 1,300,000
Software Quality Engineer — AI‑Driven Testing & Automation
Software Quality Engineer — AI‑Driven Testing & Automation

multiSEARCH Recruitment • Johannesburg

On-site
ZAR 662,800 - 1,159,900
Senior AI Engineer - Customer Ops & ML Agents
Senior AI Engineer - Customer Ops & ML Agents

AnyVan Ltd. • Cape Town

On-site
ZAR 1,200,000 - 1,800,000
Competitive Salary
Generous time off
Health & wellbeing
+2
AI Backend Engineer: Build Scalable AI Services
AI Backend Engineer: Build Scalable AI Services

Placements24 • East London

On-site
ZAR 700,000 - 1,100,000
Competitive salary
Performance bonus
Comprehensive health insurance package
+1
AI Engineer (Intermediate II / Senior I) - Cape Town (Durbanville) - up to 1mil per annum
AI Engineer (Intermediate II / Senior I) - Cape Town (Durbanville) - up to 1mil per annum

Ats Client • Cape Town

Hybrid
ZAR 900,000 - 1,100,000
Hybrid work model
Permanent role
Lead AI Quality Engineer & Platform Architect
Lead AI Quality Engineer & Platform Architect

Circle • South Africa

Remote
ZAR 2,527,000 - 3,088,000
Equity grants
35 days paid time off
Remote-first culture
Head of AI Engineering – Lead & Scale AI Solutions
Head of AI Engineering – Lead & Scale AI Solutions

Placements24 • Centurion

On-site
ZAR 900,000 - 1,500,000
Competitive salary package
Health and wellness benefits
Career development & training
Ai Practice Lead
Ai Practice Lead

E-Solutions • Johannesburg

On-site
ZAR 1,200,000 - 1,800,000
AI Engineering Lead: Scale Gen AI & LLMs
AI Engineering Lead: Scale Gen AI & LLMs

Network Recruitment • Johannesburg

On-site
ZAR 1,000,000 - 1,800,000
Senior Backend Engineer: AI Systems & API Orchestration
Senior Backend Engineer: AI Systems & API Orchestration

Manpowergroup • Cape Town

On-site
ZAR 700,000 - 1,000,000