AI Test Automation Engineer

Globant; Egypt (AR); On-site; USD 120,000 - 180,000; Full time

Job summary

Globant in Egypt is seeking an AI Test Automation Engineer to lead quality transformation for our flagship agentic banking platform. You will design deterministic pytest suites, validate AI behavior, and build LLM-as-a-judge evaluation scripts to gate releases.

You will mock LLM endpoints, inspect traces from LangGraph or Microsoft Agent Framework, and integrate tests into GitLab CI, delivering weekly quality evidence and robust guardrails.

Qualifications

• 4+ years of test automation with Python and pytest, including API testing and JSON schema validation.
• 1+ years hands-on experience evaluating non-deterministic AI applications.
• Proven track record building LLM-as-a-judge scoring scripts and maintaining versioned evaluation datasets.
• Ability to validate tool selection, inspect multi-step agent execution traces, and mock agent endpoints for CI workflows.
• Experience designing negative-path test suites for prompt injections, hallucinations, and guardrails.
• Hands-on integration of test stages into GitLab CI or equivalent with trace analysis tools.

Responsibilities

• Design and implement deterministic pytest suites to validate agent behavior and JSON schema compliance.
• Build and maintain LLM-as-a-judge evaluation scripts that score outputs for correctness and tone.
• Mock LLM endpoints and core APIs to ensure repeatable agent loops in CI pipelines.
• Inspect traces from LangGraph or Microsoft Agent Framework to diagnose failures and guard against non-determinism.
• Integrate test suites into GitLab CI and deliver weekly quality evidence for releases.

Skills

Test automation

LLM testing

LLM evaluation engineering

Guardrail testing

CI/CD integration

Tools

Python

pytest

LangGraph

Microsoft Agent Framework

LangChain

LlamaIndex

Job description

At Globant, we are working to make the world a better place, one step at a time. We enhance business development and enterprise solutions to prepare them for a digital future. With a diverse and talented team present in more than 30 countries, we are strategic partners to leading global companies in their business process transformation.

Join Globant as an AI Test Automation Engineer and lead the quality transformation for our flagship agentic banking platform. We are looking for an automation expert skilled in Python, pytest, and LLM/Agentic system testing to solve one of tech's newest challenges: systematically validating non-deterministic AI behavior. In this role, you will build automated evaluation datasets, inspect agent execution traces, and deploy LLM-as-a-judge frameworks to gate production releases with absolute confidence.

Screening Requirements

• Automation Mastery: 4+ years of test automation experience using Python and pytest (custom fixtures, mocking, and CI pipeline integration).
• LLM & Agentic System Testing: 1+ years of hands‑on experience evaluating non‑deterministic AI applications.
• LLM Evaluation Engineering: Proven track record building LLM-as-a-judge scoring scripts and maintaining versioned evaluation datasets (golden paths, edge cases, regression suites).
• Agent Deep‑Dives & Traces: Ability to validate tool selection and parameters, inspect multi‑step agent execution traces (e.g., LangGraph, Microsoft Agent Framework), and mock agent endpoints for CI workflows.
• Adversarial & Guardrail Testing: Experience designing negative‑path test suites for prompt injections, hallucinated financial actions, and trust‑level boundaries (Suggest / Approve / Act).
• Observability & CI Integration: Hands‑on integration of test stages into GitLab CI (or equivalent) using LLM observability tools like OPIK or similar trace analysis platforms.

Key Responsibilities

• Agent Test Design & Automation: Design and implement deterministic pytest suites to validate agent behavior, intent handling, tool‑call correctness, strict JSON schemas, and failure paths.
• LLM Evaluation Engineering: Build and maintain LLM-as-a-judge evaluation scripts that score agent outputs for correctness, tone, and compliance, defining scoring thresholds and pass/fail release gates.
• Mocking & Trace Analysis: Build mocks and fixtures for LLM endpoints and core APIs so agent loops run repeatably in CI pipelines. Execute, mock, and capture traces from LangGraph or Microsoft Agent Framework agent loops within test suites.
• Guardrail & Security Testing: Continuously validate that agents refuse prompt injection attempts, do not hallucinate actions, and strictly follow compliance guardrails.
• CI/CD Integration & Reporting: Integrate test suites into GitLab CI with DevOps, leverage OPIK for trace analysis, and deliver automated quality evidence gating weekly drops.

Required Experience & Qualifications

• Test Automation (Essential): 4+ years of test automation with strong Python and pytest. Experience testing APIs, contract testing, and strict JSON schema validation.
• AI & LLM Testing (Essential): 1+ years of experience with behavioral acceptance criteria, evaluation datasets, and LLM-as-a-judge techniques. Practical understanding of managing non‑determinism via statistical thresholds and automated judges.
• Frameworks (Essential): Hands‑on familiarity with agentic frameworks (LangGraph, Microsoft Agent Framework, LangChain, or LlamaIndex) sufficient to execute and mock agent loops.
• Observability & Domain (Desirable): Experience with OPIK or equivalent trace analysis platforms. Prior experience testing in banking, fintech, or regulated financial environments.
• Knowledge of Arabic language is plus.

This job can be filled in Egypt/MENA Region.

Create with us digital products that people love. We will bring businesses and consumers together through AI technology and creativity, driving digital transformation to impact the world positively.

We may use AI and machine learning technologies in our recruitment process. Globant is an Equal Opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, veteran status, or any other characteristic protected by applicable federal, state, or local law. Globant is also committed to providing reasonable accommodations for qualified individuals with disabilities in our job application procedures. If you need assistance or an accommodation due to a disability, please let your recruiter know.

Final compensation offered is based on multiple factors such as the specific role, hiring location, as well as individual skills, experience, and qualifications. In addition to competitive salaries, we offer a comprehensive benefits package. Learn more about life at Globant here: Globant Experience Guide.