Senior AI/LLM Automation Test Engineer

Eleks; Argentina; On-site; ARS 138,487,000 - 200,037,000; Full time

Job summary

ELEKS Quality Assurance Office is seeking a Senior Automation Test Engineer (AI/LLM) in Argentina to join the client delivery team. You will focus on testing AI-powered applications, including LLMs and RAG solutions, while developing automated tests in Python and PyTest and validating REST APIs and data pipelines.

The role requires experience with LLM validation, JSONL regression testing, Postman/Newman, and integration with Jenkins CI/CD, Docker, and Git, plus strong English skills.

Qualifications

• Experience testing LLM/RAG behavior, including intent routing, retrieval quality, prompt safety, and evaluation/regression testing.
• Experience in test automation using Python and PyTest.
• Experience testing REST APIs and FastAPI-based applications.
• Experience with SQL and Databricks data validation.
• Experience with LLM intent-routing validation.
• Experience with RAG retrieval-quality testing.
• Experience with prompt-injection and PII-leakage testing.
• Experience with groundedness, hallucination, and citation validation.
• Experience with RAGAS or DeepEval evaluation.
• Experience with automated regression testing using JSONL datasets.
• Experience with Postman and Newman.
• Experience with Jenkins CI/CD integration.
• Experience with Docker and Git.
• Experience with Application Insights or OpenTelemetry monitoring.
• At least Upper-Intermediate level of English.

Responsibilities

• Develop, execute, and maintain automated tests using Python and PyTest.
• Perform REST API and FastAPI testing.
• Validate data using SQL and Databricks.
• Validate LLM intent routing and RAG retrieval quality.
• Perform prompt-injection and PII-leakage testing.
• Validate groundedness, hallucinations, and citations in LLM-generated responses.
• Conduct LLM evaluations using RAGAS or DeepEval.
• Perform automated regression testing using JSONL datasets.
• Use Postman and Newman for API testing.
• Integrate automated tests with Jenkins CI/CD.
• Use Docker and Git in test automation activities.
• Monitor applications using Application Insights or OpenTelemetry.

Skills

LLM/RAG testing

Python

PyTest

REST APIs

FastAPI

SQL

Databricks data validation

Prompt-injection testing

PII leakage testing

Groundedness/Hallucination validation

RAGAS/DeepEval evaluation

JSONL regression testing

Postman/Newman

Jenkins CI/CD

Docker & Git

Application Insights/OpenTelemetry

Upper-Intermediate English

Tools

OpenTelemetry

Postman

Newman

Docker

Git

Jenkins

Databricks

FastAPI

Job description

ELEKS Quality Assurance Office is seeking a Senior Automation Test Engineer (AI/LLM) in Argentina to join the client delivery team. You will focus on testing AI-powered applications, including LLMs and RAG solutions, while developing automated tests in Python and PyTest and validating REST APIs and data pipelines.

The role requires experience with LLM validation, JSONL regression testing, Postman/Newman, and integration with Jenkins CI/CD, Docker, and Git, plus strong English skills.