Senior Automation Test Engineer – AI/LLM, PyTest & CI/CD

Eleks; Argentina; On-site; ARS 1,200,000 - 2,100,000; Full time

Job summary

ELEKS Quality Assurance Office in Argentina seeks a Senior Automation Test Engineer (AI/LLM) to join the client delivery team. You will drive automated testing for AI-powered applications, including LLMs and RAG solutions, using Python, PyTest, and REST API validation.

The role emphasizes data validation with SQL/Databricks, prompt-safety testing, and CI/CD integration with Jenkins, Docker, and Git. Strong English is required, and you will collaborate globally with a health-insurance client.

Qualifications

• Experience testing LLM/RAG behavior and prompt safety.
• Automation testing with Python and PyTest.
• REST API and FastAPI testing experience.
• SQL and Databricks data validation.
• LLM intent-routing and RAG retrieval-quality testing.
• Prompt-injection and PII-leakage testing.
• Groundedness, hallucination, and citation validation in LLMs.
• Experience with JSONL datasets and automated regression testing.
• Postman and Newman used for API testing.
• Jenkins CI/CD integration, Docker and Git workflows.
• Monitoring with Application Insights or OpenTelemetry.
• Upper-Intermediate English proficiency.

Responsibilities

• Develop, execute, and maintain automated tests using Python and PyTest.
• Perform REST API and FastAPI testing.
• Validate data using SQL and Databricks.
• Validate LLM intent routing and RAG retrieval quality.
• Perform prompt-injection and PII-leakage testing.
• Validate groundedness, hallucinations, and citations in LLM outputs.
• Conduct LLM evaluations using RAGAS or DeepEval.
• Execute automated regression testing using JSONL datasets.
• Use Postman and Newman for API testing.
• Integrate automated tests with Jenkins CI/CD.
• Utilize Docker and Git in test automation activities.
• Monitor applications with Application Insights or OpenTelemetry.

Skills

Python

PyTest

REST APIs

FastAPI

SQL

Databricks

LLM testing

RAG testing

Prompt injection

PII leakage testing

Groundedness checks

JSONL datasets

Postman

Newman

Jenkins CI/CD

Docker

Git

OpenTelemetry

Application Insights

English Proficiency

Tools

Postman

Newman

Jenkins CI/CD

Docker

Git

OpenTelemetry

Application Insights

Databricks

Job description

ELEKS Quality Assurance Office in Argentina seeks a Senior Automation Test Engineer (AI/LLM) to join the client delivery team. You will drive automated testing for AI-powered applications, including LLMs and RAG solutions, using Python, PyTest, and REST API validation.

The role emphasizes data validation with SQL/Databricks, prompt-safety testing, and CI/CD integration with Jenkins, Docker, and Git. Strong English is required, and you will collaborate globally with a health-insurance client.