Senior Automation Test Engineer (AI/LLM)

Eleks; Argentina; On-site; ARS 1,200,000 - 2,100,000; Full time

Job summary

ELEKS Quality Assurance Office in Argentina seeks a Senior Automation Test Engineer (AI/LLM) to join the client delivery team. You will drive automated testing for AI-powered applications, including LLMs and RAG solutions, using Python, PyTest, and REST API validation.

The role emphasizes data validation with SQL/Databricks, prompt-safety testing, and CI/CD integration with Jenkins, Docker, and Git. Strong English is required, and you will collaborate globally with a health-insurance client.

Qualifications

• Experience testing LLM/RAG behavior and prompt safety.
• Automation testing with Python and PyTest.
• REST API and FastAPI testing experience.
• SQL and Databricks data validation.
• LLM intent-routing and RAG retrieval-quality testing.
• Prompt-injection and PII-leakage testing.
• Groundedness, hallucination, and citation validation in LLMs.
• Experience with JSONL datasets and automated regression testing.
• Postman and Newman used for API testing.
• Jenkins CI/CD integration, Docker and Git workflows.
• Monitoring with Application Insights or OpenTelemetry.
• Upper-Intermediate English proficiency.

Responsibilities

• Develop, execute, and maintain automated tests using Python and PyTest.
• Perform REST API and FastAPI testing.
• Validate data using SQL and Databricks.
• Validate LLM intent routing and RAG retrieval quality.
• Perform prompt-injection and PII-leakage testing.
• Validate groundedness, hallucinations, and citations in LLM outputs.
• Conduct LLM evaluations using RAGAS or DeepEval.
• Execute automated regression testing using JSONL datasets.
• Use Postman and Newman for API testing.
• Integrate automated tests with Jenkins CI/CD.
• Utilize Docker and Git in test automation activities.
• Monitor applications with Application Insights or OpenTelemetry.

Skills

Python

PyTest

REST APIs

FastAPI

SQL

Databricks

LLM testing

RAG testing

Prompt injection

PII leakage testing

Groundedness checks

JSONL datasets

Postman

Newman

Jenkins CI/CD

Docker

Git

OpenTelemetry

Application Insights

English Proficiency

Tools

Postman

Newman

Jenkins CI/CD

Docker

Git

OpenTelemetry

Application Insights

Databricks

Job description

ELEKS Quality Assurance Office is looking for a Senior Automation Test Engineer (AI/LLM) in Argentina

ABOUT PROJECT

Our customer is among the top 5 health insurance companies in the US. They are dedicated to expanding access to high-quality, cost-effective healthcare.

We are looking for an experienced Automation Test Engineer to join the project engagement and work embedded within the client’s delivery team, focusing on testing AI-powered applications, including Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) solutions.

REQUIREMENTS

• Experience testing LLM/RAG behavior, including intent routing, retrieval quality, prompt safety, and evaluation/regression testing
• Experience in test automation using Python and PyTest
• Experience testing REST APIs and FastAPI-based applications
• Experience with SQL and Databricks data validation
• Experience with LLM intent-routing validation
• Experience with RAG retrieval-quality testing
• Experience with prompt-injection and PII-leakage testing
• Experience with groundedness, hallucination, and citation validation
• Experience with RAGAS or DeepEval evaluation
• Experience with automated regression testing using JSONL datasets
• Experience with Postman and Newman
• Experience with Jenkins CI/CD integration
• Experience with Docker and Git
• Experience with Application Insights or OpenTelemetry monitoring
• At least Upper-Intermediate level of English

RESPONSIBILITIES

• Develop, execute, and maintain automated tests using Python and PyTest
• Perform REST API and FastAPI testing
• Validate data using SQL and Databricks
• Validate LLM intent routing and RAG retrieval quality
• Perform prompt-injection and PII-leakage testing
• Validate groundedness, hallucinations, and citations in LLM-generated responses
• Conduct LLM evaluations using RAGAS or DeepEval
• Perform automated regression testing using JSONL datasets
• Use Postman and Newman for API testing
• Integrate automated tests with Jenkins CI/CD
• Use Docker and Git in test automation activities
• Monitor applications using Application Insights or OpenTelemetry

WHAT YOU WILL GET WITH ELEKS

• Close cooperation with a customer
• Challenging tasks
• Ability to influence project technologies
• Team of professionals
• Dynamic environment with low level of bureaucracy

ELEKS is a custom software development company. We deliver value to our clients, thanks to our expertise and experience gained from working as a software innovation partner since 1991.

Our 2000+ professionals located in the Delivery Centers across Eastern Europe and sales offices in Europe and North America, provide our clients with a full range of software engineering services. These include product development, QA, R&D, design, technology consulting and dedicated teams.