Remote Automation QA Engineer: Python & AI Validation

Ciklum; Poland; On-site; PLN 130,000 - 200,000; Full time

Benefits offered by this job

Flexible remote working

Luxmed premium medical package

MyBenefit cafeteria access

Udemy access

Language courses

Company-paid certifications

Job summary

Ciklum is seeking an Automation QA Engineer to join our Poland-based team on a full-time basis. You will build and run automated Python validation harnesses for a RAG-based pipeline, integrating scorecards into Azure DevOps and ensuring data integrity across ingestion and vector store retrieval.

Collaborate with ML engineers, focus on reducing hallucinations and improving grounding, and participate in a cross-functional squad delivering high-quality AI-enabled solutions.

Qualifications

• 3+ years of backend test automation, data quality engineering, or Python development.
• Strong Python skills for backend testing and data manipulation; traditional UI automation tools are not in scope.
• Hands-on familiarity with LLM validation frameworks (RAGAS or equivalent automated validation engines).
• Experience configuring automated script execution, regression tests, and quality gates inside Git-based pipelines (Azure DevOps preferred).
• Knowledge of PostgreSQL and search/indexing concepts.

Responsibilities

• Programmatic RAG Evaluation: Build and run automated validation test suites using Python to score pipeline faithfulness, answer relevance, and context precision.
• CI/CD Quality Gates: Integrate automated evaluation scorecards into Azure DevOps pipelines to prevent regression in code-generation quality.
• Data Integrity Validation: Develop automated checks for data ingestion, chunking consistency, and vector store retrieval accuracy.
• Phase-Gate Reporting: Generate programmatic quality scorecards and regression reports for project gate milestones.
• Squad Collaboration: Work with ML engineers to isolate, reproduce, and patch retrieval or chunking issues.

Skills

Python

Backend testing

Data quality

LLM validation

Tools

Azure DevOps

PostgreSQL

RAGAS

Job description

Ciklum is seeking an Automation QA Engineer to join our Poland-based team on a full-time basis. You will build and run automated Python validation harnesses for a RAG-based pipeline, integrating scorecards into Azure DevOps and ensuring data integrity across ingestion and vector store retrieval.

Collaborate with ML engineers, focus on reducing hallucinations and improving grounding, and participate in a cross-functional squad delivering high-quality AI-enabled solutions.