GenAI Quality Engineer — LLM/RAG Testing & Automation

Armour Transportation Systems; Dartmouth; On-site; CAD 97,000 - 129,000; Full time

Job summary

Armour Transportation Systems is seeking a GenAI Testing Specialist in Dartmouth to lead end-to-end AI testing efforts. You will design and execute comprehensive test strategies for LLM-powered solutions, RAG pipelines, and agentic AI workflows.

You will validate prompts, APIs, and UIs, build evaluation datasets, and establish rigorous quality gates integrated with CI/CD. A Bachelor's or Master's in CS or related field plus 5+ years in quality engineering is preferred.

Qualifications

• Strong experience in software quality engineering for enterprise apps.
• Hands-on GenAI concepts, LLM behaviour, prompt engineering, embeddings, and RAG architectures.
• Proficiency in Python with PyTest or Robot Framework and API testing libraries.
• Experience evaluating LLM outputs using deterministic checks and human review workflows.
• Knowledge of GenAI evaluation and observability tools such as LangSmith or similar.

Responsibilities

• Develop and execute test strategies for GenAI applications including LLM-powered solutions.
• Validate requirements across prompts, model responses, APIs, and UI integrations.
• Create reusable evaluation datasets, test cases, and scoring rubrics for quality metrics.
• Test RAG solutions for retrieval quality, context recall, and answer fidelity.
• Validate agentic AI behaviour including tool use, reasoning flows, and memory handling.
• Perform adversarial, security, and safety testing for prompt injections and data exposure.
• Conduct performance and scalability testing for latency, throughput, and cost under load.
• Build regression suites in Python and integrate GenAI tests into CI/CD pipelines.
• Monitor production quality and drive defect analysis and remediation with teams.

Skills

GenAI testing concepts

Python-based API & test automation

LLM evaluation & prompt testing

CI/CD & observability

RAG/agentic AI testing

Education

Bachelor's or Master's in CS/Software Eng/Data Science

Tools

PyTest

Robot Framework

GitHub Actions

Docker

Kubernetes

AWS or Azure

Job description

Armour Transportation Systems is seeking a GenAI Testing Specialist in Dartmouth to lead end-to-end AI testing efforts. You will design and execute comprehensive test strategies for LLM-powered solutions, RAG pipelines, and agentic AI workflows.

You will validate prompts, APIs, and UIs, build evaluation datasets, and establish rigorous quality gates integrated with CI/CD. A Bachelor's or Master's in CS or related field plus 5+ years in quality engineering is preferred.