Senior Quality Engineer – Test Automation, AI Testing & Java/Python

Astra-North Infoteck Inc. ~ Conquering today’s challenges, achieving tomorrow’s vision!; Toronto; On-site; CAD 90,000 - 130,000; Full time

Job summary

Astra-North Infoteck Inc. is seeking a Quality Engineer to lead test automation for AI-enabled features and robust QA across functional, API, and data layers.

You will design strategies, data sets, and frameworks in Java/Python, collaborating with DevOps and product teams in a fast-paced Agile environment. You will build scalable automation suites using Selenium, Playwright, and REST tools, address AI-specific quality concerns, and drive continuous improvement through metrics and risk-based

Qualifications

• 5+ years in Quality Engineering or QA automation.
• 5+ years programming in Java, Python, Groovy, or a comparable language.
• 5+ years building test automation frameworks.
• Hands-on with Selenium, Cucumber/Gherkin, Playwright, RestAssured.
• Experience testing APIs, databases, microservices, and regression suites.
• Knowledge of TDD/ATDD/BDD and CI/CD.
• Experience with AI testing and AI quality concepts.

Responsibilities

• Design and execute automated, exploratory, and AI-focused tests.
• Define test plans, data sets, and quality strategies.
• Develop scalable test automation frameworks.
• Validate AI integrations and data pipelines.
• Integrate testing into CI/CD with DevOps.
• Identify defects and track through resolution.
• Communicate results and risks to stakeholders.
• Collaborate with cross-functional Agile teams.

Skills

Java

Python

Groovy

Selenium

Playwright

Cucumber

TestNG

JUnit

API testing

Test automation

AI testing

Tools

Selenium

Cucumber/Gherkin

Playwright

RestAssured

JUnit

TestNG

ReadyAPI

JMeter

Job description

Quality Engineer – Test Automation, Java/Python & GenAI

Roles and Responsibilities:

• Partner with DevOps, Development, Product, and QE teams to design and execute exploratory, automated, and AI-focused tests that ensure product quality.
• Define quality strategies, test plans, test cases, and test data for functional, integration, system, regression, API, database, and acceptance testing.
• Design, develop, and maintain scalable test automation frameworks using Java, Python, or Groovy.
• Build automated tests for AI-enabled features, including prompt behavior, response accuracy, hallucination risks, safety, reliability, latency, and regression coverage.
• Validate AI integrations, APIs, microservices, data pipelines, model interactions, and third-party services.
• Establish testing approaches for both deterministic and non-deterministic AI behavior, including evaluation datasets, expected behavior, quality thresholds, and repeatability.
• Integrate automated testing and AI quality checks into CI/CD pipelines in partnership with DevOps teams.
• Use tools and frameworks such as Selenium, Cucumber/Gherkin, Playwright, RestAssured, JUnit, TestNG, ReadyAPI, and JMeter.
• Identify, document, prioritize, and track defects through resolution.
• Analyze test coverage, quality trends, AI evaluation results, and production feedback to identify gaps and drive continuous improvement.
• Collaborate with cross-functional Agile teams to deliver reliable, secure, and user-focused product features.
• Communicate test results, risks, quality metrics, and release recommendations clearly to technical and non-technical stakeholders.

Required Skills:

• 5+ years of relevant experience in Quality Engineering, Quality Assurance, test automation, or software development.
• 5+ years of programming experience with Java, Python, Groovy, or a comparable language.
• 5+ years of experience building and maintaining test automation frameworks.
• Hands-on experience with Selenium, Cucumber/Gherkin, Playwright, RestAssured, or similar automation technologies.
• Experience testing functional and non-functional requirements, APIs, integrations, databases, microservices, and regression suites.
• Experience with testing frameworks and tools such as JUnit, TestNG, qTest, ReadyAPI, and JMeter.
• Understanding of TDD, ATDD, BDD, continuous testing, and risk-based testing practices.
• Experience testing AI or machine learning-powered applications, APIs, conversational systems, or generative AI features.
• Understanding of AI quality concepts such as prompt evaluation, grounding, accuracy, relevance, toxicity, bias, hallucination detection, and response consistency.
• Ability to create effective test data and evaluation datasets for AI-driven functionality.
• Strong analytical, problem-solving, communication, and technical documentation skills.
• Ability to work effectively in a fast-paced, cross-functional Agile environment.

Nice to Have:

• Experience with AI evaluation frameworks, LLM testing, prompt engineering, retrieval-augmented generation, embeddings, or model observability.
• Experience with Git, Bitbucket, Jenkins, CI/CD pipelines, Artifactory, SonarQube, Jira, Confluence, AWS, Kubernetes, Maven, or Gradle.
• Experience testing cloud-native applications and distributed microservice architectures.
• Experience with security, performance, accessibility, or reliability testing.
• Experience developing automation scripts and reusable testing utilities.
• Experience with Tableau or other analytics and visualization platforms.
• CSTE certification or an equivalent quality engineering certification.
• Experience working directly with DevOps teams to implement automated quality gates and release controls.