ETL QA

Kumaran; Toronto; On-site; CAD 90,000 - 130,000; Full time

Job summary

Kumaran Systems is seeking an ETL QA Engineer to validate large-scale data pipelines using Databricks and PySpark. The role combines data quality checks, AI-assisted development, and prompt engineering to enhance testing efficiency and coverage.

The ideal candidate contributes to building AI-powered knowledge bases and documentation, while leveraging GitHub Copilot and GitHub Actions to streamline CI/CD pipelines. Toronto-based, on-site environment with modern data platforms.

Qualifications

• Strong hands-on experience with Databricks notebooks, jobs, clusters, Delta Lake, SQL.
• Expert PySpark skills including transformations, actions, window functions, and optimization.
• Solid understanding of ETL/ELT concepts, data warehousing and data modeling.
• Proven experience in datavalidation, data quality checks, and reconciliation techniques.
• Practical experience with GitHub, GitHub Copilot, and GitHub Actions/Runners for CI/CD pipelines.
• Experience crafting prompts for large language models (LLMs) for coding, testing, and documentation.
• Exposure to agent-based architectures or workflow/agent frameworks.
• Familiarity with AI-based tools for code analysis, test generation, or knowledge management.
• Experience building AI-driven knowledge bases or documentation systems.
• Strong documentation skills in translating data/QA concepts into knowledge articles.
• Strong QA methodologies: test planning, design, defect management, regression testing.
• Familiarity with Python-based testing libraries (pytest, unittest).
• Knowledge of data quality tools and validation rules.

Responsibilities

• Validate large-scale data pipelines and ensure data quality across systems.
• Leverage AI-assisted development and prompt engineering to improve testing coverage.
• Contribute to building and validating AI-powered knowledge bases and documentation.
• Implement and maintain CI/CD pipelines using GitHub Actions/Runners.
• Collaborate with cross-functional teams to ensure data integrity and testing efficiency.

Skills

Databricks

PySpark

ETL/ELT

Data validation

GitHub

GitHub Copilot

AI & Prompting

LLM prompts

Agent-based AI

Documentation

Testing (pytest)

Data quality

Tools

GitHub Actions

Vector search

Embeddings

Job description

Kumaran Systems: Engineering Future-Ready Transformations Since 1992, Kumaran Systems has been helping enterprises bridge the past and the future, reimagining legacy systems and embracing AI-led possibilities. With over 34 years of delivery excellence, we specialise in re-engineering core applications, driving cloud transformation, and building automation-first, GenAI-enabled ecosystems across the Banking and Financial Services, Insurance, Telecom, and Automotive sectors. We help organisations not just upgrade their systems but reimagine what’s possible.

Why Kumaran?

We blend engineering discipline with AI innovation, helping clients modernise with confidence, automate with clarity, and scale with purpose. Our global delivery model ensures agility, responsiveness, and seamless collaboration, with clients always at the heart of every engagement. At Kumaran, we don’t just solve problems, we engineer future-ready transformations.

Job Description

Role Overview

We are looking for an ETL QA Engineer with strong experience inDatabricks and PySpark, combined with hands‑on exposure to AI‑assisted development and prompt engineering. The ideal candidate will be responsible for validating large-scale data pipelines, ensuring data quality, and leveraging tools like GitHub Copilot, GitHub Actions/Runners, and AI agents to improve testing efficiency and coverage. You will also contribute to building and validating AI‑powered knowledge bases.

Required Skills & Experience

• CoreTechnical Skills
  
• Strong hands‑on experience in Databricks (notebooks, jobs, clusters, Delta Lake, SQL).
  
• Expert‑level PySpark skills, including transformations, actions, window functions, and optimization techniques.
  
• Solid understanding of ETL/ELTconcepts, data warehousing, and data modeling (star/snowflakeschemas, dimension/fact tables).
  
• Proven experience in datavalidation, data quality checks, and reconciliation techniques.
  
• Practical experience with GitHub, GitHub Copilot, and GitHub Actions/Runners for CI/CDpipelines.
  
• AI &Prompt Engineering
  
• Experience crafting effectiveprompts for large language models (LLMs) for coding, testing, anddocumentation.
  
• Exposure to agent-basedarchitectures or tools (e.g., workflow/agent frameworks thatorchestrate multi-step AI tasks).
  
• Familiarity with AI-basedtools for code analysis, test generation, or knowledge management.
  
• KnowledgeBase & Documentation
  
• Experience building AI-drivenknowledge bases or documentation systems (e.g., using vector search,embeddings, or LLM-based retrieval).
  
• Strong documentation skills,with the ability to translate complex data/QA concepts into clearknowledge articles.
  
• Testing & QA Practices
  
• Strong understanding of QAmethodologies: test planning, test design, defect management, and regression testing.
  
• Experience with automated testing frameworks (Python-based testing libraries such as pytest , unittest , or similar).
  
• Familiarity with data qualitytools/practices (e.g., validation rules, thresholds, anomaly detection).
  

Key Attributes

• Strong analytical andproblem‑solving skills with a detail-oriented mindset.
  
• Passion for data quality,automation, and continuous improvement.
  
• Ability to work in an agile,fast‑paced environment and collaborate across multiple teams like Businessand operations team and AD team.
  
• Curiosity and openness to adopting new AI tools andpractices for QA.
  

Kumaran Systems is an Equal Opportunity Employer and does not discriminate on the basis of race or ethnicity, religion, sex, national origin, age, veteran disability or genetic information or any other reason prohibited by law in employment.