ETL Testing

Kumaran Systems; Toronto; On-site; CAD 90,000 - 130,000; Full time

Job summary

Kumaran Systems in Toronto is seeking an ETL QA Engineer with strong Databricks and PySpark experience to validate large-scale data pipelines and ensure data quality. You will work with AI-assisted development and prompt engineering, crafting tests that leverage CI/CD pipelines and AI tools.

The role emphasizes building and validating AI-powered knowledge bases, implementing robust QA practices, and collaborating with data engineers and AD teams to improve testing coverage and delivery speed.

Qualifications

• Hands-on Databricks experience (notebooks, jobs, clusters, Delta Lake, SQL).
• Advanced PySpark skills including transformations, actions, window functions, and optimization.
• Strong understanding of ETL/ELT concepts, data warehousing, and data modeling (star/snowflake schemas).
• Experience with data validation, data quality checks, and reconciliation techniques.
• Practical experience with GitHub, GitHub Copilot, and GitHub Actions/Runners for CI/CD pipelines.
• Familiarity with AI & prompt engineering; crafting prompts for LLMs for coding, testing, and documentation.
• Exposure to agent-based architectures or workflow/agent frameworks for AI tasks.
• Knowledge of AI-based tools for code analysis, test generation, or knowledge management.
• Experience building AI-driven knowledge bases or documentation systems.
• Strong documentation skills to translate complex data/QA concepts into knowledge articles.
• Solid QA/testing practices including test planning, design, defect management, and regression testing.
• Experience with Python-based testing libraries (pytest/unittest).
• Familiarity with data quality tools/practices (validation rules, thresholds, anomaly detection).

Responsibilities

• Validate large-scale data pipelines and ensure data quality across datasets.
• Leverage AI-assisted development, prompt engineering, and CI/CD tools to improve testing efficiency.
• Assist in building and validating AI-powered knowledge bases and documentation systems.
• Design and implement data validation rules, thresholds, and anomaly detection.
• Collaborate with data engineers, BI/operations teams, and AD teams to improve QA coverage.

Skills

Databricks

PySpark

ETL/ELT concepts

Data warehousing

Data modeling

Data validation

Data quality checks

Reconciliation techniques

GitHub

GitHub Copilot

GitHub Actions/Runners

AI & Prompt Engineering

LLM prompts

Agent-based architectures

AI-based code analysis/tools

Knowledge base & documentation

Testing & QA practices

pytest/unittest

Data quality tools

Tools

GitHub

GitHub Copilot

GitHub Actions

Job description

We are looking for an ETL QA Engineer with strong experience in Databricks and PySpark, combined with hands-on exposure to AI-assisted development and prompt engineering. The ideal candidate will validate large-scale data pipelines, ensure data quality, and leverage tools like GitHub Copilot, GitHub Actions/Runners, and AI agents to improve testing efficiency and coverage. You will also help build and validate AI-powered knowledge bases.

Required Skills & Experience

• Core Technical Skills
• Strong hands-on experience in Databricks (notebooks, jobs, clusters, Delta Lake, SQL).
• Expert-level PySpark skills, including transformations, actions, window functions, and optimization techniques.
• Solid understanding of ETL/ELT concepts, data warehousing, and data modeling (star/snowflake schemas, dimension/fact tables).
• Proven experience in data validation, data quality checks, and reconciliation techniques.
• Practical experience with GitHub, GitHub Copilot, and GitHub Actions/Runners for CI/CD pipelines.
• AI & Prompt Engineering
• Experience crafting effective prompts for large language models (LLMs) for coding, testing, and documentation.
• Exposure to agent-based architectures or tools (e.g., workflow/agent frameworks that orchestrate multi-step AI tasks).
• Familiarity with AI-based tools for code analysis, test generation, or knowledge management.
• Knowledge Base & Documentation
• Experience building AI-driven knowledge bases or documentation systems (e.g., using vector search, embeddings, or LLM-based retrieval).
• Strong documentation skills, with the ability to translate complex data/QA concepts into clear knowledge articles.
• Testing & QA Practices
• Strong understanding of QA methodologies: test planning, test design, defect management, and regression testing.
• Experience with automated testing frameworks (Python-based testing libraries such as pytest, unittest, or similar).
• Familiarity with data quality tools/practices (e.g., validation rules, thresholds, anomaly detection).

Key Attributes

• Strong analytical and problem-solving skills with a detail-oriented mindset.
• Passion for data quality, automation, and continuous improvement.
• Ability to work in an agile, fast-paced environment and collaborate across multiple teams like Business and operations team and AD team.
• Curiosity and openness to adopting new AI tools and practices for QA.