Senior Databricks Data Engineer

KData Inc.; Brampton; On-site; CAD 150,000 - 210,000; Full time

Job summary

KData Inc. in Brampton East, Canada, is seeking a Senior, hands-on Databricks Data Engineer who lives and breathes code, query optimization, and modern data architecture. You will write production PySpark/SQL, tune Databricks clusters, and build streaming and batch pipelines.

You will own end-to-end pipeline execution from ingestion to curated Gold layer models, driving modernization of our Lakehouse platform and enforcing data governance, quality, and security.

Qualifications

• 8+ years in data engineering with 4+ years on Databricks.
• Fluent in PySpark and Python with SQL proficiency.
• Deep experience with Delta Lake and Unity Catalog.
• Hands-on cloud experience (AWS/Azure/GCP) with Databricks.
• Strong CI/CD and software engineering practices.

Responsibilities

• Hands-On pipeline development and lakehouse architecture.
• Build batch and real-time streaming pipelines.
• Optimize Spark performance and manage costs.
• Implement data governance, security, and QA.
• CI/CD automation and DevOps for Databricks assets.

Skills

Data engineering

PySpark

SQL

Python

CI/CD

Git workflows

Data modeling

Tools

Delta Lake

Delta Live Tables

Unity Catalog

Auto Loader

Databricks Workflows

Airflow

Terraform

Job description

Brampton East, Canada | Posted on 08/13/2026

We are looking for a Senior, Super Hands-On DatabricksData Engineer who lives and breathes code, query optimization, and moderndata architecture. In this role, you won't just design architectures onwhiteboards—you will write production PySpark/SQL, optimize Databricksclusters, build streaming and batch pipelines, and enforce data governance.

You will own end-to-end pipeline execution from rawingestion to curated Gold layer models, playing a lead role in modernizing ourLakehouse platform.

Key Responsibilities

1. Hands-On Pipeline Development & LakehouseArchitecture

• Design,build, and maintain enterprise-scale batch and real-time streamingpipelines using PySpark, SQL, Delta Live Tables (DLT), and AutoLoader.
  
• Implementand refine Medallion Architecture (Bronze Silver Gold) to support downstream BI,reporting, and Machine Learning workloads.
  
• Enforceschema evolution, ACID transactions, and data compaction using DeltaLake core constructs.
  

2. Performance Tuning & Optimization (Deep Tech)

• Diagnoseand resolve Spark performance bottlenecks: data skew, OOM errors,excessive shufflings, and memory spills.
  
• Optimizequeries using Liquid Clustering, Z-Ordering, Data Partitioning, AQE(Adaptive Query Execution), and Photon engine tuning.
  
• Benchmarkand optimize Databricks compute workloads to minimize DBU (DatabricksUnit) consumption and cloud costs (FinOps).
  

3. Governance, Security & Quality

• Implementend-to-end data governance, fine-grained access control (row/column-levelsecurity), and lineage tracking using Unity Catalog.
  
• Automateautomated data quality validation checks and alert mechanisms across thepipeline life cycle.
  

4. Operations, CI/CD & DevOps

• Automatepipeline orchestration using Databricks Asset Bundles (DABs) or DatabricksWorkflows / Apache Airflow.
  
• BuildCI/CD pipelines (GitHub Actions, Azure DevOps, or GitLab) for automatedtesting, deployment, and code promotions.
  

Requirements

Required Skills &Qualifications

Must-Haves

• Experience:8+ years in Data Engineering, with 4+ years of intensive, hands‑onproduction experience on Databricks.
  
• ProgrammingMastery: Fluent in PySpark, Advanced SQL, and Python.
  
• DatabricksEcosystem: Deep experience with Delta Lake, Unity Catalog, DeltaLive Tables (DLT), Auto Loader, and Databricks Workflows.
  
• CloudInfrastructure: Strong hands‑on experience in at least one primarycloud provider (AWS, Azure, or GCP) integration with Databricks(S3/ADLS Gen2, IAM, Key Vaults/Secret Manager).
  
• DataModeling: Solid understanding of dimensional modeling (Kimball), OneBig Table (OBT) strategies, and data vault patterns.
  
• CI/CD& Software Engineering: Proficient in Git workflows, unit testingPySpark code (pytest), and deployment automation.
  

Preferred / Nice-to-Haves

• Certifications: Databricks Certified Data Engineer Professional.
  
• Streaming: Hands‑on with Apache Kafka, Event Hubs, or Kinesis integration viaStructured Streaming.
  
• GenAI/ ML Ops: Familiarity with MLflow, Feature Store, or Vector Searchwithin Databricks.
  
• Infrastructureas Code (IaC): Experience using Terraform to provision Databricksworkspaces and storage resources.
  

Performance Indicators(How success is measured)

• PipelineReliability: Maintaining strict SLA thresholds on critical Gold-layermodels.
  
• CostEfficiency: Measurable reduction in DBU costs through effectivecompute profiling and tuning.
  
• CodeQuality: High test coverage and zero-downtime CI/CD deployments.