Databricks Engineer

CG Consulting Group; Mississauga; Hybrid; CAD 90,000 - 120,000; Full time

Job summary

A leading IT services firm is seeking a Databricks Engineer to design and optimize large-scale data pipelines. The role involves collaboration with teams to deliver data solutions, ensuring scalability and governance. This permanent, full-time position offers a competitive salary and benefits, including flexible hybrid work arrangements.

Benefits

Competitive salary and benefits

Paid annual leave: 15 days plus 10 personal/floating/sick days

Comprehensive insurance: medical, dental, vision, life, disability

Flexible hybrid work policy

RRSP with employer contribution up to 4%

Higher education certification policy

On-demand Udemy for Business access

Coaching opportunities with experienced teams

Qualifications

• Strong expertise in Databricks and Azure Cloud Services.
• Solid understanding of Spark and PySpark for big data processing.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks.
• Optimize Databricks workloads for cost efficiency and performance.
• Implement data security and governance standards.

Skills

Databricks

Azure Cloud Services

Spark

PySpark

Data Governance

Data Security

Education

Azure Data Engineer Associate

Databricks Certified Data Engineer Associate

Tools

GitLab

Databricks Asset Bundles

Job description

Databricks Engineer

This position is with a large IT services firm for a major Canadian bank and its global markets division.

Candidate must have legal work status for Canada.

Client Location: Mississauga, Ontario.

Work Arrangement: Hybrid - In-Office 3 days per week.

Role Type: Permanent, full-time role with salary and benefits.

The challenge

We are seeking a highly skilled Azure Data Engineer with strong expertise in Databricks to join our data team. The ideal candidate will design, implement, and optimize large-scale data pipelines, ensuring scalability, reliability, and performance. This role involves working closely with multiple teams and business stakeholders to deliver cutting-edge data solutions.

The Role

Responsibilities:

1. Data Pipeline Development: Build and maintain scalable ETL/ELT pipelines using Databricks. Leverage PySpark/Spark and SQL to transform and process large datasets. Integrate data from multiple sources including Azure Blob Storage, ADLS, and other relational/non-relational systems.
2. Collaboration & Analysis: Work closely with multiple teams to prepare data for dashboards and BI tools. Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.
3. Performance & Optimization: Optimize Databricks workloads for cost efficiency and performance. Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
4. Governance & Security: Implement and manage data security, access controls, and governance standards using Unity Catalog. Ensure compliance with organizational and regulatory data policies.
5. Deployment: Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks, and configurations across environments. Manage version control for Databricks artifacts and collaborate with the team to maintain development best practices.

Requirements

• Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Delta Live Pipelines, Databricks Runtime, etc.).
• Proficiency in Azure Cloud Services.
• Solid understanding of Spark and PySpark for big data processing.
• Experience with relational databases.
• Knowledge of Databricks Asset Bundles and GitLab.

Preferred Experience

• Familiarity with Databricks Runtimes and advanced configurations.
• Knowledge of streaming frameworks like Spark Streaming.
• Experience in developing real-time data solutions.

Certifications

Azure Data Engineer Associate or Databricks Certified Data Engineer Associate certification (optional).

Preferred but not required:

• Detail-oriented with the ability to deliver quickly and accurately.
• Strong analytical skills to interpret business requirements and produce design documents.
• Good time management skills to prioritize and multitask.
• Desire to learn and understand domain specifics.
• Experience in financial services or banking industry.
• Ability to work in a fast-paced environment, be flexible, and learn quickly.
• Attention to detail and multitasking abilities.

Client offers

• Global organization with 58 offices in 21 countries, offering opportunities to work abroad.
• Competitive salary and benefits.
• Paid annual leave: 15 days plus 10 personal/floating/sick days.
• Comprehensive insurance: medical, dental, vision, life, disability.
• Flexible hybrid work policy.
• RRSP with employer contribution up to 4%.
• Higher education certification policy.
• On-demand Udemy for Business with access to 5000+ courses.
• Coaching opportunities with experienced teams.
• Work on cutting-edge projects with top-tier financial institutions.
• Inclusive, diverse, and global work culture.