Senior Data Engineer

Sobeys; Stellarton; Hybrid; CAD 110,000 - 160,000; Full time

Job summary

Sobeys is seeking a Senior Data Engineer to join the Data Integration Hub within Sobeys' Integration and COEs. The role can be based in Stellarton, NS or Mississauga, ON and is designated as hybrid.

You will translate data requirements into design and implementation, working closely with business SMEs, architects and technical leads across Sobeys. The role focuses on Azure Databricks, streaming, Delta Live Tables, and optimizing performance and cost.

Qualifications

• 5+ years developing with Spark in Azure Databricks.
• Experience with Delta Live Tables and structured streaming.
• Experience building scalable data ingestion frameworks.
• Experience enabling and using Unity Catalog.
• Experience building resilient pipelines for operational systems.
• Collaborative and able to align with architects and core teams.
• Experience in performance and cost optimization of data solutions.
• Experience with Azure DevOps for CI/CD.
• Test-driven development experience.
• Azure Databricks certifications are a plus.

Responsibilities

• Translate data requirements into design and implementation within the Data Integration Hub.
• Mentor junior engineers and act as trusted advisor to leadership.
• Collaborate with SMEs, architects and technical leads across Sobeys divisions.
• Drive performance and cost optimization for data pipelines.
• Promote automation and reduce manual repetitive work.

Skills

Spark development

Azure Databricks

Delta Live Tables

Structured streaming

Data ingestion frameworks

Unity catalog

CI/CD

Test-driven development

Azure DevOps CI/CD

Certifications (Azure Databricks)

Tools

Azure Databricks notebooks

Job description

Sobeys is full of exciting opportunities, and we are always looking for bright new talent to join our team! We currently have a full time opportunity for a Senior Data Engineer role. This role can be based out of our offices located in Stellarton; NS, Mississauga, ON Senior Data Engineer role will be part of the Data Integration Hub (DIH) within the 'Integration and COEs' department of Sobeys. Part of the enhancement and maintenance of DIH to support major program deliverables. Work closely with business SMEs, architects and technical leads from various divisions within Sobeys. Highly experienced in translating data requirements to design and implementation. Highly skilled in Azure Databricks, specifically streaming and event driven solutions. Highly skilled in performance and cost optimization of solutions. Solid expertise in CI/CD practices. Provides mentorship to junior engineers and serve as trusted advisor to leadership team. #LI-Hybrid

What you have to offer:

• Significant experience (5+) developing with Spark (python/Vscode/Databricks Notebooks preferably) and specifically within Azure Databricks.
• Strong knowledge on DLT framework and structured streaming.
• Experienced in implementing scalable data ingestion frameworks.
• Experienced in enabling and using Unity catalog.
• Experienced in building a resilient solution for pipelines supporting operational systems.
• Experienced in engaging with the team towards a common sprint goal and be collaborative.
• Experienced in establishing Key Design Decisions and communicating with architects and core team for alignment.
• Actively engage in performance and cost optimization of the solution.
• Ability to identify redundant manual work and share opportunities to enable automation.
• Azure Dev Ops - CI/CD.
• Proven experience in test driven development.
• Azure and Databricks certification is a plus.