Sr Data Engineer x 4

SearchLabs; Ontario; Remote; CAD 80,000 - 120,000; Full time

Job summary

An innovative firm is looking for a Senior Data Engineer to design and maintain scalable data pipelines on Azure. This role offers the chance to work with cutting-edge technologies like Azure Databricks and Data Lake Storage. You'll be responsible for optimizing data processing, ensuring data governance, and mentoring junior engineers. Join a dynamic team where your expertise in Azure cloud technologies will drive impactful data solutions and foster collaboration across departments. If you're passionate about data engineering and eager to innovate, this opportunity is perfect for you!

Qualifications

• 5+ years of data engineering experience with Azure cloud technologies.
• Expertise in Azure Databricks, ADF, and CDC implementation.

Responsibilities

• Design and optimize scalable data pipelines using Azure Databricks and ADF.
• Implement robust data ingestion and transformation solutions with PySpark.

Skills

Azure cloud technologies

Data engineering

Azure Databricks

ADLS

Azure Data Factory (ADF)

PySpark

Delta Lake

Change Data Capture (CDC)

Azure SQL

CI/CD best practices

Tools

Databricks Auto Loader

Terraform

Job description

Location: Remote

Contract Duration: May – End of the Year

Job Overview:

We are seeking a highly skilled Senior Data Engineer to design, build, and maintain scalable and efficient data pipelines on Azure Databricks and Azure Data Lake Storage (ADLS). The ideal candidate will have expertise in Azure Data Factory (ADF), PySpark, Databricks Auto Loader, Delta Lake, and Change Data Capture (CDC) to optimize both real-time and batch data processing. This role requires strong Azure SQL skills, data governance expertise, and the ability to mentor junior engineers while collaborating with cross-functional teams.

Key Responsibilities:

1. Data Pipeline Development: Design, build, and optimize scalable data pipelines using Azure Databricks, ADLS, and Azure Data Factory (ADF).
2. Data Ingestion & Transformation: Implement robust data ingestion and transformation solutions using PySpark, Databricks Auto Loader, and Delta Lake to ensure high performance and reliability.
3. Change Data Capture (CDC): Develop and optimize real-time CDC mechanisms to efficiently track and process data changes across multiple systems.
4. Azure SQL & Query Optimization: Utilize Azure SQL for structured data management, optimizing queries, indexing, and partitioning for efficiency.
5. Data Governance & Security: Implement role-based access controls, encryption, and compliance policies to secure data and meet regulatory standards.
6. Monitoring & Performance Tuning: Continuously monitor, troubleshoot, and fine-tune ADF pipelines to optimize workflow performance and minimize latency.
7. Collaboration & Stakeholder Engagement: Work closely with data engineers, analysts, and business stakeholders to understand data requirements and deliver scalable, high-quality solutions.
8. CI/CD & Automation: Implement CI/CD best practices for data pipelines, ensuring automated deployments and efficient continuous integration/delivery.
9. Mentorship & Knowledge Sharing: Provide guidance and mentorship to apprentice and junior engineers, fostering a culture of learning and growth.
10. Continuous Improvement: Stay up to date with emerging Azure technologies, industry trends, and best practices to enhance data engineering solutions and drive innovation.

If this sounds like you, keep reading – this opportunity might be your perfect fit!

Requirements

Required Skills & Qualifications:

1. 5+ years of experience in data engineering, with a strong background in Azure cloud technologies.
2. Expertise in Azure Databricks, ADLS, ADF, PySpark, Databricks Auto Loader, and Delta Lake.
3. Strong experience in real-time and batch data processing, including CDC implementation.
4. Proficiency in Azure SQL, including query optimization, indexing, and partitioning.
5. Solid understanding of data governance, security best practices, and compliance requirements.
6. Experience with CI/CD pipelines and automation for data workflows.
7. Ability to troubleshoot and optimize complex data pipelines for performance and efficiency.
8. Strong problem-solving skills, attention to detail, and ability to work independently.
9. Excellent communication and collaboration skills, with experience mentoring junior engineers.

Nice to Have:

1. Experience with streaming technologies such as Azure Event Hubs or Kafka.
2. Familiarity with machine learning data pipelines and MLOps.
3. Knowledge of Terraform or Infrastructure as Code (IaC) for Azure resources.