Microsoft Fabric & Airflow Resource

TechDoQuest; Brampton; On-site; CAD 110,000 - 150,000; Full time

Job summary

TechDoQuest in Brampton, ON is seeking a data engineering lead to design, build, and optimize data pipelines using Fabric Data Factory and orchestrate workloads with Fabric Airflow. You will manage Lakehouse assets in OneLake and implement semantic models with Oracle mirroring, ensuring integration with Databricks and Snowflake.

Strong Azure DevOps and Git skills are essential. You will mentor offshore engineers, monitor usage with Fabric Capacity Metrics, and maintain thorough technical

Qualifications

• Strong experience with Data Fabric technologies.
• Hands-on experience building and tuning pipelines in Fabric Data Factory.
• Hands-on experience with Fabric Airflow for workload orchestration.
• Expertise managing Lakehouse and Warehouse items in OneLake.
• Solid understanding of cloud storage solutions: ADLS Gen2, AWS S3, and GCP Cloud Storage.
• Proficiency with Spark, PySpark, SQL, T SQL, and Delta Lake.
• Experience with Direct Lake semantic model configuration and Oracle mirroring.
• Familiarity with Databricks and Snowflake for interoperability testing.
• Fluency in Azure DevOps and Git for version control and CI/CD.
• Experience using the Fabric Capacity Metrics app for monitoring.

Responsibilities

• Lead offshore engineering pod with mentorship and guidance.
• Design, build, and optimize data pipelines in Fabric Data Factory.
• Orchestrate workloads using Fabric Airflow for efficient processing.
• Manage Lakehouse and Warehouse items in OneLake and configure storage shortcuts.
• Implement Direct Lake semantic models and Oracle mirroring setups.
• Utilize Spark, PySpark, SQL, T-SQL, and Delta Lake for data processing.
• Coordinate interoperability tests with Databricks and Snowflake.
• Use Azure DevOps and Git for CI/CD of Fabric artifacts.
• Monitor consumption with Fabric Capacity Metrics and benchmark discipline.
• Document technical processes and maintain clean documentation.

Skills

Data Fabric knowledge

Fabric Data Factory pipelines

Fabric Airflow orchestration

OneLake lakehouse management

Cloud storage: ADLS Gen2 / S3 / GCS

Spark / PySpark / SQL / T-SQL / Delta

Direct Lake semantic model & Oracle

Databricks & Snowflake testing

Azure DevOps & Git CI/CD

Fabric Capacity Metrics monitoring

Interoperability testing

Tools

Git

Azure DevOps

CI/CD tooling

Delta Lake tooling

Job description

This job posting is for an existing, active vacancy and we are looking to hire Microsoft Fabric & Airflow Resource at Brampton, ON.

Responsibilities:

• Lead the offshore engineering pod, providing mentorship and guidance to team members.
• Design, build, and optimize data pipelines in Fabric Data Factory.
• Orchestrate workloads using Fabric Airflow to ensure efficient data processing.
• Manage Lakehouse and Warehouse items in OneLake, including configuration of shortcuts to various cloud storage solutions.
• Implement and maintain Direct Lake semantic model configurations and Oracle mirroring setups.
• Utilize Spark, PySpark, SQL, T SQL, and Delta Lake for data processing and transformation.
• Conduct interoperability testing with Databricks and Snowflake to ensure seamless integration.
• Utilize Azure DevOps and Git for version control and CI/CD of Fabric artifacts.
• Monitor consumption using the Fabric Capacity Metrics app and perform disciplined benchmarking.
• Document technical processes and maintain clean technical documentation.
• Coordinate across time zones with an onsite architect to ensure project alignment.

Mandatory Skills:

• Strong experience with Data Fabric technologies.
• Proficient in building and tuning pipelines in Fabric Data Factory.
• Hands on experience with Fabric Airflow for workload orchestration.
• Expertise in managing Lakehouse and Warehouse items in OneLake.
• Solid understanding of cloud storage solutions: ADLS Gen2, AWS S3, and GCP Cloud Storage.
• Proficient in Spark, PySpark, SQL, T SQL, and Delta Lake.
• Experience with Direct Lake semantic model configuration and Oracle mirroring setup.
• Familiarity with Databricks and Snowflake for interoperability testing.
• Fluency in Azure DevOps and Git for version control and CI/CD.
• Experience using the Fabric Capacity Metrics app for monitoring.

Preferred Skills:

• Experience in mentoring and leading engineering teams.
• Strong analytical and problem-solving skills.
• Excellent communication skills, both verbal and written.