Azure Data Engineer: Databricks, ADF & Pipelines

Novacruitix; Minnesota; On-site; USD 120,000 - 160,000; Full time

Job summary

Novacruitix is seeking a data engineering expert in Minnesota to design and optimize scalable data pipelines using Databricks, Azure Data Factory, and PySpark. Build AI-driven insights with code and low-code tools, while modeling data with star/snowflake schemas and open formats like Delta Lake and Iceberg.

Collaborate with stakeholders to deliver performant data products, leveraging Azure cloud services. Leadership and agile delivery are highly valued in this role.

Qualifications

• Proficiency in Snowflake data modeling, querying, and performance optimization.
• Strong experience in Azure Data Factory for data integration and orchestration.
• Hands-on knowledge of Databricks for data engineering, processing, and ML.
• Ability to design, develop, and optimize complex ETL data pipelines.
• Strong problem-solving skills and ability to troubleshoot data pipelines.
• Excellent communication to interact with technical and non-technical stakeholders.
• Strong leadership to guide technical teams towards project delivery.
• Experience in Agile methodologies and project management practices.

Responsibilities

• Lead design and optimization of end-to-end data pipelines.
• Collaborate with data scientists and analysts to deliver reliable data products.
• Implement data models using Star/Snowflake schemas and modern open formats.
• Develop notebooks and jobs in Databricks to support ETL/ELT processes.
• Mentor team members and manage agile project milestones.

Skills

Leadership

Communication

Agile methodology

Project management

Problem solving

Tools

Snowflake

Azure Data Factory

Databricks

PySpark

Python

Databricks SQL

GitHub Copilot

Delta Lake

Iceberg

Databricks Genie

Job description

Novacruitix is seeking a data engineering expert in Minnesota to design and optimize scalable data pipelines using Databricks, Azure Data Factory, and PySpark. Build AI-driven insights with code and low-code tools, while modeling data with star/snowflake schemas and open formats like Delta Lake and Iceberg.

Collaborate with stakeholders to deliver performant data products, leveraging Azure cloud services. Leadership and agile delivery are highly valued in this role.