Azure Data Engineer - Cloud Pipelines & Data Lakes

MACHINE LEARNING TECHNOLOGIES LLC; Grand Rapids (MI); On-site; USD 110,000 - 150,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC is seeking an Azure Data Engineer in Grand Rapids, MI to design, build, and optimize cloud-based data pipelines and data lakes. You will implement ETL/ELT workflows, data transformations, and secure data platforms for enterprise-scale analytics and warehousing.

The role requires expertise in Azure services, data modeling, and collaboration with analysts and developers to meet data requirements and migration efforts to Azure cloud.

Qualifications

• Expertise in Azure Data Factory for data pipeline development and maintenance.
• Experience with Azure Data Lake Storage and enterprise-scale data lakes.
• Experience developing ETL/ELT workflows for data ingestion from multiple sources.
• Knowledge of Mapping Data Flows, SQL, and Spark technologies for data transformation.
• Experience with incremental data loads, Change Data Capture (CDC), and data validation.
• Ability to troubleshoot and optimize data pipelines for performance and reliability.
• Knowledge of data modeling, data warehousing, and cloud data platform security and governance.
• Experience supporting on-premises data migration to Azure cloud environments.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory.
• Build and manage enterprise-scale data lakes using Azure Data Lake Storage.
• Develop ETL/ELT workflows to ingest data from multiple sources, including databases, APIs, flat files, and cloud applications.
• Implement data transformation processes using Mapping Data Flows, SQL, and Spark technologies.
• Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
• Configure incremental data loads, Change Data Capture (CDC), and data validation processes.
• Collaborate with data analysts, business stakeholders, and development teams to understand data requirements.
• Design data models and support data warehousing solutions.
• Implement security, governance, and access controls for cloud data platforms.
• Create technical documentation and maintain data architecture standards.
• Support the migration of on-premises data solutions to Azure cloud environments.

Skills

Azure Data Factory

Azure Data Lake Storage

ETL/ELT

Mapping Data Flows

SQL

Spark

Data modeling

Data warehousing

Cloud security & governance

Tools

Azure

Data Factory

Data Lake Storage

Spark

SQL

Job description

MACHINE LEARNING TECHNOLOGIES LLC is seeking an Azure Data Engineer in Grand Rapids, MI to design, build, and optimize cloud-based data pipelines and data lakes. You will implement ETL/ELT workflows, data transformations, and secure data platforms for enterprise-scale analytics and warehousing.

The role requires expertise in Azure services, data modeling, and collaboration with analysts and developers to meet data requirements and migration efforts to Azure cloud.