Azure Data Engineer

Valiant Technologies LLC; Grand Rapids (MI); On-site; USD 110,000 - 150,000; Full time

Job summary

Valiant Technologies LLC is seeking a data engineer to design, build, and optimize reliable data pipelines on Azure. You will work with Azure Data Factory, Data Lake Storage, and cloud analytics to support enterprise data lakes and data warehousing initiatives.

The role involves CDC, incremental loads, data transformations with Spark and SQL, and collaboration with analysts and developers to translate data requirements into scalable architectures with strong emphasis on security and governance.

Qualifications

• Experience designing and maintaining data pipelines in cloud environments.
• Strong understanding of data modeling and data warehousing concepts.
• Ability to implement data governance and security controls for cloud platforms.
• Experience with incremental data loads and Change Data Capture (CDC).
• Experience migrating on-prem data solutions to cloud environments.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory.
• Build and manage enterprise-scale data lakes with Azure Data Lake Storage.
• Develop ETL/ELT workflows from multiple sources (databases, APIs, flat files, cloud apps).
• Implement transformations using Mapping Data Flows, SQL, and Spark.
• Monitor, troubleshoot, and optimize pipelines for performance and reliability.
• Configure CDC and data validation processes for data accuracy.
• Collaborate with analysts and stakeholders to understand data needs.
• Design data models and support data warehousing solutions.
• Ensure security, governance, and access controls on cloud data platforms.
• Create technical documentation and maintain data architecture standards.
• Support migration of on-premises data solutions to Azure.

Skills

Data pipelines

Cloud data engineering

Data modeling

Governance & security

Tools

Azure Data Factory

Azure Data Lake Storage

Spark

SQL

Job description

This role focuses on developing reliable data pipelines, managing enterprise-scale data lakes, implementing ETL/ELT workflows, and supporting cloud-based analytics and data warehousing solutions.

The ideal candidate brings expertise in Azure Data Factory, Azure Data Lake Storage, data transformation technologies, and cloud data engineering practices.

Key Responsibilities:

• Design, develop, and maintain data pipelines using Azure Data Factory.
• Build and manage enterprise-scale data lakes using Azure Data Lake Storage.
• Develop ETL/ELT workflows to ingest data from multiple sources, including databases, APIs, flat files, and cloud applications.
• Implement data transformation processes using Mapping Data Flows, SQL, and Spark technologies.
• Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
• Configure incremental data loads, Change Data Capture (CDC), and data validation processes.
• Collaborate with data analysts, business stakeholders, and development teams to understand data requirements.
• Design data models and support data warehousing solutions.
• Implement security, governance, and access controls for cloud data platforms.
• Create technical documentation and maintain data architecture standards.
• Support the migration of on-premises data solutions to Azure cloud environments.

Required Qualifications:

• Expertise in Azure Data Factory for data pipeline development and maintenance.
• Experience with Azure Data Lake Storage and enterprise-scale data lakes.
• Experience developing ETL/ELT workflows for data ingestion from multiple sources.
• Knowledge of Mapping Data Flows, SQL, and Spark technologies for data transformation.
• Experience with incremental data loads, Change Data Capture (CDC), and data validation.
• Ability to troubleshoot and optimize data pipelines for performance and reliability.
• Knowledge of data modeling, data warehousing, and cloud data platform security and governance.
• Experience supporting on-premises data migration to Azure cloud environments.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.