Azure Data Engineer: Pipelines, Data Lakes & CDC

Valiant Technologies LLC; Grand Rapids (MI); Presencial; USD 110.000 - 150.000; Jornada completa

Descripción de la vacante

Valiant Technologies LLC is seeking a data engineer to design, build, and optimize reliable data pipelines on Azure. You will work with Azure Data Factory, Data Lake Storage, and cloud analytics to support enterprise data lakes and data warehousing initiatives.

The role involves CDC, incremental loads, data transformations with Spark and SQL, and collaboration with analysts and developers to translate data requirements into scalable architectures with strong emphasis on security and governance.

Formación

• Experience designing and maintaining data pipelines in cloud environments.
• Strong understanding of data modeling and data warehousing concepts.
• Ability to implement data governance and security controls for cloud platforms.
• Experience with incremental data loads and Change Data Capture (CDC).
• Experience migrating on-prem data solutions to cloud environments.

Responsabilidades

• Design, develop, and maintain data pipelines using Azure Data Factory.
• Build and manage enterprise-scale data lakes with Azure Data Lake Storage.
• Develop ETL/ELT workflows from multiple sources (databases, APIs, flat files, cloud apps).
• Implement transformations using Mapping Data Flows, SQL, and Spark.
• Monitor, troubleshoot, and optimize pipelines for performance and reliability.
• Configure CDC and data validation processes for data accuracy.
• Collaborate with analysts and stakeholders to understand data needs.
• Design data models and support data warehousing solutions.
• Ensure security, governance, and access controls on cloud data platforms.
• Create technical documentation and maintain data architecture standards.
• Support migration of on-premises data solutions to Azure.

Conocimientos

Data pipelines

Cloud data engineering

Data modeling

Governance & security

Herramientas

Azure Data Factory

Azure Data Lake Storage

Spark

SQL

Descripción del empleo

Valiant Technologies LLC is seeking a data engineer to design, build, and optimize reliable data pipelines on Azure. You will work with Azure Data Factory, Data Lake Storage, and cloud analytics to support enterprise data lakes and data warehousing initiatives.

The role involves CDC, incremental loads, data transformations with Spark and SQL, and collaboration with analysts and developers to translate data requirements into scalable architectures with strong emphasis on security and governance.