Data Engineer

Datavail; Northern (KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Datavail is seeking an Azure Data Engineer with 6+ years of experience to design, develop, and optimize enterprise-scale data platforms on Microsoft Azure and Snowflake. You will build scalable ETL/ELT pipelines using ADF, Databricks, and Snowflake to ingest, transform, and load data from various sources.

You will develop data warehouses and analytical datasets supporting reporting, BI, and analytics initiatives while implementing data governance and CI/CD practices.

Qualifications

• Strong experience designing enterprise-scale data platforms on Azure and Snowflake.
• Proficiency with ADF, Databricks, Synapse, Snowflake, Python and SQL.
• Experience in data modeling, ETL/ELT development, governance, and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data solutions on Azure and Snowflake.
• Build and优化 ETL/ELT pipelines to ingest, transform, and load data from multiple sources.
• Develop enterprise data warehouses and analytical datasets for reporting and BI.
• Implement data quality, governance, and security controls.
• Collaborate with stakeholders, analysts, and architects to translate requirements.
• Automate deployments using Azure DevOps, Git, and CI/CD pipelines.
• Monitor performance and optimize pipelines for reliability and cost efficiency.
• Participate in Agile ceremonies and produce technical documentation.

Skills

Data modeling

ETL/ELT development

CI/CD practices

Agile & Scrum

Data governance & security

Collaboration with stakeholders

Tools

Azure Data Factory (ADF)

Azure Databricks

Azure Synapse Analytics

Snowflake Data Cloud

Python

PySpark

SQL

Azure Data Lake Storage (ADLS Gen2)

Azure Functions

Delta Lake

Power BI Integration

Azure DevOps

Git

Job description

About the Role

• Azure Data Engineer with 6+ years of experience designing, developing, and optimizing enterprise-scale data platforms using Microsoft Azure and Snowflake


• Proven expertise in building scalable data pipelines, data warehouses, and analytics solutions leveraging Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, Snowflake, Python, SQL, and Spark


• Experienced in data modeling, ETL/ELT development, cloud migration, performance tuning, data governance, and implementing CI/CD practices


• Strong background in integrating data from multiple sources and enabling business intelligence and advanced analytics solutions

Key Skills

• Azure Data Factory (ADF)


• Azure Databricks


• Azure Synapse Analytics


• Snowflake Data Cloud


• Python, PySpark, SQL


• Azure Data Lake Storage (ADLS Gen2)


• Azure Functions


• Delta Lake


• Data Warehousing & Data Modeling


• ETL / ELT Development


• CI/CD (Azure DevOps, Git)


• Performance Tuning & Optimization


• Data Governance & Security


• Power BI Integration


• Agile & Scrum Methodologies

What you will do:

• Design, develop, and maintain scalable data engineering solutions on Azure and Snowflake platforms


• Build and optimize ETL/ELT pipelines using Azure Data Factory, Databricks, and Snowflake to ingest, transform, and load data from various structured and unstructured sources


• Develop and maintain enterprise data warehouses, data marts, and analytical datasets supporting reporting, BI, and advanced analytics initiatives
  Implement data transformation and orchestration processes using PySpark, SQL, Python, and Snowflake stored procedures


• Design and manage cloud-based data lakes using Azure Data Lake Storage (ADLS Gen2) and Delta Lake architecture


• Collaborate with business stakeholders, data analysts, and architects to gather requirements and translate them into scalable data solutions


• Monitor and optimize data pipeline performance, ensuring reliability, scalability, and cost efficiency


• Implement data quality frameworks, validation rules, and monitoring processes to ensure accurate and trusted data


• Manage Snowflake objects including databases, schemas, warehouses, tasks, streams, and data-sharing capabilities


• Develop and maintain dimensional and normalized data models following industry best practices


• Implement security controls, access management, encryption, and governance policies across Azure and Snowflake environments


• Support cloud migration and modernization initiatives from on-premises or legacy platforms to Azure and Snowflake


• Automate deployment processes using Azure DevOps, Git repositories, and CI/CD pipelines


• Troubleshoot production issues, perform root cause analysis, and provide timely resolution for data-related challenges


• Participate in Agile ceremonies, code reviews, and technical design discussions to ensure high-quality solution delivery


• Create technical documentation, architecture diagrams, operational runbooks, and knowledge-transfer materials

Preferred Certifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203)


• SnowPro Core Certification