ADF ETL Data Engineer

Compunnel, Inc.; Minneapolis (MN); On-site; USD 80,000 - 110,000; Full time

Job summary

An established industry player is seeking a skilled Data Engineer to lead the development and optimization of ETL pipelines. This role is crucial for managing data flows and ensuring efficient data delivery to end users. You will work closely with the ETL architect to design and automate processes, leveraging tools like Azure Data Factory and Snowflake. If you have a passion for data engineering and a knack for problem-solving, this opportunity offers a chance to make a significant impact in a dynamic environment. Join a forward-thinking company that values innovation and collaboration, and take your career to the next level.

Qualifications

• 5+ years in data engineering with strong focus on data warehousing.
• Expertise in Azure Data Factory and ETL development tools.

Responsibilities

• Develop and optimize ETL pipelines for data warehousing.
• Collaborate with ETL architect to ensure high-quality data delivery.

Skills

Data Engineering

ETL Development

Problem-Solving

Data Analysis

Adaptability

Education

Bachelor’s Degree in IT, Computer Science, Mathematics, Statistics, Analytics, or Business

Advanced Degree in relevant field

Tools

Azure Data Factory

Snowflake

Informatica PowerCenter

SSIS

GitHub

PowerShell

Python

Job description

The Data Engineer will be responsible for developing, managing, and optimizing ETL pipelines and data flows to and from the data warehouse. This role involves working closely with the ETL architect, designing and automating ETL processes, and ensuring high-quality data delivery to end users. The position will focus on utilizing Azure Data Factory, Snowflake, and various other data tools to create efficient and reliable ETL pipelines for processing both structured and unstructured data.

Key Responsibilities:

1. Develop and maintain effective working relationships across departments to ensure smooth coordination and data flow.
2. Communicate efficiently with the ETL architect to understand business requirements and transform data accordingly.
3. Assist in the architecture of ETL design and contribute to designing, implementing, and automating ETL flows.
4. Investigate and identify potential issues in ETL pipelines, proposing appropriate solutions.
5. Develop and manage ETL pipelines in and out of the data warehouse using Azure Data Factory and Snowflake tools.
6. Design and implement idempotent ETL processes to allow for re-runs without errors in case of failures or interruptions.
7. Work with Snowflake Virtual Warehouses and automate data pipelines using Snowpipe.
8. Capture changes in data dimensions, maintain version control, and schedule updates using Snowflake's Stream sets and Tasks.
9. Optimize the performance of data movement both during travel and while at rest in the database.
10. Build and manage efficient orchestration systems for job scheduling, workflow execution, and data quality checks.
11. Conduct testing on ETL system code, data design, and pipelines, and resolve production issues.
12. Document implementations, test cases, and deployment details for CI/CD processes.

Required Qualifications:

1. 5+ years of experience in data engineering, specifically focusing on data warehousing.
2. 2+ years of experience creating pipelines in Azure Data Factory (ADF).
3. 5+ years of experience in ETL development using tools such as Informatica PowerCenter, SSIS, or similar.
4. 5+ years of experience with relational databases like Oracle, Snowflake, and SQL Server.
5. 3+ years of experience writing stored procedures using Oracle PL/SQL, SQL Server T-SQL, or Snowflake SQL.
6. 2+ years of experience with source control tools such as GitHub or SVN.
7. 2+ years of experience processing both structured and unstructured data.
8. Experience with HL7 and FHIR standards and processing files in these formats.
9. 3+ years of experience analyzing project requirements and developing detailed ETL specifications.
10. Excellent problem-solving and analytical skills with a focus on optimizing data pipelines.
11. Ability to adapt to new technologies and changing business needs.
12. Bachelor’s or Advanced Degree in Information Technology, Computer Science, Mathematics, Statistics, Analytics, or Business.

Preferred Qualifications:

1. 2+ years of experience with batch or PowerShell scripting.
2. 2+ years of experience with Python scripting.
3. 3+ years of data modeling experience in a data warehouse environment.
4. Familiarity with Informatica Intelligent Cloud Services (Data Integration).
5. Experience designing and building APIs in Snowflake and ADF (e.g., REST, RPC).
6. Experience with State Medicaid, Medicare, or Healthcare applications.

Certifications:
Azure certifications related to data engineering or data analytics are preferred.