Remote Azure Data Engineer - Databricks & Data Lake

Eliassen Group; Frankfort (KY); Remote; USD 90,000 - 103,000; Part time

Benefits offered by this job

Medical, Dental, Vision benefits

401k with company matching

Life insurance

Job summary

Eliassen Group seeks an Azure Data Engineer to modernize an enterprise data platform by migrating data from an on-premises Oracle data warehouse and IBM DataStage environment to a cloud-based Azure data lake architecture.

You will design, build, and optimize scalable data pipelines using Azure Databricks and Azure Data Factory, implement data-quality and validation processes, and promote data through medallion architecture layers.

Qualifications

• Hands-on data engineering in enterprise environments.
• Experience with Azure Databricks and Azure Data Factory.
• Strong SQL and Python or PySpark development.
• Experience designing batch ETL/ELT pipelines.
• Experience implementing data-quality, data-validation, and reconciliation processes.
• Knowledge of medallion or lakehouse architecture.

Responsibilities

• Design, develop, test, and maintain data pipelines using Azure Databricks and Azure Data Factory.
• Support the migration of data and ETL workloads from on-premises Oracle and DataStage environment to an Azure cloud data lake.
• Develop scalable ingestion, transformation, and data-processing solutions.
• Design and implement medallion architecture patterns across Bronze, Silver, and Gold data layers.
• Build data-quality, reconciliation, and validation checks directly into data pipelines and engineering workflows.
• Automate validation and testing to improve the reliability of data products.
• Troubleshoot pipeline failures, data-quality issues, and performance problems.
• Collaborate with engineers, analysts, architects, and business stakeholders to translate requirements into practical data solutions.
• Participate in code reviews, documentation, testing, deployment, and continuous-improvement activities.
• Promote reusable development patterns and sound engineering practices across the team.

Skills

Azure Databricks

Azure Data Factory

SQL

Python

PySpark

Data pipelines

Tools

Git

Azure DevOps

Job description

Eliassen Group seeks an Azure Data Engineer to modernize an enterprise data platform by migrating data from an on-premises Oracle data warehouse and IBM DataStage environment to a cloud-based Azure data lake architecture.

You will design, build, and optimize scalable data pipelines using Azure Databricks and Azure Data Factory, implement data-quality and validation processes, and promote data through medallion architecture layers.