Remote Azure Data Engineer - Databricks & Data Factory

Eliassen Group; Columbia (SC); Remote; USD 8,954,000 - 10,332,000; Full time

Benefits offered by this job

Medical benefits

401k with match

Dental & Vision

Job summary

Eliassen Group is seeking an Azure Data Engineer to modernize an enterprise data platform by migrating data from on-prem Oracle/DataStage to a cloud-based Azure data lake.

You will design, build, and optimize scalable data pipelines with Databricks and Data Factory, implement data quality, and promote data through Bronze/Silver/Gold layers. Strong Azure skills and collaboration are essential.

Qualifications

• Hands-on data engineering experience in enterprise environments.
• Experience with Azure Databricks and Azure Data Factory.
• Strong SQL and Python/PySpark development skills.
• Experience designing batch data pipelines and ETL/ELT workflows.
• Knowledge of medallion or lakehouse architecture (Bronze/Silver/Gold).
• Experience with relational databases and large datasets.
• Understanding of data modeling, partitioning, and performance optimization.
• Experience with version control (Git) and CI/CD.
• Strong analytical and problem-solving abilities.
• Excellent communication and collaboration with stakeholders.
• Preferred: healthcare or finance data, regulated environments, Azure DevOps, Agile/Scrum.

Responsibilities

• Design, develop, test, and maintain data pipelines using Azure Databricks and Azure Data Factory.
• Support the migration of data and ETL workloads from on-premises Oracle and DataStage to Azure cloud data lake.
• Develop scalable ingestion, transformation, and data-processing solutions.
• Design and implement medallion architecture across Bronze, Silver, Gold layers.
• Build data-quality checks and validation into pipelines and workflows.
• Automate validation and testing to improve reliability of data products.
• Troubleshoot pipeline failures and data-quality issues.
• Collaborate with engineers, analysts, architects, and stakeholders to translate requirements into data solutions.
• Participate in code reviews, documentation, testing, deployment, and improvements.
• Promote reusable development patterns and sound engineering practices.

Skills

Azure Databricks

Azure Data Factory

SQL

Python

PySpark

Data pipelines

Medallion architecture

Data quality

Git

CI/CD

Communication skills

Troubleshooting

Tools

Azure Databricks

Azure Data Factory

Git

CI/CD

Job description

Eliassen Group is seeking an Azure Data Engineer to modernize an enterprise data platform by migrating data from on-prem Oracle/DataStage to a cloud-based Azure data lake.

You will design, build, and optimize scalable data pipelines with Databricks and Data Factory, implement data quality, and promote data through Bronze/Silver/Gold layers. Strong Azure skills and collaboration are essential.