Remote Senior Azure Data Engineer: Data Lake & Pipelines

Eliassen Group; Boston (MA); Hybrid; USD 103,000 - 117,000; Full time

Job summary

Eliassen Group is seeking a Senior Azure Data Engineer to modernize an enterprise data platform by migrating data and workloads to a cloud-based Azure data lake. You will design, build, and optimize scalable pipelines using Azure Databricks and Azure Data Factory, contributing to architecture decisions and engineering best practices.

Experience with healthcare or finance data is a plus, and a strong background in data quality is essential.

Qualifications

• Strong hands-on data engineering in enterprise environments.
• Demonstrated experience with Azure Databricks and Azure Data Factory.
• Strong SQL and Python/PySpark development skills.
• Experience designing batch data pipelines and ETL/ELT workflows.
• Experience implementing data-quality and data-validation processes.
• Knowledge of medallion/lakehouse architecture (Bronze/Silver/Gold).
• Experience with relational databases and large datasets.
• Understanding of data modeling, partitioning, and performance optimization.
• Experience with Git or similar version control.
• Experience with CI/CD and automated deployment practices.

Responsibilities

• Design, develop, test, and maintain data pipelines using Azure Databricks and Azure Data Factory.
• Support migration of data and ETL workloads to an Azure cloud data lake.
• Develop scalable ingestion, transformation, and data-processing solutions.
• Design and implement medallion architecture patterns across Bronze, Silver, Gold layers.
• Build data-quality, reconciliation, and validation checks into pipelines.
• Automate validation and testing to improve data product reliability.
• Guide data architecture decisions and engineering standards.
• Troubleshoot pipeline failures and data-quality issues.
• Collaborate with engineers, analysts, architects, and stakeholders to translate requirements.
• Participate in code reviews, documentation, testing, deployment, and continuous improvement.
• Promote reusable development patterns and sound engineering practices across the team.

Skills

Azure Databricks

Azure Data Factory

SQL

Python

PySpark

Data pipelines

CI/CD

Git

Data modeling

Troubleshooting

Tools

Azure Databricks

Azure Data Factory

Oracle

IBM DataStage

Delta Lake

Databricks

Azure DevOps

Job description

Eliassen Group is seeking a Senior Azure Data Engineer to modernize an enterprise data platform by migrating data and workloads to a cloud-based Azure data lake. You will design, build, and optimize scalable pipelines using Azure Databricks and Azure Data Factory, contributing to architecture decisions and engineering best practices.

Experience with healthcare or finance data is a plus, and a strong background in data quality is essential.