Azure Data Engineer

SCIGON; Camden (NJ); On-site; USD 98,000 - 130,000; Full time

Job summary

SCIGON is seeking an experienced Azure Data Engineer to design and implement scalable data solutions on Microsoft Azure and Databricks. You will build robust ETL/ELT pipelines, optimize data flows, and collaborate with analytics teams to empower reporting and insights.

The ideal candidate has hands-on experience with Azure Databricks, ADF, Delta Lake, Spark, and Medallion Architecture, as well as Terraform and GitHub Actions for CI/CD and IaC.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.
• 5+ years of experience in Data Engineering or related technical roles.
• Strong experience with Azure Databricks, Azure Data Factory, ADLS, Apache Spark, and Delta Lake.
• Experience implementing Medallion Data Architecture.
• Hands-on Terraform for infrastructure automation.
• Experience building CI/CD pipelines using GitHub Actions.
• Experience in Agile environments.
• Excellent analytical and problem-solving skills.

Responsibilities

• Design, develop, and maintain scalable data platforms on Microsoft Azure.
• Build and optimize ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.
• Develop data transformation processes leveraging Apache Spark and Delta Lake technologies.
• Implement and maintain Medallion Architecture to support analytics and reporting needs.
• Collaborate with business stakeholders, analysts, data scientists, and application teams.
• Design efficient data models that support reporting and BI initiatives.
• Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
• Develop and maintain CI/CD pipelines using GitHub Actions.
• Provision and manage cloud infrastructure using Terraform and IaC best practices.
• Ensure data quality, governance, security, and compliance standards.
• Provide production support and resolve data integration issues.

Skills

Azure Databricks

Azure Data Factory

ADLS

Apache Spark

Delta Lake

SQL & data modeling

Medallion Architecture

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or related field

Tools

Terraform

GitHub Actions

Job description

Azure Data Engineer Salary: $97,500-$130,000 Job Summary

We are seeking an experienced Azure Data Engineer with strong Databricks expertise to design, develop, and support scalable cloud-based data solutions. This role is responsible for building and maintaining modern data platforms on Microsoft Azure and Databricks, developing robust data pipelines, and enabling analytics and reporting initiatives across the organization.

The ideal candidate will have extensive hands-on experience with Azure Databricks, Azure Data Factory, Delta Lake, Apache Spark, and Medallion Architecture. Experience with Terraform and GitHub Actions is required to support infrastructure automation, CI/CD processes, and cloud platform administration.

Key Responsibilities

• Design, develop, and maintain scalable data platforms and solutions on Microsoft Azure.
• Build and optimize ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.
• Develop data transformation processes leveraging Apache Spark and Delta Lake technologies.
• Implement and maintain Medallion Architecture (Bronze, Silver, Gold layers) to support enterprise analytics and reporting needs.
• Collaborate with business stakeholders, analysts, data scientists, and application teams to understand data requirements and deliver solutions.
• Design and implement efficient data models that support reporting, analytics, and business intelligence initiatives.
• Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and scalability.
• Develop and maintain CI/CD pipelines using GitHub Actions.
• Provision and manage cloud infrastructure using Terraform and Infrastructure as Code (IaC) best practices.
• Ensure data quality, governance, security, and compliance standards are met.
• Provide production support and resolve issues related to data integration and platform performance.
• Document technical designs, architectures, standards, and operational procedures.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.
• 5+ years of experience in Data Engineering or related technical roles.
• Strong experience with:
  • Azure Databricks
  • Azure Data Factory (ADF)
  • Azure Data Lake Storage (ADLS)
  • Apache Spark (PySpark preferred)
  • Delta Lake
  • SQL and data modeling
• Experience implementing Medallion Data Architecture.
• Hands-on experience with Terraform for infrastructure automation.
• Experience building CI/CD pipelines using GitHub Actions.
• Strong knowledge of cloud-native data engineering best practices.
• Experience working in Agile environments.
• Excellent analytical, troubleshooting, and problem-solving skills.

Preferred Qualifications

• Microsoft Azure certifications.
• Experience with Azure Synapse Analytics.
• Experience with Power BI or other analytics platforms.
• Knowledge of data governance and data security frameworks.
• Experience supporting large-scale enterprise data environments.

Key Competencies

• Data Engineering
• Cloud Architecture
• ETL/ELT Development
• Infrastructure as Code (Terraform)
• CI/CD Automation
• Data Modeling
• Problem Solving
• Communication and Collaboration
• Performance Optimization
• Stakeholder Management