Azure Data Engineer: Lakehouse & Real-Time Pipelines

Programmers.io; Cary (NC); On-site; USD 120,000 - 170,000; Full time

Job summary

Programmers.io is seeking an experienced data engineer to design and build batch and streaming data pipelines using Azure, Databricks, and Kafka. You will develop ETL/ELT using Spark, SQL, and Python/PySpark or Scala, and implement Data Lake/Lakehouse architectures.

You will integrate data from legacy and modern platforms, with a preference for migrating Postgres to Azure. Ideal candidates will have 8+ years in data engineering with Azure Databricks and hands-on SQL, Python, PySpark or Scala,

Qualifications

• 8+ years of data engineering experience in Azure environments.
• Hands-on experience with Azure Databricks, Azure Data Lake, Postgres and Spark ecosystem.

Responsibilities

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop ETL/ELT solutions using Spark, SQL and Python/PySpark or Scala.
• Implement and support Data Lake / Lakehouse architectures.
• Integrate data from legacy and modern platforms; migrate Postgres to Azure.

Skills

Azure Databricks

Azure Data Lake

Databricks

SQL

Python

PySpark

Scala

Kafka

Postgres

ADF

Job description

Programmers.io is seeking an experienced data engineer to design and build batch and streaming data pipelines using Azure, Databricks, and Kafka. You will develop ETL/ELT using Spark, SQL, and Python/PySpark or Scala, and implement Data Lake/Lakehouse architectures.

You will integrate data from legacy and modern platforms, with a preference for migrating Postgres to Azure. Ideal candidates will have 8+ years in data engineering with Azure Databricks and hands-on SQL, Python, PySpark or Scala,