Azure Data Engineer

Programmers.io; Cary (NC); On-site; USD 120,000 - 170,000; Full time

Job summary

Programmers.io is seeking an experienced data engineer to design and build batch and streaming data pipelines using Azure, Databricks, and Kafka. You will develop ETL/ELT using Spark, SQL, and Python/PySpark or Scala, and implement Data Lake/Lakehouse architectures.

You will integrate data from legacy and modern platforms, with a preference for migrating Postgres to Azure. Ideal candidates will have 8+ years in data engineering with Azure Databricks and hands-on SQL, Python, PySpark or Scala,

Qualifications

• 8+ years of data engineering experience in Azure environments.
• Hands-on experience with Azure Databricks, Azure Data Lake, Postgres and Spark ecosystem.

Responsibilities

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop ETL/ELT solutions using Spark, SQL and Python/PySpark or Scala.
• Implement and support Data Lake / Lakehouse architectures.
• Integrate data from legacy and modern platforms; migrate Postgres to Azure.

Skills

Azure Databricks

Azure Data Lake

Databricks

SQL

Python

PySpark

Scala

Kafka

Postgres

ADF

Job description

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop ETL/ELT solutions using Spark, SQL and Python/Pyspark, or Scala
• Implement and support Data Lake / Lakehouse architectures
• Integrate data from legacy and modern platforms. Preferred postgres DB migration to Azure Cloud.
• Expertise in handling Data migration using API's.
• Apply data quality checks and support data governance basics (metadata, lineage, access controls)
• Optimize Spark jobs for performance, reliability, and cost
• Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers

Required Skills & Experience:

• 8+ years of experience in data engineering in azure using Azure Databricks.
• Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
• Hands-on expertise in SQL, Python and PySpark or Scala
• Proven experience with batch & real-time ingestion
• Experience with Kafka and big data processing

Added good to have experience with:

• Informatica IDMC or equivalent cloud data integration tools
• Data tools such as Precisely or equivalent

Added Advantage

• Insurance domain knowledge (Policy, Claims, Billing, Underwriting, Actuarial, Regulatory data)
• Experience with Azure DevOps (AzDO) and GitHub-based CI/CD
• Practical usage of Copilot for data engineering and development productivity
• Exposure to Generative AI / GenAI use cases in data platforms and analytics