Senior Azure Data Engineer

Programmers.io; Cary (NC); On-site; USD 120,000 - 150,000; Full time

Job summary

Programmers.io in Cary, NC is seeking an experienced data engineer to design and build batch and streaming pipelines using Azure, Databricks, and Kafka.

The role requires strong SQL, Python, PySpark or Scala, and hands-on data lake architectures with Delta Lake. You will collaborate with cross-functional teams and contribute to governance, data migration, and cost optimization.

Qualifications

• 8+ years of data engineering experience in Azure Databricks.
• Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
• Big Data experience with Cloudera.
• Hands-on SQL, Python and PySpark/Scala.
• Proven batch and real-time ingestion experience.
• Experience with Kafka and big data processing.

Responsibilities

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop ETL/ELT solutions using Spark, SQL and Python/PySpark, or Scala.
• Implement and support Data Lake / Lakehouse architectures.
• Integrate data from legacy and modern platforms with PostgreSQL migrations to Azure.
• Apply data quality checks and support governance basics.
• Optimize Spark jobs for performance, reliability, and cost.
• Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers.

Skills

Azure Databricks

Azure Data Lake

ADF

Postgres

Cloudera

SQL

Python

PySpark

Scala

Kafka

Batch ingestion

Real-time ingestion

Tools

Informatica PowerCenter

Informatica IDMC

Azure DevOps

GitHub CI/CD

Copilot

Job description

Local to Cary NC only!
Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
Develop ETL/ELT solutions using Spark, SQL and Python/Pyspark, or Scala
Implement and support Data Lake / Lakehouse architectures
Integrate data from legacy and modern platforms. Preferred postgres DB migration to Azure Cloud.
Expertise in handling Data migration using API's.
Apply data quality checks and support data governance basics (metadata, lineage, access controls)
Optimize Spark jobs for performance, reliability, and cost
Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers

Required Skills & Experience:

• 8+ years of experience in data engineering in azure using Azure Databricks.
• Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
• Strong Data Engineering experience in Big Data (Cloudera)
• Hands-on expertise in SQL, Python and PySpark or Scala
• Proven experience with batch & real-time ingestion
• Experience with Kafka and big data processing

Added good to have experience with:

• Informatica PowerCenter
• Informatica IDMC or equivalent cloud data integration tools
• SQL Server, Oracle, Azure SQL
• Cosmos DB
• Unix/Linux
• Maestro (or similar enterprise schedulers)
• Data tools such as Precisely or equivalent

Added Advantage

• Insurance domain knowledge (Policy, Claims, Billing, Underwriting, Actuarial, Regulatory data)
• Experience with Azure DevOps (AzDO) and GitHub-based CI/CD
• Practical usage of Copilot for data engineering and development productivity
• Exposure to Generative AI / GenAI use cases in data platforms and analytics

Role Descriptions:

• Microsoft Azure (ADB/ADF)

Essential Skills:

• Microsoft Azure (ADB/ADF)