Azure Data Engineer: Spark Pipelines & Lakehouse

Tech Mirrors; Cary (NC); On-site; USD 96,000 - 152,000; Full time

Job summary

Tech Mirrors in Cary, NC seeks an experienced Azure Data Engineer for a contract role. You will design and implement batch and streaming data pipelines using Azure Databricks, Kafka, and Spark, and drive data lakehouse architectures across platforms.

Responsibilities include ETL/ELT development with Python/SQL, Postgres migration to Azure, data quality checks, and data governance basics, while collaborating with cross-functional teams to ensure scalable and cost-efficient data solutions.

Qualifications

• 8+ years of experience in data engineering in azure using Azure Databricks.
• Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
• Strong Data Engineering experience in Big Data (Cloudera).
• Hands-on expertise in SQL, Python and PySpark or Scala.
• Proven experience with batch & real-time ingestion.
• Experience with Kafka and big data processing.

Responsibilities

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop ETL/ELT solutions using Spark, SQL and Python/PySpark, or Scala.
• Implement and support Data Lake / Lakehouse architectures.
• Integrate data from legacy and modern platforms. Preferred Postgres DB migration to Azure Cloud.
• Apply data quality checks and support data governance basics (metadata, lineage, access controls).
• Optimize Spark jobs for performance, reliability, and cost.
• Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers.

Skills

SQL

Python

PySpark

Scala

Tools

Azure Databricks

Azure Data Lake

Azure Data Factory

Postgres

Kafka

Spark

Job description

Tech Mirrors in Cary, NC seeks an experienced Azure Data Engineer for a contract role. You will design and implement batch and streaming data pipelines using Azure Databricks, Kafka, and Spark, and drive data lakehouse architectures across platforms.

Responsibilities include ETL/ELT development with Python/SQL, Postgres migration to Azure, data quality checks, and data governance basics, while collaborating with cross-functional teams to ensure scalable and cost-efficient data solutions.