Senior Azure Databricks Data Engineer C2C jobs Cary, NC

Tech Mirrors; Cary (NC); On-site; USD 140,000 - 180,000; Full time

Job summary

Tech Mirrors in Cary, NC seeks an experienced Azure Data Engineer to design and build batch and streaming data pipelines using Azure, Databricks, and Kafka. You will develop ETL/ELT solutions with Spark, SQL and Python/PySpark, and implement Data Lake/Lakehouse architectures while migrating data from legacy systems.

Ideal candidates have 8+ years of data engineering on Azure, strong SQL/Python skills, and hands-on experience with batch and real-time ingestion.

Qualifications

• 8+ years of experience in data engineering in Azure using Azure Databricks.
• Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
• Hands-on experience with SQL, Python and PySpark or Scala.
• Proven experience with batch & real-time ingestion.

Responsibilities

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop ETL/ELT solutions using Spark, SQL and Python/PySpark, or Scala.
• Implement and support Data Lake / Lakehouse architectures and migrate data to Azure Cloud.
• Collaborate with cross-functional teams and stakeholders.

Skills

SQL

Python

PySpark

Scala

Kafka

Education

Bachelor's degree in Computer Science or related

Tools

Microsoft Azure (ADB/ADF)

Azure Databricks

Azure Data Lake

Postgres

Kafka

Job description

Job Title: :Azure Data Engineer
Location: Cary North Carolina (looking only Locals)
Contract

Onsite opportunity

Skills : Data Bricks,Microsoft Azure (ADB/ADF)

Job Description:

Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
Develop ETL/ELT solutions using Spark, SQL and Python/Pyspark, or Scala
Implement and support Data Lake / Lakehouse architectures
Integrate data from legacy and modern platforms. Preferred postgres DB migration to Azure Cloud.
Expertise in handling Data migration using API’s.
Apply data quality checks and support data governance basics (metadata, lineage, access controls)
Optimize Spark jobs for performance, reliability, and cost
Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers

Required Skills & Experience:

• 8+ years of experience in data engineering in azure using Azure Databricks.
• Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
• Strong Data Engineering experience in Big Data (Cloudera)
• Hands-on expertise in SQL, Python and PySpark or Scala
• Proven experience with batch & real-time ingestion
• Experience with Kafka and big data processing

Added good to have experience with:

• Informatica PowerCenter
• Informatica IDMC or equivalent cloud data integration tools
• SQL Server, Oracle, Azure SQL
• Cosmos DB
• Unix/Linux
• Maestro (or similar enterprise schedulers)
• Data tools such as Precisely or equivalent

Added Advantage

• Insurance domain knowledge (Policy, Claims, Billing, Underwriting, Actuarial, Regulatory data)
• Experience with Azure DevOps (AzDO) and GitHub-based CI/CD
• Practical usage of Copilot for data engineering and development productivity
• Exposure to Generative AI / GenAI use cases in data platforms and analytics

Education:

At least a bachelor’s degree (or equivalent experience) in Computer Science, Software/Electronics Engineering, Information Systems, or a closely related field is required for the project