Azure Data Engineer Senior AmeriHealth Caritas · Remote US

QuickCruit, Inc.; Pennsylvania; Hybrid; USD 120,000 - 180,000; Full time

Job summary

QuickCruit, Inc. is seeking a Senior Azure Data Engineer to design, build, and optimize large-scale cloud data solutions on Microsoft Azure.

You will lead ingestion, real-time streaming, and dimensional modeling efforts, mentoring junior developers in an Agile setting and applying Gen AI to data tasks where applicable.

Qualifications

• Hands-on experience with Azure Data Factory, Azure Databricks, HVR, Kafka, Event Hubs, Snowflake, and Azure Data Lake Storage.
• Strong PySpark and SQL skills with data modeling and ETL/ELT design.
• Knowledge of Gen AI techniques applied to data engineering workflows.

Responsibilities

• Architect and implement scalable cloud data solutions on Microsoft Azure.
• Develop robust ETL/ELT pipelines for batch and streaming workloads.
• Design real-time ingestion and CDC patterns with Event Hubs, Kafka, and HVR.
• Optimize Databricks jobs, manage cost, and ensure data quality and lineage.

Skills

Azure Data Factory

Azure Databricks

HVR

Kafka

Event Hubs

Snowflake

Azure Data Lake Storage

PySpark

SQL

GenAI

Delta Lake

Azure DevOps

Git

Education

Master’s degree in Computer Science

Tools

Azure DevOps

Git

Delta Lake

Job description

THIS IS A HYBRID, OFFICE BASED POSITION WORKING AT OUR ELLIS PRESERVE OFFICE LOCATED IN NEWTOWN SQUARE, PA. WE WILL NOT CONSIDER REMOTE CANDIDATES! Role Overview The Senior Azure Data Engineer designs, builds, and optimizes large scale cloud data solutions on Microsoft Azure. The role requires deep hands on expertise with modern ingestion frameworks to ingest diverse data sources, on distributed compute, real-time streaming, and dimensional modeling, combined with strong Agile delivery practices and leadership in guiding junior developers. Experience with GenAI technologies is increasingly important as the organization evolves toward intelligent data products. Work Arrangement This a HYBRID office based position working in our Ellis Preserve office located in Newtown Square, PA. Core office days are Tuesday, Wednesday, and Thursday. Responsibilities Architecture & Implementation: Design and implement scalable engineering solutions using Azure Data Factory, Azure Databricks, Azure Event Hubs, Kafka, HVR, Snowflake, and Azure Data Lake Storage Pipeline Development: Build robust ETL/ELT pipelines for batch and streaming workloads using PySpark and SQL, ensuring fault tolerance, schema evolution handling, and idempotency API Ingestion: Implement reliable API data ingestion patterns including authentication, pagination, rate limit handling, retries, and schema mapping Real time Integration: Design and operate event driven ingestion and processing using Event Hubs and Kafka, and integrate CDC solutions using HVR Performance Optimization: Tune Databricks jobs and clusters (cluster sizing, autoscaling, caching, partitioning, shuffle optimization, Delta Lake optimizations) to meet SLAs and cost targets DevOps & CI/CD: Use Azure DevOps for backlog management, Git workflows, YAML pipelines, automated testing, and repeatable deployments across environments Code Quality & Mentorship: Perform code reviews, enforce coding standards and best practices, and mentor junior developers to raise team capability Gen AI Enablement: Apply Gen AI knowledge to accelerate development (code generation, data documentation, prompt engineering for data tasks) and to integrate AI driven data products where applicable Stakeholder Collaboration: Work with product owners, data scientists, analysts, and architects to translate business requirements into prioritized user stories and deliverables in Agile sprints Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc. Identify and tackle issues concerning data management to improve data quality Understand and implement best practices in management of data, including master data, reference data, metadata, data quality and lineage Education and Experience Master’s degree in computer science, Information Systems, Software Engineering Technical: Hands on experience with Azure Data Factory (6+ years), Azure Databricks (6+ years), HVR (3+ years), Kafka (5+ years), Event Hubs (3+ years), Snowflake (5+ years), Azure Data Lake Storage, PySpark (10+ years), and SQL (12+ years) Modeling: Strong expertise in dimensional modeling, SCD implementation, and data warehouse best practices Agile Delivery: Proven experience delivering data projects in Agile (Scrum or Kanban) teams DevOps: Practical experience with Azure DevOps for CI/CD, branching strategies, and pipeline automation Performance: Demonstrated ability to profile and optimize Databricks jobs and large scale data processing workloads Integration: Experience ingesting data from REST APIs and external systems, handling schema drift and error handling Leadership: Track record of conducting code reviews, providing technical guidance, and mentoring junior engineers Communication: Clear communicator able to present technical tradeoffs to technical and non technical stakeholders Experience with Collibra Data governance features Familiarity with MLOps patterns and integrating data platforms with ML pipelines Azure certifications such as DP 700, AZ 400, or Databricks Data Engineer certification Prior work with cost optimization and observability tools for cloud data platforms Practical experience applying Gen AI (LLMs, embeddings, retrieval augmentation) to data engineering workflows] Other Skills Problem solving: Analytical approach to diagnose performance and data quality issues Collaboration: Strong team player who partners effectively across product, analytics, and platform teams Mentorship: Patient, constructive coach who elevates code quality and engineering practices Adaptability: Comfortable with shifting priorities and iterative delivery in fast moving environments