Azure Data Engineer

Quantum Technologies LLC

Pennsylvania

On-site

USD 91,000 - 124,000

Full time

31 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Quantum Technologies. LLC in Malvern, PA is seeking an experienced Azure Data Engineer for a 12-month engagement to build and optimize large-scale data pipelines.

You will collect, store, and process data, develop ETL processes, and partner with business and tech teams to deliver reliable analytics solutions. You'll lead migration from Synapse to Databricks, implement batch and real-time pipelines with PySpark, Event Hubs, and Delta Lake, enforce data governance, and drive CI/CD automation using

Qualifications

  • 7+ years in data engineering, 3+ in a lead/senior role.
  • Strong expertise in Azure Databricks, Synapse, PySpark.
  • Proven experience with real-time streaming (Event Hubs, Kafka).
  • Deep knowledge of Azure ecosystem (Data Lake, Data Factory, Key Vault).
  • Strong SQL, performance tuning, and CI/CD pipeline skills.

Responsibilities

  • Collect, store, process and analyze large datasets to build ETL processes.
  • Lead migration of data pipelines from Synapse to Databricks.
  • Architect and implement batch and real-time data pipelines using PySpark and Event Hubs.
  • Establish data engineering best practices and mentor junior engineers.
  • Drive CI/CD automation with AzDO and Git.
  • Ensure data quality, governance, and compliance (Collibra).
  • Collaborate with business/tech teams and leverage insurance domain knowledge.
  • Develop reusable frameworks to reduce development effort and costs.
  • Develop quality code with performance optimizations at development stage.

Skills

Azure Databricks
PySpark
SQL
Data modeling
CI/CD
Leadership

Tools

Databricks
Event Hubs
Delta Lake
Kafka
Azure Data Factory
Cosmos DB

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

  • Job Title: Azure data Engineer
  • Location: Malvern, PA
  • Duration: 12 Months + Extension
  • Bill Rate: $78/hour
  • Job Type: W-2, 1099, C2C
  • Client: To Be Discussed Later
  • Work Authorization: US-Citizen, H-1B, OPT-EAD, GC-EAD
Job Description
Key Responsibilities
  • Collect, store, process and analyze large datasets to build and implement extract, transfer, load (ETL) processes.
  • Lead migration of data pipelines from Synapse to Databricks.
  • Architect and implement batch + real-time data pipelines using PySpark, Databricks, and Event Hubs.
  • Establish data engineering best practices and mentor junior engineers.
  • Drive CI/CD automation with AzDO and Git.
  • Ensure data quality, governance, and compliance (Collibra).
  • Collaborate with business/tech teams; leverage insurance domain knowledge where applicable.
  • Develop reusable frameworks to reduce the development effort involved thereby ensuring cost savings for the projects.
  • Develop quality code with thought through performance optimizations in place right at the development stage.
  • Appetite to learn new technologies and be ready to work on new cutting-edge cloud technologies.
  • Work with team spread across the globe in driving the delivery of projects and recommend development and performance improvements.
Essential Business Experience and Technical Skills

Building and Implementing data ingestion and curation process developed using Big data tools such as Spark (Scala/python/Java), Hive, HDFS, Sqoop, Kafka, Kerberos, Impala etc. and CDP 7.x

  • Ingesting huge volumes data from various platforms for Analytics needs and writing high-performance, reliable, and maintainable ETL code Strong
  • Strong analytic skills related to working with unstructured datasets
  • Strong experience in building/designing Data warehouses, data stores for analytics consumption On prem and Cloud (real time as well as batch use cases)
  • Ability to interact with business analysts and functional analysts in getting the requirements and implementing the ETL solutions.
Required
  • 7+ years in data engineering, 3+ in a lead/senior role.
  • Strong expertise in Azure Databricks, Synapse, PySpark.
  • Proven experience with real-time streaming (Event Hubs, Structured Streaming, Kafka).
  • Deep knowledge of Azure ecosystem (Data Lake, Data Factory, Key Vault).
  • Strong SQL, performance tuning, and CI/CD pipeline skills.
  • Excellent leadership and communication skills
  • Building and Implementing data ingestion and curation process developed using Big data tools such as Spark(Scala/python/Java), Data bricks, Delta Lake, Hive, HDFS, Sqoop, Kafka, Kerberos, Impala etc.
  • Hands on expertise in: Implementing analytical data stores in Azure platform using ADLS/ADF/Data Bricks and Cosmos DB and CDP 7x
  • Ingesting huge volumes of data from various platforms for Analytics needs and writing high-performance, reliable and maintainable ETL code Strong
  • SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.
  • Proficiency and extensive Experience with Spark & Scala, Python and performance tuning is a MUST
  • Hive database management and Performance tuning is a MUST. (Partitioning / Bucketing)
  • Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.
  • Strong analytic skills related to working with unstructured datasets
  • Performance tuning and problem-solving skills is a must.
  • Code versioning experience using Bitbucket/AzDo. Working knowledge of AzDo pipelines would be a big plus.
  • Monitoring performance and advising any necessary infrastructure changes.
  • Strong experience in building designing Data warehouses, data stores for analytics consumption. (real time as well as batch use cases)
  • Eagerness to learn new technologies on the fly and ship to production
  • Expert in technical program delivery across cross-functional / LOB teams
  • Expert in driving delivery through collaboration in highly complex, matrixed environment
  • Possesses strong leadership and negotiation skills
  • Excellent communication skills, both written and verbal
  • Ability to interact with senior leadership teams in IT and business Preferred
Preferred
  • Collibra, insurance industry experience, Azure/Databricks certifications.
  • Expertise in Python and experience writing Azure functions using Python/Node.js
  • Experience using Event Hub for data integrations.
  • Hands on expertise in: Implementing analytical data stores in Azure platform using ADLS/Azure data factory /Data Bricks and Cosmos DB (mongo/graph API)
  • Experience ingesting using Azure data factory, Complex ETL using Data Bricks.
  • Eagerness to learn new technologies on the fly and ship to production.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Azure Data Engineer
Azure Data Engineer

Valiant Technologies LLC • Pennsylvania

On-site
USD 89,000 - 126,000
Azure Data Engineer
Azure Data Engineer

Robotics Technologies • Malvern

On-site
USD 89,000 - 126,000
Azure Data Engineer
Azure Data Engineer

Ethereum Technologies LLC • Pennsylvania

On-site
USD 89,000 - 126,000
Azure Data Engineer
Azure Data Engineer

Robotic Process Automation LLC • Pennsylvania

On-site
USD 89,000 - 126,000
Azure Data Engineer
Azure Data Engineer

Quantum Technologies LLC • Raleigh (NC)

On-site
USD 110,000 - 160,000
Azure Data Engineer
Azure Data Engineer

Robotics Technologies • Raleigh (NC)

On-site
USD 110,000 - 150,000
Azure Data Engineer
Azure Data Engineer

Robotic Process Automation LLC • Raleigh (NC)

On-site
USD 110,000 - 160,000
Azure Data Engineer
Azure Data Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 89,000 - 126,000
Azure Data Engineer
Azure Data Engineer

Valiant Technologies LLC • Raleigh (NC)

On-site
USD 110,000 - 150,000
Azure Data Engineer
Azure Data Engineer

Quantum Technologies LLC • Dallas (TX)

On-site
USD 83,000 - 124,000