Databricks Data Engineer

Quantum Technologies. LLC

Phoenix (AZ)

On-site

USD 81,000 - 115,000

Full time

4 hours ago
Be an early applicant

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Quantum Technologies. LLC is seeking a Databricks Data Engineer to design, build, and optimize scalable data pipelines in a fast-paced Azure-based environment.

You will work with Databricks, PySpark, Delta Lake, and Kafka to deliver end-to-end ETL/ELT solutions and CDC-enabled data processing. The role requires strong PySpark and SQL skills, hands-on Databricks experience, and experience with data quality, SCD modeling, and CI/CD workflows in GitHub.

Qualifications

  • Hands-on Databricks experience with data engineering.
  • Experience building CDC-enabled end-to-end pipelines.
  • Strong PySpark and SQL; Python a plus.
  • Azure cloud environment experience is required.
  • Familiarity with ETL/ELT design and data modeling.

Responsibilities

  • Design, build, and optimize end-to-end ETL/ELT pipelines with Databricks and Spark.
  • Develop and maintain data solutions on Azure using Databricks, Delta Lake, and Kafka.
  • Build CDC-based incremental data processing for real-time needs.
  • Work on new pipelines and enhance existing ones; review codebases.
  • Integrate data from flat files, XML, JSON, SQL Server, JDBC, APIs, MQ.
  • Implement data quality rules and quarantine error records.
  • Apply SCD Type 1 and Type 2 modelling techniques.
  • Ensure reliability, performance, and monitoring across workflows.
  • Automate workflows with Databricks Workflows and CI/CD (GitHub).
  • Collaborate with stakeholders and participate in Agile ceremonies.

Skills

CDC capabilities
End-to-end data pipelines
PySpark
SQL
Azure cloud experience
Databricks expertise

Tools

Databricks
Delta Lake
Kafka streaming
Azure
SQL Server
JDBC

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Title: Databricks Data Engineer
Location: Phoenix,AZ
Duration: 9 Months+ Extension
Bill Rate: $71/hour(C2C)
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: USC, GC, H-1B, OPT-EAD, GC-EAD, H4-EAD

Job Summary
We are seeking a highly skilled Databricks Data Engineer to support the Retail Spine initiative, working with a wide range of retail datasets to design, build, and optimize scalable data solutions. This role requires deep hands-on expertise in Databricks and the ability to contribute immediately by developing new pipelines while enhancing existing data workflows. The ideal candidate brings extensive experience building end-to-end data engineering pipelines with CDC (Change Data Capture) capabilities and thrives in a fast-paced Azure-based environment leveraging Databricks, PySpark, and Kafka streaming.

Key Responsibilities:

Design, build, and optimize end-to-end ETL/ELT pipelines using Databricks and Apache Spark (PySpark).

Develop and maintain scalable data solutions on Microsoft Azure, leveraging Databricks, Delta Lake, and Kafka streaming.

Build robust pipelines with Change Data Capture (CDC) to support incremental data processing and real-time/near real-time data needs.

Work across both new pipeline development and enhancement of existing pipelines, quickly understanding current codebases to deliver improvements.

Integrate and process data from diverse sources including:

Flat files, XML, JSON, SQL Server and JDBC systems, Kafka topics, APIs, MQ, Unity Catalog volumes and other enterprise platforms.

Implement data quality frameworks, including writing technical data quality rules and quarantining error records.

Apply canonical data modeling techniques, including SCD Type 1 and Type 2 implementations.

Ensure high levels of data reliability, performance, scalability, and monitoring across all workflows.

Automate workflows using Databricks Workflows and CI/CD pipelines (GitHub), focusing on reuse, efficiency, and standardization.

Collaborate with developers, architects, and product owners, and engage with stakeholders, source teams, and downstream consumers as needed.

Participate actively in Agile/Scrum ceremonies and deliver high-quality solutions within defined timelines.

Required Qualifications:

Extensive hands-on experience with Databricks, with ability to contribute from day one.

Proven experience building end-to-end data engineering pipelines with CDC capabilities.

Strong proficiency in PySpark and SQL (required); Python experience is a plus.

Experience working in Microsoft Azure cloud environments.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Data Engineer
Databricks Data Engineer

Valiant Technologies LLC • Phoenix (AZ)

On-site
USD 81,000 - 115,000
Databricks Data Engineer
Databricks Data Engineer

CloudBigD • Phoenix (AZ)

On-site
USD 81,000 - 115,000
Databricks Engineer
Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

On-site
USD 84,000 - 120,000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Robotics Technologies • Sunnyvale (CA)

On-site
USD 76,000 - 108,000
Databricks Data Engineer
Databricks Data Engineer

CRG • United States

On-site
USD 90,000 - 94,000
Databricks Data Engineering Lead
Databricks Data Engineering Lead

Ethereum Technologies LLC • New Jersey

Remote
USD 84,000 - 120,000
Databricks Data Engineering Lead
Databricks Data Engineering Lead

Valiant Technologies LLC • New Jersey

Remote
USD 87,000 - 117,000
Databricks Data Engineering Lead
Databricks Data Engineering Lead

Robotic Process Automation LLC • New Jersey

Remote
USD 84,000 - 120,000
Databricks Data Engineering Lead
Databricks Data Engineering Lead

Quantum Technologies LLC • New York (NY)

Remote
USD 84,000 - 120,000
Databricks Data Engineer
Databricks Data Engineer

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 170,000