Databricks Engineer

Quantiphi; United States; On-site; USD 100,000 - 130,000; Full time

Benefits offered by this job

Exposure to Fortune 500 clients

Latest technologies in AI and cloud

Leading a talented team

Job summary

A leading AI-first digital transformation company in the United States is seeking a Mid-Senior Level Data Engineer. You will be responsible for designing, developing, and maintaining scalable data pipelines using PySpark and SQL, primarily in Azure Databricks environments. The ideal candidate will have expertise in data modeling and CI/CD practices, ensuring high performance and reliability in data solutions. Join a dynamic team and work with Fortune 500 companies to drive innovation in AI and cloud technologies.

Qualifications

• Expert-level knowledge of Azure and Databricks architecture.
• Experience with CI/CD for data pipelines and automated testing.
• Strong expertise in PySpark, SQL, and data modeling for high-volume datasets.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and SQL.
• Implement data models optimized for analytics and performance.
• Collaborate with cross-functional teams to deliver high-quality data solutions.

Skills

PySpark

SQL

Azure Databricks

Data Modeling

CI/CD

Distributed Computing

Tools

Azure

GCP

AWS

Databricks

Job description

Company Highlights:

• Quantiphi has seen 2.5x growth YoY since its inception in 2013, we don’t just innovate—we lead. Headquartered in Boston, with 4000+ Quantiphi professionals across the globe.
• As an Elite/Premier Partner for Google Cloud, AWS, NVIDIA, Snowflake, and others, we’ve been recognized with:
• 17x Google Cloud Partner of the Year awards in the last 8 years
• 3x AWS AI/ML award wins
• 3x NVIDIA Partner of the Year titles
• 2x Snowflake Partner of the Year awards
• Top analyst recognitions from Gartner, ISG, and Everest Group
• First‑in‑class industry solutions across Healthcare, Financial Services, Consumer Goods, Manufacturing, and more, powered by cutting‑edge Generative AI and Agentic AI accelerators
• Certified as a Great Place to Work for the third year in a row—2021, 2022, 2023

Be part of a trailblazing team that’s shaping the future of AI, ML, and cloud innovation. Your next big opportunity starts here!

Job Description:

Key Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using PySpark, SQL, and modern data engineering frameworks.
• Perform deep‑dive analysis into complex source systems to define extraction patterns, transformation logic, and integration workflows.
• Implement data models optimized for analytics, performance, and operational excellence.
• Build and manage data systems across Azure, AWS or GCP cloud environments, ensuring reliability, scalability, and cost efficiency.
• Lead and participate in design reviews, provide mentorship to junior engineers, and take ownership of code quality and engineering best practices.
• Collaborate with cross‑functional teams including architects, analysts, product, and platform engineers to deliver high‑quality data solutions.
• Troubleshoot performance bottlenecks, optimize data processing workflows, and improve pipeline robustness.

Required Qualifications

• Expert‑level knowledge of Azure and Databricks architecture, including Lakehouse patterns and the Medallion Framework.
• Familiarity with Delta Lake, orchestration tools, or Unity Catalog.
• Experience with CI/CD for data pipelines and automated testing frameworks.
• Experience in data engineering or a related field.
• Strong expertise in PySpark, SQL, and data modeling for high‑volume datasets.
• Hands‑on experience building and optimizing data pipelines primarily in Azure Databricks, AWS or GCP environments.
• Track record of working with complex source systems and developing accurate extraction and transformation logic.
• Proven experience in mentoring engineers, performing code reviews, and leading technical design discussions.
• Solid understanding of distributed computing, data partitioning, and performance tuning.

What is in it for you:

• Be part of the fastest‑growing AI‑first digital transformation and engineering company in the world
• Lead an energetic team of highly dynamic and talented individuals
• Exposure to working with Fortune 500 companies and innovative market disruptors
• Exposure to the latest technologies related to artificial intelligence and machine learning, data and cloud

Seniority level

Mid‑Senior level

Employment type

Full‑time

Job function

IT Services and IT Consulting

Referrals increase your chances of interviewing at Quantiphi by 2x