Senior Databricks Data Engineer

GAP Careers

Austin (TX)

On-site

USD 140,000 - 200,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Growth Acceleration Partners (GAP) is seeking a Senior Databricks Data Engineer to design, build, and maintain scalable data pipelines within an Azure/Databricks ecosystem. You will own data modeling, orchestration, and data quality while coordinating with application developers and business stakeholders.

You will work with PySpark, SQL, Unity Catalog, and Lakeflow in a collaborative, security-conscious environment to deliver governed data for enterprise applications.

Qualifications

  • Bachelor’s degree in a technical field such as CS, data engineering, software engineering, or information systems.
  • 5 years of professional data engineering experience with modern cloud data architectures.
  • Hands-on experience building data pipelines on Databricks using PySpark and SQL.
  • Deep expertise with Delta Lake architectures and Unity Catalog governance.
  • Experience integrating Databricks workflows with Azure SQL or Azure Cosmos DB.
  • Experience with Git-based CI/CD in GitHub linking PRs to Issues.

Responsibilities

  • Architect, build, and maintain scalable Databricks data pipelines.
  • Design data models integrating Databricks with Azure SQL/Azure Cosmos DB.
  • Orchestrate data pipelines with Lakeflow Jobs.
  • Enforce data quality, testing, and monitoring standards.
  • Manage deployment lifecycles with Databricks Asset Bundles & CI/CD.
  • Collaborate with developers and product owners to meet needs.
  • Follow security and governance using Microsoft Entra ID and Unity Catalog.

Skills

Databricks
PySpark
SQL
Delta Lake
Unity Catalog
Azure SQL
Azure Cosmos DB
GitHub / CI-CD
Data Modeling
Data Governance
Lakeflow Jobs
Python

Education

Bachelor’s Degree in Computer Science, Data Engineering, Software Engineering, Information Systems, or related technical field

Tools

Databricks Asset Bundles
GitHub
Unity Catalog
Lakeflow

Job description

Job description

WHAT WE DO

Founded in 2007, Growth Acceleration Partners (GAP) is a consulting and technology services company. We consult, design, build and modernize revenue-generating software and data engineering solutions for clients. With modernization services and AI tools, we help businesses achieve a competitive advantage through technology. GAP’s remote, integrated engineering teams use end-to-end solutions to innovate and align with your business goals. We have 600 English-speaking engineers based in Latin America and approximately 20 U.S.-based engineers. With some of the highest customer satisfaction scores in the industry, GAP’s focus is customer and employee success.

GAP is a woman-owned company headquartered in Austin Texas. We are a values-based company focused on growing our people by investing in education, onsite English classes and training in the latest technologies, including AI, data analytics and machine learning. Our goal is to provide solutions for our customers that help them achieve critical business outcomes, while enabling our GAPSters and our communities to attain long‑term success.

Summary

We are looking for a Senior Databricks Data Engineer to design, build, and maintain scalable data pipelines and integrations within a robust Azure and Databricks ecosystem. In this high-impact role, you will work as part of a dedicated team to deliver large, defined projects, seamlessly integrating Databricks with modern application stacks. You will take technical ownership of data modeling, pipeline orchestration, and data quality while collaborating closely with cross-functional application developers and business stakeholders. Operating with a high level of autonomy, you will leverage PySpark, SQL, and Unity Catalog to ensure secure, performant, and governed data delivery for critical enterprise applications.

Education
  • Bachelor’s Degree in Computer Science, Data Engineering, Software Engineering, Information Systems, or a related technical field.
Professional Experience
  • 5 years of professional experience in Data Engineering, specializing in modern cloud data architectures.

  • Proven hands‑on experience building and maintaining complex data pipelines on Databricks using PySpark and SQL.

  • Deep expertise working with Delta Lake architectures and implementing governance via Unity Catalog.

  • Demonstrated experience integrating Databricks workflows with Azure SQL or Azure Cosmos DB.

  • Experience operating in Git-based CI/CD environments, specifically managing all work in GitHub where every pull request is linked to a GitHub Issue.

Key Responsibilities
  • Architect, build, and maintain scalable Databricks data pipelines to support existing systems and new application integrations.

  • Design and implement robust data models that integrate seamlessly between Databricks and transactional databases like Azure SQL or Azure Cosmos DB.

  • Orchestrate data pipelines and schedule workflows utilizing Lakeflow Jobs.

  • Enforce rigorous data quality, testing, and monitoring standards across all data engineering processes.

  • Manage deployment lifecycles using Databricks Asset Bundles and Git-based CI/CD workflows.

  • Collaborate continuously with frontend/backend application developers (React, C# .NET) and Product Owners to ensure data infrastructure meets application requirements.

  • Operate strictly within organizational security and governance models, leveraging Microsoft Entra ID for access and Unity Catalog for data governance.

Required Technical Skills

Strong experience in several of the following areas:

  • Data Engineering & Processing: Deep proficiency in Databricks, PySpark, advanced SQL, and Delta Lake.

  • Governance & Orchestration: Unity Catalog (grants, lineage, access control) and Lakeflow Jobs.

  • Cloud & Databases: Seamless integration with Azure SQL and Azure Cosmos DB.

  • DevOps & CI/CD: Databricks Asset Bundles, Git/GitHub workflows (linking PRs to Issues), and automated deployment strategies.

  • Testing & Quality: Proven methodologies for data quality assurance, automated testing, and pipeline monitoring.

Nice to Have
  • Experience with Databricks AI services such as Model Serving, Mosaic AI Agents, Vector Search, or Genie Spaces.

  • Familiarity with Python API development using FastAPI and Pydantic.

  • Exposure to financial domain data, such as Concur expense data, 73Strings data, or fund accounting analytics.

  • Experience with observability tools, including OpenTelemetry logs, spans, and metrics.

Soft Skills
  • Advanced English proficiency (written and verbal).

  • Strong communication skills for working directly with business users and translating business needs into technical solutions.

  • Ability to ramp up quickly on existing codebases and shared enterprise packages.

  • Highly collaborative mindset, capable of integrating smoothly with established development teams and processes.

At Growth Acceleration Partners, we’re an equal opportunity employer committed to building a diverse and inclusive team. We value everyone’s unique background, and we provide equal opportunities regardless of race, color, creed, religion, sexual orientation, gender identity, age, national origin, disability, marital status, veteran status or any other personal right protected by law. We foster a culture of belonging and strive to provide a welcoming environment where everyone feels safe to contribute and grow.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Databricks Engineer
Senior Databricks Engineer

Growth Acceleration Partners • United States

On-site
USD 140,000 - 180,000
Databricks Engineer
Databricks Engineer

Dataflix • United States

Remote
USD 130,000 - 170,000
Senior Databricks Engineer
Senior Databricks Engineer

Remote • San Francisco (CA)

On-site
USD 150,000 - 190,000
Senior Azure Databricks Data Engineer
Senior Azure Databricks Data Engineer

EXL • Detroit (MI)

On-site
USD 120,000 - 190,000
Databricks Data Engineer
Databricks Data Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 190,000
Databricks Data Engineer
Databricks Data Engineer

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 140,000 - 180,000
Databricks Data Engineer
Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 170,000 - 230,000
Data Engineer
Data Engineer

Catena Operations Ltd. • Malta (MT)

On-site
USD 90,000 - 140,000
Databricks Data Architect
Databricks Data Architect

NewVision Software • United States

Remote
USD 120,000 - 180,000
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 160,000