Senior Databricks Data Engineer

GAP Careers; Austin (TX); Hybrid; USD 130,000 - 190,000; Full time

Job summary

Growth Acceleration Partners (GAP) is seeking a Senior Databricks Data Engineer to design, build, and maintain scalable data pipelines within an Azure/Databricks ecosystem. You will own data modeling, pipeline orchestration, and data quality while collaborating with application developers and stakeholders.

You will leverage PySpark, SQL, and Unity Catalog to deliver governed data for enterprise applications, with strong emphasis on secure, performant data delivery and end-to-end data governance.

Qualifications

• Bachelor’s degree in a technical field is required.
• 5+ years of data engineering experience with modern cloud data architectures.
• Hands-on experience building data pipelines on Databricks with PySpark/SQL.
• Experience with Delta Lake governance and Unity Catalog.

Responsibilities

• Architect, build, and maintain scalable Databricks data pipelines.
• Design data models integrating Databricks with Azure SQL/Azure Cosmos DB.
• Orchestrate pipelines and schedule with Lakeflow Jobs.
• Ensure data quality, testing, and monitoring across processes.
• Manage deployments using Databricks Asset Bundles and Git-based CI/CD.
• Collaborate with frontend/backend developers and product owners to meet requirements.

Skills

Databricks

PySpark

SQL

Delta Lake

Unity Catalog

Azure SQL

GitHub

CI/CD

Lakeflow Jobs

Data Modeling

Education

Bachelor’s Degree in Computer Science, Data Engineering, Software Engineering, Information Systems, or a related technical field

Tools

Databricks Asset Bundles

Lakeflow

GitHub

OpenTelemetry

Job description

WHAT WE DO

Founded in 2007, Growth Acceleration Partners (GAP) is a consulting and technology services company. We consult, design, build and modernize revenue-generating software and data engineering solutions for clients. With modernization services and AI tools, we help businesses achieve a competitive advantage through technology. GAP’s remote, integrated engineering teams use end-to-end solutions to innovate and align with your business goals. We have 600+ English-speaking engineers based in Latin America and approximately 20 U.S.-based engineers. With some of the highest customer satisfaction scores in the industry, GAP’s focus is customer and employee success.

GAP is a woman-owned company headquartered in Austin Texas. We are a values-based company focused on growing our people by investing in education, onsite English classes and training in the latest technologies, including AI, data analytics and machine learning. Our goal is to provide solutions for our customers that help them achieve critical business outcomes, while enabling our GAPSters and our communities to attain long‑term success.

Summary

We are looking for a Senior Databricks Data Engineer to design, build, and maintain scalable data pipelines and integrations within a robust Azure and Databricks ecosystem. In this high-impact role, you will work as part of a dedicated team to deliver large, defined projects, seamlessly integrating Databricks with modern application stacks. You will take technical ownership of data modeling, pipeline orchestration, and data quality while collaborating closely with cross-functional application developers and business stakeholders. Operating with a high level of autonomy, you will leverage PySpark, SQL, and Unity Catalog to ensure secure, performant, and governed data delivery for critical enterprise applications.

Education

• Bachelor’s Degree in Computer Science, Data Engineering, Software Engineering, Information Systems, or a related technical field.

Professional Experience

• 5+ years of professional experience in Data Engineering, specializing in modern cloud data architectures.

• Proven hands‑on experience building and maintaining complex data pipelines on Databricks using PySpark and SQL.

• Deep expertise working with Delta Lake architectures and implementing governance via Unity Catalog.

• Demonstrated experience integrating Databricks workflows with Azure SQL or Azure Cosmos DB.

• Experience operating in Git-based CI/CD environments, specifically managing all work in GitHub where every pull request is linked to a GitHub Issue.

Key Responsibilities

• Architect, build, and maintain scalable Databricks data pipelines to support existing systems and new application integrations.

• Design and implement robust data models that integrate seamlessly between Databricks and transactional databases like Azure SQL or Azure Cosmos DB.

• Orchestrate data pipelines and schedule workflows utilizing Lakeflow Jobs.

• Enforce rigorous data quality, testing, and monitoring standards across all data engineering processes.

• Manage deployment lifecycles using Databricks Asset Bundles and Git-based CI/CD workflows.

• Collaborate continuously with frontend/backend application developers (React, C# .NET) and Product Owners to ensure data infrastructure meets application requirements.

• Operate strictly within organizational security and governance models, leveraging Microsoft Entra ID for access and Unity Catalog for data governance.

Required Technical Skills

Strong experience in several of the following areas:

• Data Engineering & Processing: Deep proficiency in Databricks, PySpark, advanced SQL, and Delta Lake.

• Governance & Orchestration: Unity Catalog (grants, lineage, access control) and Lakeflow Jobs.

• Cloud & Databases: Seamless integration with Azure SQL and Azure Cosmos DB.

• DevOps & CI/CD: Databricks Asset Bundles, Git/GitHub workflows (linking PRs to Issues), and automated deployment strategies.

• Testing & Quality: Proven methodologies for data quality assurance, automated testing, and pipeline monitoring.

Nice to Have

• Experience with Databricks AI services such as Model Serving, Mosaic AI Agents, Vector Search, or Genie Spaces.

• Familiarity with Python API development using FastAPI and Pydantic.

• Exposure to financial domain data, such as Concur expense data, 73Strings data, or fund accounting analytics.

• Experience with observability tools, including OpenTelemetry logs, spans, and metrics.

Soft Skills

• Advanced English proficiency (written and verbal).

• Strong communication skills for working directly with business users and translating business needs into technical solutions.

• Ability to ramp up quickly on existing codebases and shared enterprise packages.

• Highly collaborative mindset, capable of integrating smoothly with established development teams and processes.

At Growth Acceleration Partners, we're an equal opportunity employer committed to building a diverse and inclusive team. We value everyone's unique background, and we provide equal opportunities regardless of race, color, creed, religion, sexual orientation, gender identity, age, national origin, disability, marital status, veteran status or any other personal right protected by law. We foster a culture of belonging and strive to provide a welcoming environment where everyone feels safe to contribute and grow.