Data Engineer, Databricks (Senior)

Full Tilt Data, LLC; United States; Hybrid; USD 120,000 - 170,000; Full time

Benefits offered by this job

Health benefits

Discretionary bonuses

Reimbursement for professional dev/trn

Job summary

Full Tilt Data, LLC is seeking a Senior Data Engineer specializing in Databricks to deliver secure, scalable data solutions for a federal program. The role focuses on Databricks/PySpark pipelines, data modeling, and AI-enabled search capabilities, with a background check requirement and quarterly in-person office visits.

Remote work is allowed with periodic in-person meetings; industry-leading data governance, CI/CD, and Delta Sharing practices are emphasized, along with mentoring a growing team

Qualifications

• 7–10+ years in data engineering with 3–4+ years on Databricks/PySpark.
• Leadership of complex pipelines in production Databricks environments.
• Strong dimensional data modeling with star schema and fact/dimension design.
• Hands-on RAG or vector search in production or near-production.
• CI/CD design and Databricks Asset Bundles (DAB) experience.
• Experience with policy-as-code (OPA/Rego) integration is a plus.

Responsibilities

• Own and drive architecture decisions for Databricks pipelines for performance and reliability.
• Redesign star schema and unified data model structures, including fact/dimension tables.
• Architect and implement RAG and vector search for AI-assisted querying.
• Define and enforce DAB and CI/CD best practices across the team.
• Integrate OPA/Rego with Databricks and Spark for dynamic access control.
• Establish patterns for Delta Sharing, OCR, and Databricks Apps.
• Mentor engineers, review designs, and support capacity rebuilding.
• Represent engineering in standups and client technical discussions.

Skills

Databricks/PySpark

Python

SQL

Mentoring

CI/CD

Data modeling

OPA/Rego (policy-as-code)

Tools

Databricks

Unity Catalog

Job description

Full Tilt Data, LLC is a trusted data, analytics, and IT consulting firm specializing is health related services for the federal government. Established in 2023 by a group of founders that bring 15+ years of industry experience. We are passionate about harnessing the power of data through our comprehensive data management solutions. We mobilize the right people, skills, and technologies to help all types of organizations and companies improve their performance and data management.

Position Summary

We are looking for a Data Engineer, Databricks (Senior) to help deliver modern, secure data solutions for a high-impact federal program. In this role, you will support the development of scalable cloud lakehouse capabilities, data pipelines, access-control frameworks, applications, and APIs that enable federal agencies to integrate, analyze, and share mission-critical data. Candidates must be able to pass a background check equivalent to a Federal Public Trust clearance. This is a remote position, with a requirement to come into the office quarterly for in-person meetings.

Key Responsibilities

• Own and drive architecture decisions for complex Databricks pipelines, including performance, scalability, and reliability improvements.
• Lead redesign and improvement of star schema / unified data model (UDM) structures, including fact and dimension table strategy, for the program's canonical data model.
• Architect and implement RAG and vector search capability to power AI-assisted contract search and natural language querying, including approach reuse across related contracts.
• Define and enforce Databricks Asset Bundles (DAB) and CI/CD best practices across the engineering team, improving deployment consistency and release velocity.
• Lead integration of the OPA/Rego policy decision framework with Databricks and Spark, enabling dynamic, query-time enforcement of access controls, masking, and usage restrictions.
• Establish patterns for Delta Sharing, text extraction/OCR, and Databricks Apps as the team expands into these areas.
• Package, document, and harden solutions into reference implementations and example configurations that enable adoption by additional agencies without direct contractor involvement.
• Mentor mid-level engineers, review designs and code, and help fill technical gaps as the team rebuilds capacity in these focus areas.
• Represent engineering in standups, monthly status reporting, and technical discussions with the client.

Databricks Focus Areas for This Role

Candidates Should Have Direct, Hands-on Experience In One Or More Of The Following, And Be Comfortable Ramping Quickly Across The Rest

• Complex pipeline development and improvement on Databricks/PySpark
• Star schema and unified data model (UDM) design and refactoring
• RAG / vector search implementation for search and natural language querying use cases
• Fact and dimension table design for analytic workloads
• Databricks Asset Bundles (DAB) and CI/CD best practices
• Delta Sharing, text extraction/OCR, and Databricks Apps (secondary priority areas)

Required Qualifications

• 7-10+ years of data engineering experience, with at least 3-4+ years of deep, hands-on Databricks/PySpark experience.
• Demonstrated experience leading complex pipeline design and improvement in a production Databricks environment.
• Strong track record with dimensional data modeling - star schema, fact/dimension table design - at scale.
• Hands-on experience implementing RAG and/or vector search capabilities in a production or near-production setting.
• Strong command of Databricks Asset Bundles (DAB) and CI/CD pipeline design and best practices.
• Advanced Python and SQL skills; comfortable owning integration code end-to-end.
• Demonstrated ability to mentor other engineers and set technical direction for a team.
• Ability to obtain/maintain a Federal Public Trust clearance.

Preferred Qualifications

• Experience integrating Open Policy Agent (OPA) / Rego or comparable policy-as-code frameworks with a data platform.
• Experience with Delta Sharing, OCR/text extraction pipelines, or Databricks Apps.
• Experience with a general-purpose backend language and a modern frontend framework for adjacent backend/API or UI development.
• Prior experience on a federal contract.
• Databricks certification(s) at the Professional level (Data Engineer Professional; Generative AI Engineer Associate is a plus given the RAG scope of this role).
• Experience standing up policy or governance frameworks for multi-agency data sharing.
• Familiarity with Unity Catalog governance features (fine-grained access control, row/column-level security, data lineage) at scale.
• Familiarity with federal compliance frameworks (e.g., NIST 800-53, FISMA, ATO processes) or experience handling CUI/PII.
• An active Public Trust (or higher) clearance or investigation already in process.
• Experience with data quality/testing frameworks.
• Experience setting technical direction or serving as a de facto tech lead on a data engineering team, independent of formal management title.

Compensation

The salary range provided represents the estimated compensation for new hires in this position, applicable across all locations. Actual offers may vary based on factors such as the candidate's skills, qualifications, experience, and market conditions. Full Tilt Data complements its base salary offering with a competitive package that includes health benefits, discretionary bonuses, and reimbursement for professional development and training.

Benefits

• health benefits
• discretionary bonuses
• reimbursement for professional development and training

Full Tilt Data provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.