Data Engineer, Databricks (Mid-Level)

Full Tilt Data, LLC; United States; Hybrid; USD 110,000 - 160,000; Full time

Benefits offered by this job

Health benefits

Discretionary bonuses

Reimbursement for professional develop

Job summary

Full Tilt Data, LLC is a trusted data, analytics, and IT consulting firm serving federal health-related programs. We seek a Mid-Level Data Engineer, Databricks to build scalable lakehouse capabilities, data pipelines, and access-control mechanisms for a high-impact federal program.

You will work on Data modeling, Delta Sharing, and AI-enabled search features, with occasional in-person meetings quarterly. A Federal Public Trust clearance is required or in process.

Qualifications

• 3-5+ years of data engineering experience, with 1-2 years in Databricks.
• Strong Python and PySpark skills for building data pipelines.
• Knowledge of star schema, fact/dimension tables, and dimensional modeling.
• Experience with SQL and structured/semi-structured data sources.
• Familiarity with Git-based CI/CD deployment workflows.
• Ability to obtain/maintain a Federal Public Trust clearance.
• Comfortable in a fast-paced, collaborative team environment.

Responsibilities

• Design, build, and maintain data pipelines on Databricks using Python/PySpark.
• Develop canonical data models (UDM) and star schemas for onboarding datasets.
• Implement Retrieval-Augmented Generation and vector search for AI-driven queries.
• Contribute to DABs and CI/CD pipelines for deployment consistency.
• Create Python integrations linking OPA/Rego policies to Databricks/Spark.
• Support Delta Sharing, OCR workflows, and Databricks Apps.
• Package patterns into reusable references for agency use.
• Participate in weekly standups and monthly status reports.

Skills

Python

PySpark

Dimensional data modeling

SQL

CI/CD

Fast-paced environment

Tools

Databricks

Git-based workflows

OPA / Rego

Databricks Asset Bundles (DAB)

Delta Sharing

Unity Catalog

Vector search tooling

Job description

Full Tilt Data, LLC is a trusted data, analytics, and IT consulting firm specializing is health related services for the federal government. Established in 2023 by a group of founders that bring 15+ years of industry experience. We are passionate about harnessing the power of data through our comprehensive data management solutions. We mobilize the right people, skills, and technologies to help all types of organizations and companies improve their performance and data management.

Position Summary

We are looking for a Data Engineer, Databricks (Mid-Level) to help deliver modern, secure data solutions for a high-impact federal program. In this role, you will support the development of scalable cloud lakehouse capabilities, data pipelines, access-control frameworks, applications, and APIs that enable federal agencies to integrate, analyze, and share mission-critical data. Candidates must be able to pass a background check equivalent to a Federal Public Trust clearance. This is a remote position, with a requirement to come into the office quarterly for in-person meetings.

Key Responsibilities

• Design, build, and maintain data pipelines on Databricks using Python and PySpark, including schema normalization, metadata capture, data quality checks, and lineage tracking.
• Develop and improve canonical/unified data model (UDM) structures, including star schema, fact, and dimension table design, to support repeatable onboarding of new agency datasets.
• Build and integrate Retrieval-Augmented Generation (RAG) and vector search capabilities to support AI-assisted contract search and natural language querying.
• Contribute to Databricks Asset Bundles (DAB) and CI/CD pipelines to improve deployment consistency, testing, and release velocity.
• Develop Python-based integration layers connecting the OPA/Rego policy engine to Databricks and Spark, enabling dynamic enforcement of access controls and data masking at query time.
• Support Delta Sharing configurations, text extraction/OCR workflows, and Databricks Apps as the team expands into these areas.
• Package, document, and harden pipeline and data model patterns into reusable, well-documented reference implementations that other agencies and teams can adopt independently.
• Participate in weekly standups and contribute to monthly status reporting on task progress and milestones.

Databricks Focus Areas for This Role

• Complex pipeline development and improvement on Databricks/PySpark
• Star schema and unified data model (UDM) design and refactoring
• RAG / vector search implementation for search and natural language querying use cases
• Fact and dimension table design for analytic workloads
• Databricks Asset Bundles (DAB) and CI/CD best practices
• Delta Sharing, text extraction/OCR, and Databricks Apps (secondary priority areas)

Required Qualifications

• 3-5+ years of hands-on data engineering experience, including at least 1-2 years working directly in Databricks.
• Strong proficiency in Python and PySpark for building and troubleshooting data pipelines.
• Working knowledge of dimensional data modeling concepts (star schema, fact/dimension tables) and willingness to grow this into a core strength.
• Experience with SQL and working across structured and semi-structured data sources.
• Familiarity with CI/CD concepts and version-controlled deployment workflows (Git-based).
• Comfortable working in a fast-paced, collaborative team environment and picking up new tools quickly.
• Ability to obtain/maintain a Federal Public Trust clearance.

Preferred Qualifications

• Exposure to Databricks Asset Bundles (DAB) or similar infrastructure-as-code deployment tooling.
• Exposure to vector search, embeddings, or RAG-style architectures - specifically vector database/embedding tooling (e.g., Databricks Vector Search, pgvector, FAISS, or Chroma) and embedding or generation model integration (e.g., Databricks Model Serving, Azure OpenAI, Bedrock).
• Familiarity with Open Policy Agent (OPA) / Rego or other policy-as-code frameworks.
• Experience with a general-purpose backend language and a modern frontend framework for adjacent API or UI work.
• Prior experience on a federal contract or in a regulated data environment.
• Databricks certification(s) (Data Engineer Associate/Professional, or Generative AI Engineer Associate).
• Familiarity with Unity Catalog governance features (fine-grained access control, row/column-level security, data lineage).
• Familiarity with federal compliance frameworks (e.g., NIST 800-53, FISMA, ATO processes) or experience handling CUI/PII.
• An active Public Trust (or higher) clearance or investigation already in process.
• Experience with data quality/testing frameworks

The salary range provided represents the estimated compensation for new hires in this position, applicable across all locations.

Actual offers may vary based on factors such as the candidate's skills, qualifications, experience, and market conditions. Full Tilt Data complements its base salary offering with a competitive package that includes health benefits, discretionary bonuses, and reimbursement for professional development and training.

Full Tilt Data provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.