Senior Databricks Lakehouse Engineer - Streaming Lead

SiloSmashers; Arlington (VA); On-site; USD 120,000 - 190,000; Full time

Job summary

SiloSmashers, Inc. in Arlington, Virginia, seeks a Senior Data Engineer to architect, develop, and optimize Databricks Lakehouse pipelines for a government client. You will lead technical decisions, drive medallion-architecture, and ensure governance across enterprise platforms.

The role requires expertise in PySpark, Delta Lake, Unity Catalog, and CI/CD for Databricks, with experience in federal environments and DevSecOps practices. Regular site attendance is expected.

Qualifications

• Eight+ years of data engineering/ETL in enterprise cloud environments.
• Bachelor's or higher in relevant field, or 12 years of equivalent experience.
• Databricks Data Engineer Professional Certification and cloud certifications preferred.

Responsibilities

• Serve as principal authority for Databricks data engineering and Lakehouse standards.
• Lead design reviews and architectural decisions with federal SMEs and platform teams.
• Build and optimize batch and streaming pipelines with PySpark, Delta Lake, and Auto Loader.
• Enforce medallion-architecture ingestion and governed data storage tiering.
• Implement performance tuning and cost-optimization for Databricks workloads.
• Provide root cause analysis and production support for pipelines and jobs.
• Coordinate with Privacy & Governance for Unity Catalog RBAC/ABAC and lineage.

Skills

Databricks Lakehouse

PySpark

Delta Lake

CI/CD pipelines

AWS/Azure

Unity Catalog

Security compliance

Mentoring

Agile

ETL/ELT

Education

Bachelor's degree in CS/IS/Engineering

12 years total equivalent experience

Tools

GitHub Actions

Azure DevOps

Terraform

Databricks Repos

Airflow

Job description

SiloSmashers, Inc. in Arlington, Virginia, seeks a Senior Data Engineer to architect, develop, and optimize Databricks Lakehouse pipelines for a government client. You will lead technical decisions, drive medallion-architecture, and ensure governance across enterprise platforms.

The role requires expertise in PySpark, Delta Lake, Unity Catalog, and CI/CD for Databricks, with experience in federal environments and DevSecOps practices. Regular site attendance is expected.