Data Engineer: Databricks Lakehouse & Streaming Pipelines

SiloSmashers; Arlington (VA); On-site; USD 120,000 - 180,000; Full time

Job summary

SiloSmashers, Inc. is seeking an experienced Data Engineer to design, build, and optimize Databricks Lakehouse data pipelines for a government client.

The role involves implementing batch and streaming ETL/ELT processes using PySpark, Delta Lake, and structured streaming, with cloud services across AWS or Azure. You will collaborate with federal teams, ensure secure data integrations, and participate in CI/CD workflows while maintaining compliance and data governance requirements.

Qualifications

• Ability to obtain and maintain a Public Trust clearance.
• 8+ years of data engineering/ETL/ELT within cloud environments.
• Bachelor's degree or 4+ years of equivalent experience in last 6 years.
• Databricks Data Engineer Associate Certification.
• At least one foundational cloud certification (AWS/Azure/GCP).
• Preferred: Databricks Lakehouse Architect or Generative AI certifications.
• Regular, physical attendance at customer sites is required.

Responsibilities

• Architect, develop, and maintain high-performance data ingestion, transformation, streaming, and analytics pipelines in Databricks Lakehouse environments.
• Collaborate with federal technical teams and cross-task platform, SRE, governance, privacy, and development personnel.
• Leverage cloud native services (AWS/Azure) for secure data integrations and platform configurations.
• Work with Git-based workflows and CI/CD tooling (GitHub Actions, Azure DevOps).
• Explore introductory AI/ML-enabled engineering features in cloud-native platforms.

Skills

Databricks Lakehouse

Databricks Workflows/Jobs

Structured Streaming

PySpark

Cloud (AWS/Azure)

Git CI/CD

Terraform/IaC basics

AI/ML awareness

Education

Bachelor's degree in CS/IS/Engineering

4+ years experience in lieu of degree

Tools

Git

Azure DevOps

Job description

SiloSmashers, Inc. is seeking an experienced Data Engineer to design, build, and optimize Databricks Lakehouse data pipelines for a government client.

The role involves implementing batch and streaming ETL/ELT processes using PySpark, Delta Lake, and structured streaming, with cloud services across AWS or Azure. You will collaborate with federal teams, ensure secure data integrations, and participate in CI/CD workflows while maintaining compliance and data governance requirements.