Sr Data Engineer

SiloSmashers; Arlington (VA); On-site; USD 120,000 - 190,000; Full time

Job summary

SiloSmashers, Inc. in Arlington, Virginia, seeks a Senior Data Engineer to architect, develop, and optimize Databricks Lakehouse pipelines for a government client. You will lead technical decisions, drive medallion-architecture, and ensure governance across enterprise platforms.

The role requires expertise in PySpark, Delta Lake, Unity Catalog, and CI/CD for Databricks, with experience in federal environments and DevSecOps practices. Regular site attendance is expected.

Qualifications

• Eight+ years of data engineering/ETL in enterprise cloud environments.
• Bachelor's or higher in relevant field, or 12 years of equivalent experience.
• Databricks Data Engineer Professional Certification and cloud certifications preferred.

Responsibilities

• Serve as principal authority for Databricks data engineering and Lakehouse standards.
• Lead design reviews and architectural decisions with federal SMEs and platform teams.
• Build and optimize batch and streaming pipelines with PySpark, Delta Lake, and Auto Loader.
• Enforce medallion-architecture ingestion and governed data storage tiering.
• Implement performance tuning and cost-optimization for Databricks workloads.
• Provide root cause analysis and production support for pipelines and jobs.
• Coordinate with Privacy & Governance for Unity Catalog RBAC/ABAC and lineage.

Skills

Databricks Lakehouse

PySpark

Delta Lake

CI/CD pipelines

AWS/Azure

Unity Catalog

Security compliance

Mentoring

Agile

ETL/ELT

Education

Bachelor's degree in CS/IS/Engineering

12 years total equivalent experience

Tools

GitHub Actions

Azure DevOps

Terraform

Databricks Repos

Airflow

Job description

SiloSmashers, Inc. is a leading management and IT consulting firm, serving as a trusted advisor and solutions partner to a diverse range of government, business, and not-for-profit organizations. Applying proven expertise and methodologies, we help our customers align and mobilize every aspect of their operations, strengths, and resources to achieve Peak Performance and increased business value today and through the years ahead.

Position Overview

SiloSmashers is hiring for a Senior Data Engineer to support a government customer. This position will be responsible for architecting, developing, optimizing, and maintaining high‑performance data ingestion, transformation, streaming, and analytical pipelines within Databricks‑enabled Lakehouse environments. The selected candidate will serve as a senior technical expert for Databricks Lakehouse engineering, advanced Delta Lake optimization, governed data design (Unity Catalog), medallion‑architecture enforcement, structured streaming, and platform‑integrated cloud data engineering across enterprise platforms. This role is expected to collaborate closely with the Project Manager / Contractor Lead, federal technical teams, and cross‑task platform, SRE, governance, privacy, and development personnel.

Key Responsibilities

• Serve as the principal technical authority for Databricks data engineering, pipeline optimization, streaming, and Lakehouse standards across programs.
• Lead design reviews, technical roadmaps, and architectural decisions in partnership with federal SMEs and platform/security teams.
• Build and optimize batch and streaming pipelines using PySpark, Lakeflow Spark Declarative Pipelines, Delta Lake, Auto Loader, and Workflows with robust error handling, retries, alerting, and monitoring.
• Lead and enforce medallion‑architecture ingestion and refinement patterns, data lifecycle management, and governed data storage tiering.
• Implement advanced performance tuning and cost‑efficiency strategies, including cluster/sizing optimization, serverless compute adoption, workload health monitoring, and resource governance.
• Provide root cause analysis, disruption/outage reporting, and production support for Databricks jobs, pipelines, applications, and platform features.
• Coordinate with Privacy & Governance teams to enforce Unity Catalog RBAC/ABAC, row/column‑level security, governed tags, lineage, and compliance controls.
• Mentor mid‑level/junior engineers, perform code reviews, and uphold high engineering quality through testing strategies, quality gates, reproducible builds, and DevSecOps best practices.

Required Skills

• Demonstrated expertise designing and implementing Databricks Lakehouse solutions, including:
  • Delta Lake table design, schema evolution, expectations, and medallion‑architecture (bronze/silver/gold).
  • Databricks Workflows, Jobs, Auto Loader, COPY INTO, Structured Streaming, Databricks SQL Warehouses, and Unity Catalog governance models.
  • Performance and cost optimization (Photon, file sizing, Z‑ORDER, clustering/partitioning, caching, AQE, shuffle/skew mitigation).
• Hands‑on experience building distributed data pipelines using Python, PySpark, and cloud‑native services across AWS and/or Microsoft Azure.
• Demonstrated experience administering and optimizing relational and NoSQL data systems (e.g., PostgreSQL, Aurora, DynamoDB) and designing efficient, secure schemas.
• Experience developing and maintaining CI/CD pipelines using Git‑based workflows (e.g., GitHub Actions, Azure DevOps), Databricks Repos, Asset/Bundles, Terraform/IaC, and automated testing frameworks.
• Experience supporting federal environments and compliance‑aligned development.
• Experience applying agile development methodologies, including backlog management, sprint planning, and iterative delivery.
• Experience in AI/ML DevSecOps or agentic‑AI engineering, including Databricks Model Serving, Genie, Agent Bricks, or integration with AWS Kiro or other government‑approved platforms. Mandatory

Qualifications & Requirements

• Ability to obtain and maintain a Public Trust clearance.
• Eight (8) or more years of experience (7 of which must be within the last 10 years) in data engineering, ETL/ELT pipeline development, or distributed data processing within enterprise cloud environments.
• Bachelor's or master's degree in computer science, information systems, engineering, or related field; or twelve (12) total years of equivalent experience in lieu of a degree.
• Databricks Data Engineer Professional Certification
• At least one cloud-focused certification (e.g. AWS Developer Associate, Solutions Architect Associate/Professional, or equivalent).
• Preferred but not required: Databricks Lakehouse Architect, Generative AI Associate/Professional, or related ML/AI certifications.
• This position requires regular, physical attendance at customer sites.