Senior Data Engineer

SiloSmashers, Inc.; Alexandria (VA); On-site; USD 140,000 - 155,000; Full time

Job summary

SiloSmashers, Inc. is hiring a Senior Data Engineer in Alexandria, VA to architect, develop, and optimize Databricks Lakehouse pipelines for a government customer. You will lead streaming and batch data flows, enforce governance, and mentor engineers while collaborating with federal teams.

The role requires 8+ years of data engineering experience, strong Databricks expertise, and cloud platform proficiency (AWS/Azure). On-site work in Alexandria, VA is expected.

Qualifications

• Eight or more years in data engineering or ETL/ELT in enterprise cloud environments.
• Experience designing and operating Databricks Lakehouse architectures.
• Ability to obtain and maintain a Public Trust clearance.

Responsibilities

• Architect, develop, optimize, and maintain high‑performance data ingestion, transformation, streaming, and analytics pipelines.
• Lead design reviews and architectural decisions with federal SMEs and security teams.
• Mentor engineers and uphold DevSecOps practices across pipelines.

Skills

Databricks

PySpark

Delta Lake

Data pipelines

Cloud platforms

Education

Bachelor’s or Master’s in CS/IS/Engineering

Tools

Databricks

AWS

Azure

Job description

If you are unable to complete this application due to a disability, contact this employer to ask for an accommodation or an alternative application process.

Full Time Alexandria, VA, US

2 days ago Requisition ID: 1002

Salary Range: $140,000.00 To $155,000.00 Annually

SiloSmashers is hiring for a Senior Data Engineer to support a government customer. This position will be responsible for architecting, developing, optimizing, and maintaining high ‑ performance data ingestion, transformation, streaming, and analytical pipelines within Databricks ‑ enabled Lakehouse environments.

The selected candidate will serve as a senior technical expert for Databricks Lakehouse engineering, advanced Delta Lake optimization, governed data design (Unity Catalog), medallion ‑ architecture enforcement, structured streaming, and platform ‑ integrated cloud data engineering across enterprise platforms. This role is expected to collaborate closely with the Project Manager / Contractor Lead, federal technical teams, and cross ‑ task platform, SRE, governance, privacy, and development personnel.

• Serve as the principal technical authority for Databricks data engineering, pipeline optimization, streaming, and Lakehouse standards across programs.
• Lead design reviews, technical roadmaps, and architectural decisions in partnership with federal SMEs and platform/security teams.
• Build and optimize batch and streaming pipelines using PySpark, Lakeflow Spark Declarative Pipelines, Delta Lake, Auto Loader, and Workflows with robust error handling, retries, alerting, and monitoring.
• Lead and enforce medallion‑architecture ingestion and refinement patterns, data lifecycle management, and governed data storage tiering.
• Implement advanced performance tuning and cost‑efficiency strategies, including cluster/sizing optimization, serverless compute adoption, workload health monitoring, and resource governance.
• Provide root cause analysis, disruption/outage reporting, and production support for Databricks jobs, pipelines, applications, and platform features.
• Coordinate with Privacy & Governance teams to enforce Unity Catalog RBAC/ABAC, row/column‑level security, governed tags, lineage, and compliance controls.
• Mentor mid‑level/junior engineers, perform code reviews, and uphold high engineering quality through testing strategies, quality gates, reproducible builds, and DevSecOps best practices.
• Demonstrated expertise designing and implementing Databricks Lakehouse solutions, including:
  • Delta Lake table design, schema evolution, expectations, and medallion‑architecture (bronze/silver/gold).
  • Databricks Workflows, Jobs, Auto Loader, COPY INTO, Structured Streaming, Databricks SQL Warehouses, and Unity Catalog governance models.
  • Performance and cost optimization (Photon, file sizing, Z‑ORDER, clustering/partitioning, caching, AQE, shuffle/skew mitigation).
• Hands‑on experience building distributed data pipelines using Python, PySpark, and cloud‑native services across AWS and/or Microsoft Azure.
• Demonstrated experience administering and optimizing relational and NoSQL data systems (e.g., PostgreSQL, Aurora, DynamoDB) and designing efficient, secure schemas.
• Experience developing and maintaining CI/CD pipelines using Git‑based workflows (e.g., GitHub Actions, Azure DevOps), Databricks Repos, Asset/Bundles, Terraform/IaC, and automated testing frameworks.
• Experience supporting federal environments and compliance‑aligned development.
• Experience applying agile development methodologies, including backlog management, sprint planning, and iterative delivery.
• Experience in AI/ML DevSecOps or agentic‑AI engineering, including Databricks Model Serving, Genie, Agent Bricks, or integration with AWS Kiro or other government‑approved platforms.
• Ability to obtain and maintain a Public Trust clearance.
• Eight (8) or more years of experience (7 of which must be within the last 10 years) in data engineering, ETL/ELT pipeline development, or distributed data processing within enterprise cloud environments.
• Bachelor’s or master’s degree in computer science, information systems, engineering, or related field; or twelve (12) total years of equivalent experience in lieu of a degree.
• Databricks Data Engineer Professional Certification
• At least one cloud-focused certification (e.g. AWS Developer Associate, Solutions Architect Associate/Professional, or equivalent).
• Preferred but not required: Databricks Lakehouse Architect, Generative AI Associate/Professional, or related ML/AI certifications.
• This position may require regular, physical attendance at customer sites.