Data Engineer

SiloSmashers; Arlington (VA); On-site; USD 120,000 - 180,000; Full time

Job summary

SiloSmashers, Inc. is seeking an experienced Data Engineer to design, build, and optimize Databricks Lakehouse data pipelines for a government client.

The role involves implementing batch and streaming ETL/ELT processes using PySpark, Delta Lake, and structured streaming, with cloud services across AWS or Azure. You will collaborate with federal teams, ensure secure data integrations, and participate in CI/CD workflows while maintaining compliance and data governance requirements.

Qualifications

• Ability to obtain and maintain a Public Trust clearance.
• 8+ years of data engineering/ETL/ELT within cloud environments.
• Bachelor's degree or 4+ years of equivalent experience in last 6 years.
• Databricks Data Engineer Associate Certification.
• At least one foundational cloud certification (AWS/Azure/GCP).
• Preferred: Databricks Lakehouse Architect or Generative AI certifications.
• Regular, physical attendance at customer sites is required.

Responsibilities

• Architect, develop, and maintain high-performance data ingestion, transformation, streaming, and analytics pipelines in Databricks Lakehouse environments.
• Collaborate with federal technical teams and cross-task platform, SRE, governance, privacy, and development personnel.
• Leverage cloud native services (AWS/Azure) for secure data integrations and platform configurations.
• Work with Git-based workflows and CI/CD tooling (GitHub Actions, Azure DevOps).
• Explore introductory AI/ML-enabled engineering features in cloud-native platforms.

Skills

Databricks Lakehouse

Databricks Workflows/Jobs

Structured Streaming

PySpark

Cloud (AWS/Azure)

Git CI/CD

Terraform/IaC basics

AI/ML awareness

Education

Bachelor's degree in CS/IS/Engineering

4+ years experience in lieu of degree

Tools

Git

Azure DevOps

Job description

SiloSmashers, Inc. is a leading management and IT consulting firm, serving as a trusted advisor and solutions partner to a diverse range of government, business, and not-for-profit organizations. Applying proven expertise and methodologies, we help our customers align and mobilize every aspect of their operations, strengths, and resources to achieve Peak Performance and increased business value today and through the years ahead.

Position Overview

SiloSmashers is hiring for a Data Engineer to support digital transformation efforts for a government customer. The selected candidate will serve as part of a team responsible for architecting, developing, optimizing, and maintaining high‑performance data ingestion, transformation, streaming, and analytical pipelines within Databricks‑enabled Lakehouse environments. This role is expected to collaborate closely with the federal technical teams, and cross‑task platform, SRE, governance, privacy, and development personnel.

Required Skills

• Foundational experience designing and implementing Databricks Lakehouse solutions, including: Basic Delta Lake table design, schema evolution, and familiarity with medallion architecture (bronze/silver/gold).
• Hands on experience with Databricks Workflows/Jobs, Auto Loader, COPY INTO, Structured Streaming, and introductory Databricks SQL Warehouse usage. Understanding of common performance considerations (e.g., cluster sizing, partitioning, basic caching strategies).
• Practical experience building batch or streaming pipelines using Python and PySpark, along with cloud native services in AWS or Microsoft Azure.
• Experience implementing secure cloud data integrations or platform configurations such as external locations, storage credentials, secret scopes, or identity federation basics.
• Experience working with Git based workflows and CI/CD tooling (e.g., GitHub Actions, Azure DevOps), plus familiarity with Databricks Repos, Asset Bundles, or basic Terraform/IaC concepts.
• Introductory experience or interest in AI/ML enabled engineering, such as Databricks Model Serving, Genie, Agent Bricks, or equivalent cloud native ML platforms.

Mandatory Qualifications & Requirements

• Ability to obtain and maintain a Public Trust clearance.
• Eight (8) or more years of experience (2 of which much be within the last five (5) years) in data engineering, ETL/ELT development, or distributed data processing within cloud environments or platforms (e.g. AWS, Azure, GCP, Snowflake, Databricks, etc.).
• Bachelor's degree in computer science, information systems, engineering, or related field; or four (4) or more years of equivalent experience within the last six (6) years in lieu of a degree.
• Databricks Data Engineer Associate Certification.
• At least one foundational cloud certification (e.g., AWS Cloud Practitioner, AWS Developer Associate, Azure Fundamentals, or equivalent).
• Preferred but not required: Databricks Lakehouse Architect, Generative AI Associate/Professional, or related ML/AI certifications.
• This position requires regular, physical attendance at customer sites.