Senior Databricks & Spark Data Engineer

Seneca Holdings; Washington (District of Columbia); On-site; USD 120,000 - 160,000; Full time

Job summary

Seneca Holdings seeks an experienced Data Engineer to design, build, and maintain scalable data pipelines within a Databricks E2 environment hosted on AWS. The role focuses on end-to-end data lifecycle from ingestion to analytics-ready datasets, with emphasis on security, governance, and multi-tenant compliance.

You will work with Databricks, PySpark, Pandas, SQL, and real-time ingestion using DMS, building medallion-layer pipelines and collaborating in an Agile team.

Qualifications

• Proven, hands-on experience as a Data Engineer or Databricks Developer building production-grade data pipelines.

Responsibilities

• Analyze and collect data from various sources, including relational databases, APIs, external data providers, and real-time streaming sources.
• Design and implement efficient, scalable data pipelines to cleanse, transform, and aggregate data for downstream reporting and analytics.
• Build and maintain Databricks solutions using medallion architecture, including Bronze, Silver, and Gold layers, to create structured, reliable, and progressively refined datasets.
• Develop source-to-target mapping documentation for all data pipelines and conduct thorough unit testing to validate data accuracy and pipeline logic.
• Write advanced SQL to create aggregations across complex datasets and analyze data for anomalies, quality issues, and inconsistencies.
• Implement real-time and near-real-time data ingestion using AWS Database Migration Service (DMS) and other AWS-native services.
• Develop data-processing logic using Python and/or R, leveraging Spark and related packages such as PySpark and Pandas for large-scale data transformation.
• Manage source code, versioning, and deployments using GitLab, and support automated build and release processes through CI/CD pipelines.
• Collaborate with cross-functional teams in an Agile project environment, participating in sprint planning, stand-ups, and iterative delivery.
• Ensure that data engineering practices align with FISMA High and multi-tenant security, governance, and compliance requirements.
• Leverage AI automation tools to support data engineering pipeline development, testing, and validation and to accelerate delivery.

Skills

Databricks on AWS

Apache Spark

PySpark

Pandas

Python

R

GitLab

CI/CD pipelines

Agile/Scrum

SQL

Delta Lake

DMS

RDS

Terraform

Spark Structured Streaming

Kafka/Kinesis

Tools

Databricks

AWS

Delta Lake

DMS

RDS

GitLab

CI/CD

Terraform/CloudFormation

Kafka/Kinesis

Job description

Seneca Holdings seeks an experienced Data Engineer to design, build, and maintain scalable data pipelines within a Databricks E2 environment hosted on AWS. The role focuses on end-to-end data lifecycle from ingestion to analytics-ready datasets, with emphasis on security, governance, and multi-tenant compliance.

You will work with Databricks, PySpark, Pandas, SQL, and real-time ingestion using DMS, building medallion-layer pipelines and collaborating in an Agile team.