Databricks Data Engineer Lead: Spark & ETL on AWS

Red Cedar Consultancy, LLC; Washington (District of Columbia); On-site; USD 110,000 - 115,000; Full time

Benefits offered by this job

401(k) matching

Dental insurance

Health insurance

Health savings account

Paid time off

Professional development assistance

Referral program

Vision insurance

Job summary

Red Cedar Consultancy, LLC is seeking a Senior Data Engineer / Databricks Specialist to support a federal project by building high-performance Spark applications and a secure AWS Databricks data platform.

You will design and maintain ETL/ELT pipelines using Medallion Architecture, optimize Spark jobs, manage Databricks resources, and ensure DevSecOps controls to meet FISMA/FedRAMP requirements. Onsite in Washington, DC with a High-Risk Public Trust clearance needed.

Qualifications

• 3+ years of dedicated, hands-on Databricks development.
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
• Strong proficiency in Python and SQL, plus REST API data ingestion.
• Experience with AWS S3, IAM, Lambda, Glue, and Aurora.

Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
• Design, build, and maintain ETL/ELT pipelines following Medallion Architecture.
• Administer Databricks resources: workspaces, clusters, scheduled jobs, access controls, Unity Catalog, notebooks.
• Manage AWS services (S3, Lambda, IAM, Aurora, Glue) and provision infrastructure with Terraform.
• Optimize queries and Spark jobs; troubleshoot large-scale data processing solutions.
• Integrate structured, semi-structured, and unstructured data from APIs and databases.
• Implement security and compliance in DevSecOps CI/CD for FISMA/FedRAMP and ATO reassessments.
• Add data quality checks, anomaly detection, and automated lineage tracking.
• Collaborate in Agile/Scrum for multi-contractor federal environments; provide technical leadership.

Skills

Databricks development

PySpark

Spark SQL

Delta Lake

Python

SQL

REST API

AWS S3

IAM

Lambda

Glue

Terraform

Jenkins

GitLab CI

Agile/Scrum

High-Risk Public Trust clearance

Education

Bachelor’s degree in Computer Science / Information Technology / Data Science

Tools

Unity Catalog

Delta Live Tables (DLT)

MLflow

Model Registry

Databricks workspaces

Databricks clusters

Job description

Red Cedar Consultancy, LLC is seeking a Senior Data Engineer / Databricks Specialist to support a federal project by building high-performance Spark applications and a secure AWS Databricks data platform.

You will design and maintain ETL/ELT pipelines using Medallion Architecture, optimize Spark jobs, manage Databricks resources, and ensure DevSecOps controls to meet FISMA/FedRAMP requirements. Onsite in Washington, DC with a High-Risk Public Trust clearance needed.