Senior Databricks Data Engineer - Spark & AWS

Red Cedar Consultancy, LLC; Washington (District of Columbia); On-site; USD 130,000 - 180,000; Full time

Job summary

Red Cedar Consultancy, LLC is seeking a Senior Data Engineer to design and optimize high-performance Spark applications in AWS Databricks. The role focuses on scalable ETL/ELT pipelines, Medallion Architecture, and secure, compliant data solutions for mission-critical federal environments.

The candidate will administer Databricks workspaces and AWS services, lead data integration efforts, and collaborate with Agile teams while supporting DevSecOps practices and production on-call needs.

Qualifications

• 3+ years of hands-on Databricks development experience.
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
• Strong Python and SQL skills with REST API data ingestion.
• Hands-on experience with AWS S3, IAM, Lambda, Glue, and Aurora.
• Terraform IaC and CI/CD tools (Jenkins, GitLab CI).
• Able to work in Agile/Scrum environments and federal contracts.
• Bachelor's degree in Computer Science, IT, Data Science, or related field.
• Active or previous High-Risk Public Trust clearance.

Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
• Design, build, and maintain ETL/ELT pipelines using Medallion Architecture (Bronze, Silver, Gold) with data integrity and lineage.
• Administer Databricks workspaces, clusters, scheduled jobs, access controls, Unity Catalog, and notebooks.
• Manage AWS services including S3, Lambda, IAM, Aurora, and Glue; provision infrastructure with Terraform.
• Optimize queries and Spark jobs; troubleshoot large-scale data processing solutions.
• Integrate structured, semi-structured, and unstructured data from APIs and databases.
• Embed security/compliance in DevSecOps CI/CD for FISMA/FedRAMP, ATO reassessments, and POA&M.
• Implement data quality checks, anomaly detection, validation rules, and automated lineage tracking.
• Collaborate with cross-functional Agile/Scrum teams; provide technical leadership and on-call support.

Skills

Databricks development

PySpark

Spark SQL

Delta Lake

Python

SQL

REST API ingestion

Agile/Scrum

Data quality & lineage

Data engineering

Education

Bachelor's degree in Computer Science, Information Technology, Data Science, or related field

Tools

AWS S3

IAM

Lambda

Glue

Aurora

Terraform

Jenkins

GitLab CI

Unity Catalog

Delta Live Tables

MLflow

Terraform

Job description

Red Cedar Consultancy, LLC is seeking a Senior Data Engineer to design and optimize high-performance Spark applications in AWS Databricks. The role focuses on scalable ETL/ELT pipelines, Medallion Architecture, and secure, compliant data solutions for mission-critical federal environments.

The candidate will administer Databricks workspaces and AWS services, lead data integration efforts, and collaborate with Agile teams while supporting DevSecOps practices and production on-call needs.