Databricks Data Engineer | Spark, Delta Lake on AWS

Red Cedar Consultancy, LLC; Maryland; On-site; USD 150,000 - 190,000; Full time

Job summary

Red Cedar Consultancy, LLC seeks a Senior Data Engineer / Databricks Specialist to design and optimize high-performance Spark applications and enterprise data infrastructure in AWS Databricks for a federal project. You will build scalable ETL/ELT pipelines using Medallion Architecture and ensure data integrity and lineage across Bronze, Silver, and Gold layers.

The role requires hands-on PySpark, Python, SQL, and Delta Lake, plus experience with Terraform, CI/CD, and secure DevSecOps practices

Qualifications

• 3+ years of hands-on Databricks development experience
• Production experience with PySpark, Spark SQL, Delta Lake
• Strong Python and SQL skills; REST API based data ingestion
• Experience with AWS S3, IAM, Lambda, Glue, Aurora
• Terraform IaC and CI/CD tools (Jenkins, GitLab CI)
• Agile/Scrum delivery environment experience
• Clearance: High-Risk Public Trust (Tier 4) or equivalent

Responsibilities

• Develop and optimize scalable Spark apps in AWS Databricks using Python, PySpark, and SQL
• Design, build, and maintain Medallion Architecture ETL/ELT pipelines
• Administer Databricks workspaces, clusters, jobs, and Unity Catalog
• Manage AWS services (S3, Lambda, IAM, Glue, Aurora) and Terraform provisioning
• Optimize queries and Spark jobs; troubleshoot large-scale data processing
• Ingest structured, semi-structured, and unstructured data from APIs/databases
• Integrate security/compliance controls into DevSecOps CI/CD for FedRAMP/ATO
• Implement data quality checks, lineage, and anomaly detection
• Collaborate with Agile/Scrum teams; provide technical leadership and on-call support

Skills

Databricks development

PySpark

Python

SQL

REST API data ingestion

AWS basics

Agile / Scrum

Education

Bachelor's or higher in Computer Science/IT/Data Science/Engineering

Tools

Terraform

Jenkins

GitLab CI

Unity Catalog

Delta Live Tables

MLflow

Model Registry

AWS S3

Job description

Red Cedar Consultancy, LLC seeks a Senior Data Engineer / Databricks Specialist to design and optimize high-performance Spark applications and enterprise data infrastructure in AWS Databricks for a federal project. You will build scalable ETL/ELT pipelines using Medallion Architecture and ensure data integrity and lineage across Bronze, Silver, and Gold layers.

The role requires hands-on PySpark, Python, SQL, and Delta Lake, plus experience with Terraform, CI/CD, and secure DevSecOps practices