Senior Databricks Engineer - AWS Spark & Data Pipelines

Red Cedar Consultancy; Washington (District of Columbia); On-site; USD 140,000 - 180,000; Full time

Job summary

Red Cedar Consultancy seeks a Databricks Specialist to support a federal project in Washington, DC. You will design and optimize high-performance Spark applications and enterprise data infrastructure in AWS Databricks, focusing on scalable ETL/ELT pipelines with PySpark, SQL, Delta Lake, and Medallion Architecture.

Responsibilities include administering Databricks workspaces, managing AWS services, and implementing DevSecOps practices; collaborate with cross-functional Agile teams in

Qualifications

• Bachelor's degree or higher in CS/IT/Data Science or a related engineering discipline.
• 3+ years of hands-on Databricks / Spark development experience.
• Proficient in Python and SQL with experience in REST API data ingestion.
• Strong AWS experience (S3, IAM, Lambda, Glue, Aurora).
• Experience with Terraform and CI/CD tools (Jenkins, GitLab CI).
• Experience in Agile/Scrum delivery environments.
• Active or prior High-Risk Public Trust clearance.

Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks.
• Design, build, and maintain ETL/ELT pipelines using Medallion Architecture (Bronze/Silver/Gold).
• Administer Databricks workspaces, clusters, jobs, and Unity Catalog.
• Manage AWS services and provision infrastructure with Terraform.
• Optimize queries and Spark jobs and troubleshoot large-scale data processing.
• Implement data quality checks and automated lineage tracking.
• Collaborate with Agile teams in multi-contractor federal environments and provide technical leadership.

Skills

Python

SQL

REST API data ingestion

Agile/Scrum

Data engineering

Education

Bachelor's degree in Computer Science, Information Technology, Data Science, or related field

Databricks Certified Data Engineer Associate

Databricks Certified Data Engineer Professional

Databricks Certified Machine Learning Associate

AWS Certified Solutions Architect – Associate/Professional

AWS Certified Data Analytics – Specialty

Tools

Databricks

Apache Spark

Delta Lake

Terraform

Jenkins

GitLab CI

AWS S3

IAM

Glue

Aurora

Job description

Red Cedar Consultancy seeks a Databricks Specialist to support a federal project in Washington, DC. You will design and optimize high-performance Spark applications and enterprise data infrastructure in AWS Databricks, focusing on scalable ETL/ELT pipelines with PySpark, SQL, Delta Lake, and Medallion Architecture.

Responsibilities include administering Databricks workspaces, managing AWS services, and implementing DevSecOps practices; collaborate with cross-functional Agile teams in