Senior Data Engineer

Red Cedar Consultancy, LLC; Washington (District of Columbia); On-site; USD 110,000 - 115,000; Full time

Benefits offered by this job

401(k) matching

Dental insurance

Health insurance

Health savings account

Paid time off

Professional development assistance

Referral program

Vision insurance

Job summary

Red Cedar Consultancy, LLC is seeking a Senior Data Engineer / Databricks Specialist to support a federal project by building high-performance Spark applications and a secure AWS Databricks data platform.

You will design and maintain ETL/ELT pipelines using Medallion Architecture, optimize Spark jobs, manage Databricks resources, and ensure DevSecOps controls to meet FISMA/FedRAMP requirements. Onsite in Washington, DC with a High-Risk Public Trust clearance needed.

Qualifications

• 3+ years of dedicated, hands-on Databricks development.
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
• Strong proficiency in Python and SQL, plus REST API data ingestion.
• Experience with AWS S3, IAM, Lambda, Glue, and Aurora.

Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
• Design, build, and maintain ETL/ELT pipelines following Medallion Architecture.
• Administer Databricks resources: workspaces, clusters, scheduled jobs, access controls, Unity Catalog, notebooks.
• Manage AWS services (S3, Lambda, IAM, Aurora, Glue) and provision infrastructure with Terraform.
• Optimize queries and Spark jobs; troubleshoot large-scale data processing solutions.
• Integrate structured, semi-structured, and unstructured data from APIs and databases.
• Implement security and compliance in DevSecOps CI/CD for FISMA/FedRAMP and ATO reassessments.
• Add data quality checks, anomaly detection, and automated lineage tracking.
• Collaborate in Agile/Scrum for multi-contractor federal environments; provide technical leadership.

Skills

Databricks development

PySpark

Spark SQL

Delta Lake

Python

SQL

REST API

AWS S3

IAM

Lambda

Glue

Terraform

Jenkins

GitLab CI

Agile/Scrum

High-Risk Public Trust clearance

Education

Bachelor’s degree in Computer Science / Information Technology / Data Science

Tools

Unity Catalog

Delta Live Tables (DLT)

MLflow

Model Registry

Databricks workspaces

Databricks clusters

Job description

The Senior Data Engineer / Databricks Specialist will support a federal project by building and optimizing high-performance Spark applications and AWS Databricks enterprise data infrastructure, with an emphasis on secure and reliable ETL/ELT for mission-critical environments.

Key Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
• Design, build, and maintain ETL/ELT pipelines following Medallion Architecture (Bronze, Silver, and Gold), ensuring data integrity, completeness, and lineage.
• Administer Databricks resources including workspaces, clusters, scheduled jobs, access controls, Unity Catalog, and notebooks.
• Manage AWS services such as S3, Lambda, IAM, Aurora, and Glue, and provision infrastructure using Terraform.
• Optimize queries and Spark jobs, and troubleshoot large-scale data processing solutions.
• Integrate structured, semi-structured, and unstructured data from APIs and databases.
• Implement security and compliance controls in DevSecOps CI/CD pipelines using Jenkins, GitLab CI, or equivalent tooling to support FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
• Add data quality checks, anomaly detection, validation rules, and automated lineage tracking.
• Work with cross-functional teams in Agile/Scrum delivery for multi-contractor federal environments, providing technical leadership and production on-call support.

Required Qualifications

• 3+ years of dedicated, hands-on Databricks development.
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
• Strong proficiency in Python and SQL, plus working knowledge of REST API based data ingestion.
• Hands-on production experience with AWS S3, IAM, Lambda, Glue, and Aurora.
• Experience with Terraform infrastructure as code and CI/CD tools such as Jenkins or GitLab CI, or equivalent.
• Experience delivering in Agile/Scrum environments.
• Bachelor’s degree in Computer Science, Information Technology, Data Science, or a related engineering discipline.
• Active or previous High-Risk Public Trust (Tier 4) clearance.

Preferred Qualifications

• Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and MLflow or Model Registry for AI/ML pipelines.
• Experience supporting federal government contracts and FISMA/FedRAMP environments.

Preferred Certifications

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Databricks Certified Machine Learning Associate
• AWS Certified Solutions Architect – Associate or Professional
• AWS Certified Data Analytics – Specialty

Technologies and Tools

• AWS Databricks, Spark, Python, PySpark, SQL, Delta Lake
• Medallion Architecture, Unity Catalog
• Databricks workspaces, Databricks clusters, Unity Catalog notebooks, Databricks Scheduled Jobs
• S3, Lambda, IAM, Aurora, Glue, Terraform
• Jenkins, GitLab CI, REST API
• Apache Spark, Delta Live Tables (DLT), MLflow, Model Registry
• DevSecOps CI/CD, ETL/ELT, Agile/Scrum

Location and Salary

Location: Washington, DC (onsite). Salary: USD 110,000 - 115,000 per year.

Benefits

• 401(k) matching
• Dental insurance
• Employee discount
• Flexible spending account
• Health insurance
• Health savings account
• Life insurance
• Paid time off
• Professional development assistance
• Referral program
• Vision insurance

Security Clearance

Confidential (Required)

Application Screening Question

Do you have at least 3 years of hands-on experience with AWS Databricks, including PySpark, Spark SQL, and Delta Lake?