Senior Data Engineer

Red Cedar Consultancy; Washington (District of Columbia); On-site; USD 130,000 - 170,000; Full time

Job summary

Red Cedar Consultancy seeks a Senior Data Engineer to design and optimize Spark-based data pipelines on AWS Databricks. You will build scalable ETL/ELT processes, implement Medallion Architecture, and ensure secure, compliant data workflows for federal projects.

You will manage Databricks environments, integrate diverse data sources, and contribute to DevSecOps practices with Jenkins and GitLab CI. This role requires 3+ years in Databricks and a relevant security clearance, in a DC-area setting.

Qualifications

• Databricks development experience in production (min 3 years).
• Proficiency in PySpark, Spark SQL, and Delta Lake.
• Strong Python and SQL; REST API data ingestion.
• Hands-on AWS: S3, IAM, Lambda, Glue, Aurora.
• Terraform IaC and CI/CD (Jenkins, GitLab CI).
• Experience in Agile/Scrum delivery environments.
• Bachelor's degree in CS/IT/Data Science or related field.
• Active or previous High-Risk Public Trust clearance (Tier 4).

Responsibilities

• Develop and optimize scalable Spark apps in AWS Databricks using Python, PySpark, SQL.
• Design and maintain Medallion Architecture ETL/ELT pipelines across Bronze-Silver-Gold layers.
• Manage Databricks workspaces, clusters, jobs, access controls, Unity Catalog, notebooks.
• Manage AWS services (S3, Lambda, IAM, Glue; Terraform provisioning).
• Optimize queries and Spark jobs; debug large-scale data solutions.
• Ingest structured, semi-structured, and unstructured data via APIs and databases.
• Integrate DevSecOps CI/CD with Jenkins, GitLab CI for FISMA/FedRAMP compliance.
• Implement data quality checks, lineage, anomaly detection, validation rules.
• Collaborate with Agile teams; provide technical leadership and on-call production support.

Skills

Databricks development

Python

SQL

Agile/Scrum

REST API ingestion

Education

Bachelor's degree (CS/IT/Data Science)

Tools

AWS S3

IAM

Lambda

Glue

Aurora

Terraform

Jenkins

GitLab CI

Job description

Job Title: Senior Data Engineer

Location

Washinton DC

Summary

The Senior Data Engineer / Databricks Specialist will support a federal project by developing, designing, and optimizing high-performance Spark applications and enterprise data infrastructure in AWS Databricks. The role focuses on scalable ETL/ELT pipelines using Python, PySpark, SQL, and Delta Lake; Medallion Architecture; and secure, reliable data solutions for mission-critical environments.

Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
• Design, build, and maintain ETL/ELT pipelines using Medallion Architecture (Bronze, Silver, and Gold layers), ensuring data integrity, completeness, and lineage.
• Administer Databricks workspaces, clusters, scheduled jobs, access controls, Unity Catalog, and notebooks.
• Manage AWS services including S3, Lambda, IAM, Aurora, and Glue; provision infrastructure with Terraform.
• Optimize queries and Spark jobs; troubleshoot and debug large-scale data processing solutions.
• Integrate structured, semi-structured, and unstructured data from APIs and databases.
• Integrate security and compliance controls into DevSecOps CI/CD pipelines using Jenkins, GitLab CI, or equivalent tools, supporting FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
• Implement data quality checks, anomaly detection, validation rules, and automated lineage tracking.
• Collaborate with cross-functional Agile/Scrum teams in multi-contractor federal environments; provide technical leadership and production on-call support.

Required Qualifications

• At least 3 years of dedicated, hands-on Databricks development experience.
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
• Strong proficiency in Python and SQL and working knowledge of REST API-based data ingestion.
• Strong hands-on experience with AWS S3, IAM, Lambda, Glue, and Aurora.
• Production experience with Terraform infrastructure as code and CI/CD tools such as Jenkins, GitLab CI, or equivalent.
• Experience working in Agile/Scrum delivery environments.
• Bachelor's degree in Computer Science, Information Technology, Data Science, or a related engineering discipline.
• Active or previous High-Risk Public Trust (Tier 4) clearance.

Preferred Qualifications

• Hands-on experience with Unity Catalog, Delta Live Tables (DLT), and MLflow or Model Registry for AI/ML pipelines.
• Experience supporting federal government contracts and FISMA/FedRAMP environments.

Preferred Certifications

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Databricks Certified Machine Learning Associate
• AWS Certified Solutions Architect - Associate or Professional
• AWS Certified Data Analytics - Specialty