Senior Data Engineer

Red Cedar Consultancy, LLC; Maryland; On-site; USD 150,000 - 190,000; Full time

Job summary

Red Cedar Consultancy, LLC seeks a Senior Data Engineer / Databricks Specialist to design and optimize high-performance Spark applications and enterprise data infrastructure in AWS Databricks for a federal project. You will build scalable ETL/ELT pipelines using Medallion Architecture and ensure data integrity and lineage across Bronze, Silver, and Gold layers.

The role requires hands-on PySpark, Python, SQL, and Delta Lake, plus experience with Terraform, CI/CD, and secure DevSecOps practices

Qualifications

• 3+ years of hands-on Databricks development experience
• Production experience with PySpark, Spark SQL, Delta Lake
• Strong Python and SQL skills; REST API based data ingestion
• Experience with AWS S3, IAM, Lambda, Glue, Aurora
• Terraform IaC and CI/CD tools (Jenkins, GitLab CI)
• Agile/Scrum delivery environment experience
• Clearance: High-Risk Public Trust (Tier 4) or equivalent

Responsibilities

• Develop and optimize scalable Spark apps in AWS Databricks using Python, PySpark, and SQL
• Design, build, and maintain Medallion Architecture ETL/ELT pipelines
• Administer Databricks workspaces, clusters, jobs, and Unity Catalog
• Manage AWS services (S3, Lambda, IAM, Glue, Aurora) and Terraform provisioning
• Optimize queries and Spark jobs; troubleshoot large-scale data processing
• Ingest structured, semi-structured, and unstructured data from APIs/databases
• Integrate security/compliance controls into DevSecOps CI/CD for FedRAMP/ATO
• Implement data quality checks, lineage, and anomaly detection
• Collaborate with Agile/Scrum teams; provide technical leadership and on-call support

Skills

Databricks development

PySpark

Python

SQL

REST API data ingestion

AWS basics

Agile / Scrum

Education

Bachelor's or higher in Computer Science/IT/Data Science/Engineering

Tools

Terraform

Jenkins

GitLab CI

Unity Catalog

Delta Live Tables

MLflow

Model Registry

AWS S3

Job description

The Senior Data Engineer / Databricks Specialist will support a federal project by developing, designing , and optimizing high-performance Spark applications and enterprise data infrastructure in AWS Databricks. The role focuses on scalable ETL/ELT pipelines using Python, PySpark, SQL, and Delta Lake; Medallion Architecture; and secure, reliable data solutions for mission-critical environments.

Responsibilities

• Develop and optimize scalable Spark applications in AWS Databricks using Python, PySpark, and SQL.
• Design, build, and maintain ETL/ELT pipelines using Medallion Architecture (Bronze, Silver, and Gold layers), ensuring data integrity, completeness, and lineage.
• Administer Databricks workspaces, clusters, scheduled jobs, access controls, Unity Catalog, and notebooks.
• Manage AWS services including S3, Lambda, IAM, Aurora, and Glue; provision infrastructure with Terraform.
• Optimize queries and Spark jobs; troubleshoot and debug large-scale data processing solutions.
• Integrate structured, semi-structured, and unstructured data from APIs and databases.
• Integrate security and compliance controls into DevSecOps CI/CD pipelines using Jenkins, GitLab CI, or equivalent tools, supporting FISMA/FedRAMP requirements, ATO reassessments, and POA&M resolution.
• Implement data quality checks, anomaly detection, validation rules, and automated lineage tracking.
• Collaborate with cross-functional Agile/Scrum teams in multi-contractor federal environments; provide technical leadership and production on-call support.

Required Qualifications

• At least 3 years of dedicated, hands‑on Databricks development experience.
• Production experience with PySpark, Apache Spark, Spark SQL, and Delta Lake.
• Strong proficiency in Python and SQL and working knowledge of REST API‑based data ingestion.
• Strong hands‑on experience with AWS S3, IAM, Lambda, Glue, and Aurora.
• Production experience with Terraform infrastructure as code and CI/CD tools such as Jenkins, GitLab CI, or equivalent.
• Experience working in Agile/Scrum delivery environments.
• College’s degree in Computer Science, Information Technology, Data Science, or a related engineering discipline.
• Active or previous High‑Risk Public Trust (Tier 4) clearance.

Preferred Qualifications

• Hands‑on experience with Unity Catalog, Delta Live Tables (DLT), and MLflow or Model Registry for AI/ML pipelines.
• Experience supporting federal government contracts and FISMA/FedRAMP environments.

Preferred Certifications

• AWS Certified Solutions Architect – Associate or Professional
• AWS Certified Data Analytics – Specialty