Senior Data Engineer | Databricks & Spark Architect

Red Cedar Consultancy; Washington (District of Columbia); In loco; USD 130.000 - 170.000; Tempo pieno

Descrizione del lavoro

Red Cedar Consultancy seeks a Senior Data Engineer to design and optimize Spark-based data pipelines on AWS Databricks. You will build scalable ETL/ELT processes, implement Medallion Architecture, and ensure secure, compliant data workflows for federal projects.

You will manage Databricks environments, integrate diverse data sources, and contribute to DevSecOps practices with Jenkins and GitLab CI. This role requires 3+ years in Databricks and a relevant security clearance, in a DC-area setting.

Competenze

• Databricks development experience in production (min 3 years).
• Proficiency in PySpark, Spark SQL, and Delta Lake.
• Strong Python and SQL; REST API data ingestion.
• Hands-on AWS: S3, IAM, Lambda, Glue, Aurora.
• Terraform IaC and CI/CD (Jenkins, GitLab CI).
• Experience in Agile/Scrum delivery environments.
• Bachelor's degree in CS/IT/Data Science or related field.
• Active or previous High-Risk Public Trust clearance (Tier 4).

Mansioni

• Develop and optimize scalable Spark apps in AWS Databricks using Python, PySpark, SQL.
• Design and maintain Medallion Architecture ETL/ELT pipelines across Bronze-Silver-Gold layers.
• Manage Databricks workspaces, clusters, jobs, access controls, Unity Catalog, notebooks.
• Manage AWS services (S3, Lambda, IAM, Glue; Terraform provisioning).
• Optimize queries and Spark jobs; debug large-scale data solutions.
• Ingest structured, semi-structured, and unstructured data via APIs and databases.
• Integrate DevSecOps CI/CD with Jenkins, GitLab CI for FISMA/FedRAMP compliance.
• Implement data quality checks, lineage, anomaly detection, validation rules.
• Collaborate with Agile teams; provide technical leadership and on-call production support.

Conoscenze

Databricks development

Python

SQL

Agile/Scrum

REST API ingestion

Formazione

Bachelor's degree (CS/IT/Data Science)

Strumenti

AWS S3

IAM

Lambda

Glue

Aurora

Terraform

Jenkins

GitLab CI

Descrizione del lavoro

Red Cedar Consultancy seeks a Senior Data Engineer to design and optimize Spark-based data pipelines on AWS Databricks. You will build scalable ETL/ELT processes, implement Medallion Architecture, and ensure secure, compliant data workflows for federal projects.

You will manage Databricks environments, integrate diverse data sources, and contribute to DevSecOps practices with Jenkins and GitLab CI. This role requires 3+ years in Databricks and a relevant security clearance, in a DC-area setting.