Senior Data Engineer - Databricks, Delta Lake & AWS Pipelines

Guidehouse; McLean (VA); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Medical, Rx, Dental & Vision Insurance

401(k) Retirement Plan

Parental Leave

Tuition Reimbursement and Learning

Job summary

Guidehouse is seeking a senior data engineer in McLean, VA to design, build, and optimize scalable data pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services. You will architect batch and streaming processes, ensure data quality, lineage, and security across enterprise datasets, and collaborate with governance and ARB teams.

You will lead end-to-end delivery, align solutions with ARB/governance, and implement observability and error handling to achieve high

Qualifications

• Bachelor’s degree required.
• 8+ years of data engineering in cloud environments.
• 4+ years with Databricks designing scalable pipelines.
• Proficient in Python, PySpark, and SQL for large-scale processing.
• Experience with Delta Lake, medallion architecture, and data ingestion frameworks.
• AWS data platforms and services integration in enterprise ecosystems.
• Production-ready data solutions with metadata, lineage, and quality controls.

Responsibilities

• Design, build, and optimize scalable data ingestion, transformation, and analytics pipelines using Databricks and AWS.
• Engineer standardized pipelines for batch and near real-time processing, integrating legacy sources with cloud services.
• Support end-to-end delivery lifecycle including discovery, profiling, and technical design with governance alignment.
• Implement and maintain governed pipelines with metadata, lineage, and data quality controls.

Skills

Python

PySpark

SQL

Databricks

Delta Lake

Delta Live Tables

Auto Loader

Observability

Metadata management

Education

Bachelor’s degree

Tools

Databricks Unity Catalog

Delta Sharing

Genie

Clean Rooms

Kafka

Kinesis

EventBridge

Terraform

GitHub Actions

Job description

Guidehouse is seeking a senior data engineer in McLean, VA to design, build, and optimize scalable data pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services. You will architect batch and streaming processes, ensure data quality, lineage, and security across enterprise datasets, and collaborate with governance and ARB teams.

You will lead end-to-end delivery, align solutions with ARB/governance, and implement observability and error handling to achieve high