Data Engineer - Mid

Guidehouse; McLean (VA); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Medical, Rx, Dental & Vision Insurance

401(k) Retirement Plan

Parental Leave

Tuition Reimbursement and Learning

Job summary

Guidehouse is seeking a senior data engineer in McLean, VA to design, build, and optimize scalable data pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services. You will architect batch and streaming processes, ensure data quality, lineage, and security across enterprise datasets, and collaborate with governance and ARB teams.

You will lead end-to-end delivery, align solutions with ARB/governance, and implement observability and error handling to achieve high

Qualifications

• Bachelor’s degree required.
• 8+ years of data engineering in cloud environments.
• 4+ years with Databricks designing scalable pipelines.
• Proficient in Python, PySpark, and SQL for large-scale processing.
• Experience with Delta Lake, medallion architecture, and data ingestion frameworks.
• AWS data platforms and services integration in enterprise ecosystems.
• Production-ready data solutions with metadata, lineage, and quality controls.

Responsibilities

• Design, build, and optimize scalable data ingestion, transformation, and analytics pipelines using Databricks and AWS.
• Engineer standardized pipelines for batch and near real-time processing, integrating legacy sources with cloud services.
• Support end-to-end delivery lifecycle including discovery, profiling, and technical design with governance alignment.
• Implement and maintain governed pipelines with metadata, lineage, and data quality controls.

Skills

Python

PySpark

SQL

Databricks

Delta Lake

Delta Live Tables

Auto Loader

Observability

Metadata management

Education

Bachelor’s degree

Tools

Databricks Unity Catalog

Delta Sharing

Genie

Clean Rooms

Kafka

Kinesis

EventBridge

Terraform

GitHub Actions

Job description

• Design, build, and optimize scalable, production-grade data ingestion, transformation, and analytics-ready pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services, enabling trusted, timely data access across enterprise use cases.
• Engineer standardized, repeatable data pipelines supporting batch and near real-time processing, integrating legacy data sources and modern cloud-native services to advance enterprise data availability and eliminate data access gaps.
• Execute full delivery lifecycle by supporting intake, discovery, source profiling, and technical design, while maintaining requirements traceability and aligning solutions to Architecture Review Board (ARB) and governance expectations.
• Implement and maintain governed data pipelines with embedded metadata, lineage, and data quality controls, ensuring pipelines meet defined technical, security, and documentation requirements before production deployment.
• Develop and operationalize data engineering frameworks that incorporate observability, monitoring, alerting, and resilient error handling to maintain platform stability and support ≥99.9% availability targets.
• Partner with platform engineering and cloud operations teams to integrate pipelines with AWS services (S3, Glue, Kafka/Kinesis, APIs), enabling secure, scalable data movement and cross-platform interoperability.
• Enable governed analytics and self-service data consumption through curated datasets, semantic layers, and SQL warehouse integration, supporting enterprise reporting, dashboards, and advanced analytics use cases.
• Apply security and compliance controls aligned to IRS cybersecurity policies, including RBAC/ABAC, data masking, encryption, and audit logging to protect sensitive data and maintain regulatory compliance.
  
  

Job Family

Data Science & Analysis

Travel Required

Up to 10%

Clearance Required

Ability to Obtain Public Trust

What You Will Do

• Design, build, and optimize scalable, production-grade data ingestion, transformation, and analytics-ready pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services, enabling trusted, timely data access across enterprise use cases.
• Engineer standardized, repeatable data pipelines supporting batch and near real-time processing, integrating legacy data sources and modern cloud-native services to advance enterprise data availability and eliminate data access gaps.
• Execute full delivery lifecycle by supporting intake, discovery, source profiling, and technical design, while maintaining requirements traceability and aligning solutions to Architecture Review Board (ARB) and governance expectations.
• Implement and maintain governed data pipelines with embedded metadata, lineage, and data quality controls, ensuring pipelines meet defined technical, security, and documentation requirements before production deployment.
• Develop and operationalize data engineering frameworks that incorporate observability, monitoring, alerting, and resilient error handling to maintain platform stability and support ≥99.9% availability targets.
• Partner with platform engineering and cloud operations teams to integrate pipelines with AWS services (S3, Glue, Kafka/Kinesis, APIs), enabling secure, scalable data movement and cross-platform interoperability.
• Enable governed analytics and self-service data consumption through curated datasets, semantic layers, and SQL warehouse integration, supporting enterprise reporting, dashboards, and advanced analytics use cases.
• Apply security and compliance controls aligned to IRS cybersecurity policies, including RBAC/ABAC, data masking, encryption, and audit logging to protect sensitive data and maintain regulatory compliance.
  
  

What You Will Need

• Bachelor’s degree is required
• EIGHT (8) or more years of experience in data engineering within cloud-based environments,
• FOUR (4) or more years of hands-on Databricks experience designing scalable data pipelines.
• Experience in Python, PySpark, and SQL for large-scale data processing and pipeline development in lakehouse architectures.
• Experience building and optimizing data pipelines leveraging Delta Lake, medallion architecture, and modern data ingestion frameworks (batch and streaming).
• Experience with AWS data platforms and services (e.g., S3, IAM, VPC, Glue, streaming frameworks) and integration with enterprise data ecosystems.
• Experience delivering production-ready data solutions incorporating metadata management, lineage, data quality, and observability frameworks.
  
  

What Would Be Nice To Have

• Familiarity with DevSecOps and CI/CD pipeline implementation, including automation, testing, and deployment within cloud data environments.
• Knowledge of data governance, security, and compliance requirements within regulated environments, including FISMA and FedRAMP High
• Experience supporting federal data platforms or large-scale enterprise data modernization efforts, particularly within IRS, Treasury, or similar regulated environments.
• Hands-on experience with Databricks Unity Catalog, Delta Sharing, Genie, and Clean Rooms for governed data access and collaboration.
• Experience implementing streaming and near real-time data pipelines using Kafka, Kinesis, EventBridge, or similar technologies.
• Familiarity with Informatica EDC/Axon or enterprise metadata/catalog tooling.
• Experience with performance optimization techniques (Photon, Z-ordering, liquid clustering) to improve large-scale data workloads.
• Exposure to CI/CD automation using Terraform, GitHub Actions, and Databricks Asset Bundles for infrastructure and pipeline deployment.
• Advanced cloud or Databricks certifications (AWS, Databricks) in good standing.
  
  

What We Offer

Guidehouse offers a comprehensive, total rewards package that includes competitive compensation and a flexible benefits package that reflects our commitment to creating a diverse and supportive workplace.

Benefits Include

• Medical, Rx, Dental & Vision Insurance
• Personal and Family Sick Time & Company Paid Holidays
• Parental Leave
• 401(k) Retirement Plan
• Group Term Life and Travel Assistance
• Voluntary Life and AD&D Insurance
• Health Savings Account, Health Care & Dependent Care Flexible Spending Accounts
• Transit and Parking Commuter Benefits
• Short-Term & Long-Term Disability
• Tuition Reimbursement, Personal Development, Certifications & Learning Opportunities
• Employee Referral Program
• Corporate Sponsored Events & Community Outreach
• Care.com annual membership
• Employee Assistance Program
• Supplemental Benefits via Corestream (Critical Care, Hospital Indemnity, Accident Insurance, Legal Assistance and ID theft protection, etc.)
• Position may be eligible for a discretionary variable incentive bonus
  
  

About Guidehouse

Guidehouse is an Equal Opportunity Employer–Protected Veterans, Individuals with Disabilities or any other basis protected by law, ordinance, or regulation.

Guidehouse will consider for employment qualified applicants with criminal histories in a manner consistent with the requirements of applicable law or ordinance including the Fair Chance Ordinance of Los Angeles and San Francisco.

If you have visited our website for information about employment opportunities, or to apply for a position, and you require an accommodation, please contact Guidehouse Recruiting at 1-571-633-1711 or via email at [e-mail]. All information you provide will be kept confidential and will be used only to the extent required to provide needed reasonable accommodation.

All communication regarding recruitment for a Guidehouse position will be sent from Guidehouse email domains including @guidehouse.com or [e-mail]. Correspondence received by an applicant from any other domain should be considered unauthorized and will not be honored by Guidehouse. Note that Guidehouse will never charge a fee or require a money transfer at any stage of the recruitment process and does not collect fees from educational institutions for participation in a recruitment event. Never provide your banking information to a third party purporting to need that information to proceed in the hiring process.

If any person or organization demands money related to a job opportunity with Guidehouse, please report the matter to Guidehouse’s Ethics Hotline. If you want to check the validity of correspondence you have received, please contact [e-mail]. Guidehouse is not responsible for losses incurred (monetary or otherwise) from an applicant’s dealings with unauthorized third parties.

Guidehouse does not accept unsolicited resumes through or from search firms or staffing agencies. All unsolicited resumes will be considered the property of Guidehouse and Guidehouse will not be obligated to pay a placement fee.