Senior Data Engineer — Lakehouse Pipelines & Metadata

SHR Consulting Group; United States; On-site; USD 180,000 - 230,000; Full time

Benefits offered by this job

Medical, dental, vision

401(k) with company contribution

Paid time off and federal holidays

Certification reimbursement and培训 支援

Life and disability insurance

Job summary

SHR Consulting Group, LLC is seeking a senior Lead Data Engineer to spearhead data pipeline engineering for a FEMA enterprise effort. The role focuses on batch and streaming pipelines, metadata-driven ingestion, schema evolution, and reliable data processing in a regulated environment.

Lead technical delivery and mentor other engineers, while ensuring scalable, cost-efficient solutions and robust testing and deployment practices. Contingent upon contract award.

Qualifications

• Extensive hands-on experience designing and developing enterprise data pipelines.
• Demonstrated experience with Databricks or comparable cloud-based Lakehouse/data engineering platforms.
• Strong experience with metadata-driven data ingestion frameworks.
• Strong knowledge of schema evolution and enterprise data-processing patterns.
• Demonstrated ability to troubleshoot live production pipeline failures.
• Experience debugging complex data-storage and processing structures.
• Experience designing batch and streaming pipelines.
• Experience with cloud-based data engineering in Federal or similarly regulated environments.
• Strong understanding of CI/CD, automated testing, data-quality validation, and deployment automation.
• Experience with scalable data architectures and high-volume enterprise data environments.
• Ability to provide hands-on technical leadership to engineering teams.

Responsibilities

• Lead technical delivery for FEMA pipeline engineering, operations, performance, and migration.
• Design, construct, test, deploy, and maintain scalable batch and streaming data pipelines.
• Build and maintain metadata-driven ingestion pipelines.
• Implement robust schema evolution and data-processing patterns.
• Diagnose and resolve production pipeline failures and complex data-storage issues.
• Design pipelines consistent with FEMA's multi-zone/Medallion architecture.
• Develop modular, parameterized, and reusable pipeline patterns.
• Implement scalable ingestion and transformation solutions supporting increasing data volume, velocity, and concurrency.
• Implement engineering practices for reliability and cost efficiency, including: Incremental processing; Idempotent and restartable jobs; Partitioning; Storage/file-layout optimization; Compute right-sizing.
• Develop automated testing, data-quality validation, and deployment automation.
• Diagnose schema drift, missing or late data, checkpoint/watermark failures, orchestration faults, and performance degradation.
• Conduct root-cause analysis of significant or recurring production issues.
• Support migration of legacy data, pipelines, reporting workloads, and dependencies.
• Support source-system onboarding and integration using APIs, connectors, batch/file transfers, CDC, and custom integration patterns.
• Provide technical leadership and mentoring to Senior Data Engineers and other engineering staff.

Skills

Enterprise data pipelines

Databricks

Metadata ingestion

Schema evolution

CI/CD

Python

SQL

Streaming pipelines

Cloud data platforms

Leadership

Troubleshooting

Tools

Databricks

REST APIs

Git

Job description

SHR Consulting Group, LLC is seeking a senior Lead Data Engineer to spearhead data pipeline engineering for a FEMA enterprise effort. The role focuses on batch and streaming pipelines, metadata-driven ingestion, schema evolution, and reliable data processing in a regulated environment.

Lead technical delivery and mentor other engineers, while ensuring scalable, cost-efficient solutions and robust testing and deployment practices. Contingent upon contract award.