Lead Data Engineer

SHR Consulting Group; Virginia, Northern (IL, KY); Hybrid; USD 150,000 - 165,000; Full time

Job summary

SHR Consulting Group, LLC is seeking a senior Lead Data Engineer to head data pipeline engineering for a FEMA enterprise effort. This role will lead design, development, and deployment of scalable batch and streaming pipelines, focusing on metadata-driven ingestion and robust data architectures.

Ideal candidates will have extensive experience with Databricks or lakehouse platforms, CI/CD, and cloud environments regulated for federal data.

Qualifications

• Extensive hands-on experience designing and developing enterprise data pipelines.
• Experience with Databricks or comparable lakehouse platforms.
• Strong knowledge of schema evolution and data-processing patterns.
• Experience with cloud-based data engineering in Federal or regulated environments.
• Strong understanding of CI/CD, automated testing, data-quality validation, and deployment automation.
• Experience with scalable data architectures and high-volume enterprise data environments.

Responsibilities

• Lead technical delivery for FEMA pipeline engineering, operations, performance, and migration.
• Design, construct, test, deploy, and maintain scalable batch and streaming data pipelines.
• Build and maintain metadata-driven ingestion pipelines.
• Implement robust schema evolution and data-processing patterns.
• Diagnose and resolve production pipeline failures and complex data-storage issues.
• Design pipelines consistent with FEMA's multi-zone/Medallion architecture.
• Develop modular, parameterized, and reusable pipeline patterns.
• Implement scalable ingestion and transformation solutions supporting increasing data volume, velocity, and concurrency.
• Implement engineering practices for reliability and cost efficiency, including incremental processing, idempotent and restartable jobs, partitioning, storage layout optimization, and compute right-sizing.
• Develop automated testing, data-quality validation, and deployment automation.
• Diagnose schema drift, missing or late data, checkpoint/watermark failures, orchestration faults, and performance degradation.
• Conduct root-cause analysis of significant production issues.
• Support migration of legacy data, pipelines, reporting workloads, and dependencies.
• Support source-system onboarding and integration using APIs, connectors, batch/file transfers, CDC, and custom integration patterns.
• Provide technical leadership and mentoring to Senior Data Engineers and other engineering staff.

Skills

Databricks

Python

SQL

REST APIs

CDC

CI/CD

Git

Tools

Git/version control

Job description

If you are unable to complete this application due to a disability, contact this employer to ask for an accommodation or an alternative application process.

Bluemont, VA, US

2 days ago Requisition ID: 1507

Salary Range: $150,000.00 To $165,000.00 Annually

SHR Consulting Group, LLC is a small business delivering enterprise IT, cybersecurity, and program management services to the Department of Defense and federal civilian agencies. We support mission-critical infrastructure at the Pentagon and across the National Capital Region, and we invest in our people through competitive compensation, professional certification support, and long-term career growth on stable, multi-year programs.

We are seeking a senior Lead Data Engineeras a Key Personnel for a bid opportunity to lead data pipeline engineering, operations, performance, migration, and integration activities supporting an enterprise FEMA effort.

This position is CONTINGENT UPON CONTRACT AWARD. Candidates selected for consideration must be willing to authorize the submission of their resume and qualifications in support of our proposal response.

Key Responsibilities

• Lead technical delivery for FEMA pipeline engineering, operations, performance, and migration.
• Design, construct, test, deploy, and maintain scalable batch and streaming data pipelines.
• Build and maintain metadata-driven ingestion pipelines.
• Implement robust schema evolution and data-processing patterns.
• Diagnose and resolve production pipeline failures and complex data-storage issues.
• Design pipelines consistent with FEMA's multi-zone/Medallion architecture.
• Develop modular, parameterized, and reusable pipeline patterns.
• Implement scalable ingestion and transformation solutions supporting increasing data volume, velocity, and concurrency.
• Implement engineering practices for reliability and cost efficiency, including:
  • Incremental processing
  • Idempotent and restartable jobs
  • Partitioning
  • Storage/file-layout optimization
  • Compute right-sizing
• Develop automated testing, data-quality validation, and deployment automation.
• Diagnose schema drift, missing or late data, checkpoint/watermark failures, orchestration faults, and performance degradation.
• Conduct root-cause analysis of significant or recurring production issues.
• Support migration of legacy data, pipelines, reporting workloads, and dependencies.
• Support source-system onboarding and integration using APIs, connectors, batch/file transfers, CDC, and custom integration patterns.
• Provide technical leadership and mentoring to Senior Data Engineers and other engineering staff.

Required Qualifications

• Extensive hands-on experience designing and developing enterprise data pipelines.
• Demonstrated experience with Databricks or comparable cloud-based Lakehouse/data engineering platforms.
• Strong experience with metadata-driven data ingestion frameworks.
• Strong knowledge of schema evolution and enterprise data-processing patterns.
• Demonstrated ability to troubleshoot live production pipeline failures.
• Experience debugging complex data-storage and processing structures.
• Experience designing batch and streaming pipelines.
• Experience with cloud-based data engineering in Federal or similarly regulated environments.
• Strong understanding of CI/CD, automated testing, data-quality validation, and deployment automation.
• Experience with scalable data architectures and high-volume enterprise data environments.
• Ability to provide hands-on technical leadership to engineering teams.

Highly Desired Technical Experience

• Databricks
• SQL
• Python
• Medallion/Lakehouse architectures
• Structured Streaming or comparable streaming technologies
• REST APIs and source-system integration
• Change Data Capture (CDC)
• CI/CD and Infrastructure/Deployment Automation
• Data quality and observability tooling
• Git/version control
• FinOps and cloud performance optimization

Preferred Qualifications

• Prior FEMA or DHS experience.
• Experience migrating legacy Federal data environments to modern cloud data platforms.
• Experience supporting data platforms used for AI/ML and Retrieval-Augmented Generation (RAG).
• Experience engineering feature pipelines, curated AI datasets, vector/embedding data stores, or AI data products.
• Experience with Federal security requirements including NIST, FISMA, FedRAMP, and DHS 4300A.

Clearance Requirements

• Must be able to obtain and maintain the Government-required suitability/fitness determination and IT access authorization.
• Competitive salary commensurate with experience.
• Comprehensive medical, dental, and vision coverage.
• 401(k) with company contribution.
• Paid time off and eleven federal holidays.
• Certification reimbursement and training support.
• Life and disability insurance.

SHR Consulting Group, LLC is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.