Data Engineering Lead

Dunhill Professional Search & Government Solutions; Virginia (MN); Remote; USD 175,000 - 180,000; Full time

Job summary

A technology consulting firm is seeking a Data Engineering Lead to oversee the design and optimization of scalable data pipelines. This fully remote role requires expertise in data engineering principles and leading technical teams, with responsibilities including architecture design, data governance, and stakeholder collaboration. Ideal candidates will have at least 12 years of experience and advanced proficiency in distributed data processing technologies.

Qualifications

• 12+ years of experience in a technical field.
• At least 2 years in a technical leadership or lead engineer role.
• Industry leading certification for technical expertise preferred.

Responsibilities

• Lead and mentor a team of data engineers.
• Drive the design and implementation of scalable data architectures.
• Design, develop, and maintain complex ETL/ELT processes.
• Establish and enforce data governance and quality standards.

Skills

Expert-level proficiency in SQL

Experience with relational and NoSQL databases

Extensive experience with distributed data processing frameworks

Strong understanding of data warehousing concepts

Proficiency in Python

Education

Bachelor’s Degree in a related field

Master’s Degree preferred

Tools

Apache Spark

AWS

Azure

GCP

Job description

Job title

Data Engineering Lead

Work arrangement

Telework. This role is 100% remote.

Base pay

$175,000.00/yr - $180,000.00/yr

Job Description

Seeking an experienced and highly skilled Data Engineering Technical Lead to spearhead the design, development, and optimization of robust and scalable data pipelines and data infrastructure. This role requires a deep understanding of data engineering principles, advanced proficiency in distributed data processing technologies, and a proven track record of leading technical teams in an agile environment. The Technical Lead will play a critical role in shaping our data architecture, ensuring data quality and governance, and enabling data-driven initiatives across the organization, particularly within the healthcare and supply chain logistics domains. This role is 100% remote.

Key Responsibilities

• Lead, mentor, and guide a team of data engineers in best practices for data pipeline development, data modeling, and infrastructure management. Provide technical oversight and ensure adherence to architectural standards.
• Drive the design and implementation of highly scalable, reliable, and efficient data architectures, including data lakes, data warehouses, and streaming platforms. Define technical roadmaps and strategies for data engineering initiatives.
• Design, develop, and maintain complex ETL/ELT processes and data pipelines using modern data engineering tools and frameworks (e.g., Apache Spark, Kafka, Airflow). Ensure data ingestion, transformation, and loading processes are optimized for performance and reliability.
• Establish and enforce data governance policies, data quality standards, and data security protocols. Implement robust monitoring and alerting for data pipelines and data quality issues.
• Leverage expertise in cloud-based data platforms (e.g., AWS, Azure, GCP) to design and implement cloud-native data solutions, optimizing for cost, performance, and scalability.
• Collaborate closely with product owners, data scientists, data analysts, and other engineering teams to translate business requirements into technical specifications and deliver data solutions that align with strategic objectives. Act as a primary interface with customers, business stakeholders, and cross-functional teams to gather requirements, provide updates, and ensure alignment on data initiatives.
• Identify and resolve complex data-related performance bottlenecks and architectural challenges. Implement strategies for query optimization and data storage efficiency.
• Actively participate in agile ceremonies, including sprint planning, backlog refinement, daily stand-ups, and sprint reviews, contributing to a culture of continuous delivery and improvement. Lead and own the full Software Development Life Cycle (SDLC) for data solutions, from conceptualization and design through deployment, testing, and operational support.
• Create and maintain comprehensive technical documentation, including data flow diagrams, architectural designs, data dictionaries, and operational runbooks.

Minimum Qualifications

• Bachelor’s Degree in a related field; Master’s Degree preferred
• Industry leading certification for technical expertise preferred
• 12+ years of experience in a technical field with at least 2 years in a technical leadership or lead engineer role.

Other Job Specific Skills

• Expert-level proficiency in SQL and experience with relational and NoSQL databases.
• Extensive experience with distributed data processing frameworks (e.g., Apache Spark, Hadoop) and stream processing technologies (e.g., Apache Kafka, Flink).
• Strong understanding of data warehousing concepts, dimensional modeling, and data lake architectures.
• Proven experience with cloud-based data platforms (e.g., AWS Redshift, S3, Glue, EMR; Azure Data Lake, Synapse, Databricks; GCP BigQuery, Dataflow).
• Proficiency in at least one programming language commonly used in data engineering (e.g., Python, Scala, Java).
• Demonstrated experience with CI/CD practices for data pipelines and infrastructure as code (IaC).
• Solid understanding of data governance, data security, and compliance best practices (e.g., HIPAA, GDPR).

Seniority level

• Mid-Senior level

Employment type

• Full-time

Job function

• Information Technology and Project Management

Industries

• IT Services and IT Consulting