Lead Data Engineer

Recru; Houston (TX); On-site; USD 130,000 - 150,000; Full time

Job summary

A leading IT services company is seeking a Lead Data Engineer to architect and manage robust data infrastructure. The role demands expertise in scalable data pipelines, cloud environments, and Medallion Architecture. Candidates should have considerable experience in data engineering and knowledge of tools like Databricks and Apache Spark. This is a visible leadership role, focusing on data governance and collaboration with data scientists for complex analytics. Strong communication skills and executive presence are essential.

Qualifications

• 10+ years of experience in data engineering with a proven track record.
• Experience with Medallion Architecture in Databricks.
• Strong executive presence and communication skills.

Responsibilities

• Design and implement scalable data pipelines using Apache Spark and Hadoop.
• Lead the Data Management Community of Practice.
• Ensure compliance with data governance policies.

Skills

Data pipeline design

Medallion Architecture

Cloud services (AWS, Azure)

Apache Spark

SQL

Python

Data governance

Education

Bachelor's Degree in Computer Science or related field

Tools

Databricks

Apache Airflow

Fivetran

Job description

Lead Data Engineer – Recru

As a Lead Data Engineer within the Trailblazer initiative, you will play a crucial role in architecting, implementing, and managing robust, scalable data infrastructure. This position demands a blend of systems engineering, data integration, and data analytics skills to enhance our data capabilities, supporting advanced analytics, machine learning projects, and real‑time data processing needs. The ideal candidate brings deep expertise in Lakehouse design principles, including layered Medallion Architecture patterns (Bronze, Silver, Gold), to drive scalable and governed data solutions. This is also a highly visible leadership role that will represent the data engineering function and lead the Data Management Community of Practice across our organization.

As a Lead Data Engineer you will:

• Design and implement scalable and reliable data pipelines to ingest, process, and store diverse data at scale, using technologies such as Apache Spark, Hadoop, and Kafka.
• Work within cloud environments like AWS or Azure to leverage services including but not limited to EC2, RDS, S3, Lambda, and Azure Data Lake for efficient data handling and processing.
• Architect and operationalize data pipelines following Medallion Architecture best practices within a Lakehouse framework—ensuring data quality, lineage, and usability across Bronze, Silver, and Gold layers.
• Develop and optimize data models and storage solutions (Databricks, Data Lakehouses) to support operational and analytical applications, ensuring data quality and accessibility.
• Utilize ETL tools and frameworks (e.g., Apache Airflow, Fivetran) to automate data workflows, ensuring efficient data integration and timely availability of data for analytics.
• Lead the Data Management Community of Practice, serving as the primary facilitator, coordinator, and spokesperson. Drive knowledge sharing, establish best practices, and represent the data engineering discipline across both technical and business audiences.
• Collaborate closely with data scientists, providing the data infrastructure and tools needed for complex analytical models, leveraging Python or R for data processing scripts.
• Ensure compliance with data governance and security policies, implementing best practices in data encryption, masking, and access controls within a cloud environment.
• Monitor and troubleshoot data pipelines and databases for performance issues, applying tuning techniques to optimize data access and throughput.
• Stay abreast of emerging technologies and methodologies in data engineering, advocating for and implementing improvements to the data ecosystem.

Requirements:

• Bachelor’s Degree in Computer Science, MIS, or other business discipline and 10+ years of experience in data engineering, with a proven track record in designing and operating large‑scale data pipelines and architectures or Master’s Degree in computer science, MIS, or other business discipline and 5+ years of experience in data engineering, with a proven track record in designing and operating large‑scale data pipelines and architectures
• Demonstrated experience designing and implementing Medallion Architecture in a Databricks Lakehouse environment, including layer transitions, data quality enforcement, and optimization strategies.
• Comprehensive knowledge of platforms and services like Databricks, Dataiku, and AWS native data offerings
• Solid experience with big data technologies (Apache Spark, Hadoop, Kafka) and cloud services (AWS, Azure) related to data processing and storage
• Strong experience in AWS cloud services, with hands‑on experience in integrating cloud storage and compute services with Databricks
• Proficient in SQL and programming languages relevant to data engineering (Python, Java, Scala)
• Hands on RDBMS experience (data modeling, analysis, programming, stored procedures)
• Familiarity with machine learning model deployment and management practices is a plus
• Strong executive presence and communication skills, with a proven ability to lead communities of practice, deliver presentations to senior leadership, and build alignment across technical and business stakeholders.

Nice to Have:

• AWS Certified Solution Architect
• Databricks Certified Associate Developer for Apache Spark
• DAMA CDMP

Seniority level

Mid‑Senior level

Employment type

Full‑time

Job function

Information Technology

Industries

IT Services and IT Consulting and Executive Offices