Lakehouse Data Engineering Lead

Randstad Digital Americas; San Diego (CA); On-site; USD 130,000 - 150,000; Full time

Job summary

Randstad Digital Americas seeks a hands-on Data Engineering Technical Lead to modernize our enterprise data platform into a cloud-native Lakehouse, powering analytics, AI, and BI across the organization.

As a technical lead, you will design scalable data pipelines, define data design patterns, enforce engineering standards, and leverage AI-assisted tools to accelerate modernization, improve productivity, and reduce technical debt.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, or related field.
• 8+ years of experience in data engineering or related technical fields, with at least 3+ years in a lead or senior role.
• Proven experience designing and implementing data design patterns (e.g., CDC, SCD, Medallion, Data Vault, streaming, and batch patterns).
• Deep expertise with Databricks, Apache Spark, dbt, Fivetran, Census, Airflow, and Kafka. Solid experience across Azure and/or GCP (e.g., Synapse, Data Factory, BigQuery, Pub/Sub).
• Hands-on experience modernizing legacy ETL (SSIS/SSRS) workloads into cloud-native pipelines.
• Demonstrated ability to build POCs and POVs that validate new tools, frameworks, or architectures.
• Working knowledge of AI-assisted engineering tools for development, observability, or optimization.
• Proficiency in SQL and at least one programming language (Python, Scala, or Java).
• Strong problem-solving, architectural thinking, and collaboration skills. Excellent communicator with the ability to translate technical topics to business stakeholders.

Responsibilities

• Lead the transformation of SSIS/SSRS workloads into modular, high-performance pipelines using Databricks, dbt, Fivetran, and Airflow.
• Define and implement standardized frameworks for ingestion, transformation, curation, and consumption layers across the Lakehouse.
• Experiment with new technologies (e.g., Delta Live Tables, Iceberg, streaming ingestion, AI-driven observability) to validate architecture choices and influence the enterprise roadmap.
• Use AI-enabled tools like Databricks Assistant, Cursor AI, GitHub Copilot, and dbt Mesh AI tests for code generation, automated testing, documentation, and pipeline optimization.
• Integrate predictive models to detect pipeline anomalies, data drift, and optimize compute and scheduling.
• Drive CI/CD, version control, peer reviews, and observability practices across the data platform.
• Partner with data architects, platform engineers, analysts, and business product owners to translate business needs into technical solutions.
• Provide technical guidance, foster continuous learning, and help the team adopt modern data engineering best practices.
• Continuously tune Spark workloads, storage tiers, and orchestration logic across Azure and GCP environments.

Skills

Airflow

Kafka

Spark

Apache Spark

AI

AI-enabled

automated testing

BigQuery

cloud-native

code generation

CI/CD

Cursor

analytics

data design

data platform

scalable data pipelines

streaming

data systems

Data Vault

Databricks

ETL pipelines

ETL

GitHub Copilot

Data Engineering

Java

Azure

Python

Proficiency in SQL

SSIS

SSRS

engineering tools

version control

technical debt

Excellent communicator

Strong problem-solving

high-performance

architecture

business intelligence

business needs

Census

continuous learning

collaboration skills

Pub/Sub

scheduling

documentation

Education

Bachelor's degree in Computer Science, Data Engineering, or related field

Tools

Databricks

dbt

Fivetran

Census

Airflow

Kafka

Spark

Databricks

dbt

Census

Airflow

Kafka

Job description

Randstad Digital Americas seeks a hands-on Data Engineering Technical Lead to modernize our enterprise data platform into a cloud-native Lakehouse, powering analytics, AI, and BI across the organization.

As a technical lead, you will design scalable data pipelines, define data design patterns, enforce engineering standards, and leverage AI-assisted tools to accelerate modernization, improve productivity, and reduce technical debt.