Data Engineering Technical Lead

Randstad Digital Americas; San Diego (CA); On-site; USD 130,000 - 150,000; Full time

Job summary

Randstad Digital Americas seeks a hands-on Data Engineering Technical Lead to modernize our enterprise data platform into a cloud-native Lakehouse, powering analytics, AI, and BI across the organization.

As a technical lead, you will design scalable data pipelines, define data design patterns, enforce engineering standards, and leverage AI-assisted tools to accelerate modernization, improve productivity, and reduce technical debt.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, or related field.
• 8+ years of experience in data engineering or related technical fields, with at least 3+ years in a lead or senior role.
• Proven experience designing and implementing data design patterns (e.g., CDC, SCD, Medallion, Data Vault, streaming, and batch patterns).
• Deep expertise with Databricks, Apache Spark, dbt, Fivetran, Census, Airflow, and Kafka. Solid experience across Azure and/or GCP (e.g., Synapse, Data Factory, BigQuery, Pub/Sub).
• Hands-on experience modernizing legacy ETL (SSIS/SSRS) workloads into cloud-native pipelines.
• Demonstrated ability to build POCs and POVs that validate new tools, frameworks, or architectures.
• Working knowledge of AI-assisted engineering tools for development, observability, or optimization.
• Proficiency in SQL and at least one programming language (Python, Scala, or Java).
• Strong problem-solving, architectural thinking, and collaboration skills. Excellent communicator with the ability to translate technical topics to business stakeholders.

Responsibilities

• Lead the transformation of SSIS/SSRS workloads into modular, high-performance pipelines using Databricks, dbt, Fivetran, and Airflow.
• Define and implement standardized frameworks for ingestion, transformation, curation, and consumption layers across the Lakehouse.
• Experiment with new technologies (e.g., Delta Live Tables, Iceberg, streaming ingestion, AI-driven observability) to validate architecture choices and influence the enterprise roadmap.
• Use AI-enabled tools like Databricks Assistant, Cursor AI, GitHub Copilot, and dbt Mesh AI tests for code generation, automated testing, documentation, and pipeline optimization.
• Integrate predictive models to detect pipeline anomalies, data drift, and optimize compute and scheduling.
• Drive CI/CD, version control, peer reviews, and observability practices across the data platform.
• Partner with data architects, platform engineers, analysts, and business product owners to translate business needs into technical solutions.
• Provide technical guidance, foster continuous learning, and help the team adopt modern data engineering best practices.
• Continuously tune Spark workloads, storage tiers, and orchestration logic across Azure and GCP environments.

Skills

Airflow

Kafka

Spark

Apache Spark

AI

AI-enabled

automated testing

BigQuery

cloud-native

code generation

CI/CD

Cursor

analytics

data design

data platform

scalable data pipelines

streaming

data systems

Data Vault

Databricks

ETL pipelines

ETL

GitHub Copilot

Data Engineering

Java

Azure

Python

Proficiency in SQL

SSIS

SSRS

engineering tools

version control

technical debt

Excellent communicator

Strong problem-solving

high-performance

architecture

business intelligence

business needs

Census

continuous learning

collaboration skills

Pub/Sub

scheduling

documentation

Education

Bachelor's degree in Computer Science, Data Engineering, or related field

Tools

Databricks

dbt

Fivetran

Census

Airflow

Kafka

Spark

Databricks

dbt

Census

Airflow

Kafka

Job description

Job Summary

We are seeking a hands-on Data Engineering Technical Lead to help shape, modernize, and scale our enterprise data platform. This role is central to our mission of transforming legacy data systems into a modern, cloud-native Lakehouse environment that powers analytics, AI, and business intelligence across the organization. As a technical lead, you will design and deliver scalable data pipelines, define data design patterns, enforce engineering standards, and leverage AI-assisted tools to accelerate modernization, improve productivity, and reduce technical debt. You will drive proofs of concept (POCs) and points of view (POVs) to evaluate emerging technologies and frameworks, ensuring that the platform remains innovative, cost-efficient, and future-ready.
location: San Diego, California
job type: Permanent
salary: $130,000 - 150,000 per year
work hours: 8am to 5pm
education: Bachelors

Responsibilities

• Lead the transformation of SSIS/SSRS workloads into modular, high-performance pipelines using Databricks, dbt, Fivetran, and Airflow.
• Define and implement standardized frameworks for ingestion, transformation, curation, and consumption layers across the Lakehouse.
• Experiment with new technologies (e.g., Delta Live Tables, Iceberg, streaming ingestion, AI-driven observability) to validate architecture choices and influence the enterprise roadmap.
• Use AI-enabled tools like Databricks Assistant, Cursor AI, GitHub Copilot, and dbt Mesh AI tests for code generation, automated testing, documentation, and pipeline optimization.
• Integrate predictive models to detect pipeline anomalies, data drift, and optimize compute and scheduling.
• Drive CI/CD, version control, peer reviews, and observability practices across the data platform.
• Partner with data architects, platform engineers, analysts, and business product owners to translate business needs into technical solutions.
• Provide technical guidance, foster continuous learning, and help the team adopt modern data engineering best practices.
• Continuously tune Spark workloads, storage tiers, and orchestration logic across Azure and GCP environments.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, or a related technical field.
• 8+ years of experience in data engineering or related technical fields, with at least 3+ years in a lead or senior role.
• Proven experience designing and implementing data design patterns (e.g., CDC, SCD, Medallion, Data Vault, streaming, and batch patterns).
• Deep expertise with Databricks, Apache Spark, dbt, Fivetran, Census, Airflow, and Kafka. Solid experience across Azure and/or GCP (e.g., Synapse, Data Factory, BigQuery, Pub/Sub).
• Hands-on experience modernizing legacy ETL (SSIS/SSRS) workloads into cloud-native pipelines.
• Demonstrated ability to build POCs and POVs that validate new tools, frameworks, or architectures.
• Working knowledge of AI-assisted engineering tools for development, observability, or optimization.
• Proficiency in SQL and at least one programming language (Python, Scala, or Java).
• Strong problem-solving, architectural thinking, and collaboration skills. Excellent communicator with the ability to translate technical topics to business stakeholders.

Skills

• Airflow
• Kafka
• Spark
• Apache Spark
• AI
• AI-enabled
• automated testing
• BigQuery
• cloud-native
• code generation
• CI/CD
• Cursor
• analytics
• data design
• data platform
• scalable data pipelines
• streaming
• data systems
• Data Vault
• Databricks
• ETL pipelines
• ETL
• GitHub Copilot
• Data Engineering
• Java
• Azure
• Python
• Proficiency in SQL
• SSIS
• SSRS
• engineering tools
• version control
• technical debt
• Excellent communicator
• Strong problem-solving
• high-performance
• architecture
• business intelligence
• business needs
• Census
• continuous learning
• collaboration skills
• Pub/Sub
• scheduling
• documentation

Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants.

If you require a reasonable accommodation to make your application or interview experience a great one, please contact [e-mail].

Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).

This posting is open for thirty (30) days.

Qualified applicants in San Francisco with criminal histories will be considered for employment in accordance with the San Francisco Fair Chance Ordinance.

Qualified applicants with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act.

We will consider for employment all qualified Applicants, including those with criminal histories, in a manner consistent with the requirements of applicable state and local laws, including the City of Los Angeles' Fair Chance Initiative for Hiring Ordinance.