Lead Data Engineer – Databricks & ADF ETL

Valiant Technologies LLC

Sunnyvale (CA)

On-site

USD 89,000 - 126,000

Full time

3 hours ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Valiant Technologies LLC in Sunnyvale, CA, is seeking a Lead Data Engineer to design and optimize ETL/ELT pipelines using Databricks and Azure Data Factory (ADF). You will develop scalable data workflows, write efficient SQL, and implement PySpark solutions to transform data for analytics and BI needs.

Ideal candidates have hands-on experience with Databricks, ADF, SQL, Python, and Azure data lake concepts, and can collaborate with cross-functional teams to ensure data quality and reliability in

Qualifications

Experience with Databricks and Azure Data Factory (ADF).
Strong understanding of ETL/ELT concepts.
Good knowledge of SQL and data transformations.
Basic experience with Python or PySpark.
Understanding of Azure cloud services and data lake concepts.

Responsibilities

Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
Work with Databricks for data processing and transformation.
Write and optimize SQL queries for data extraction and transformation.
Work with PySpark or Python for large-scale data processing.
Integrate data from multiple sources into data warehouses or data lakes.
Monitor, troubleshoot, and optimize data pipelines.
Collaborate with teams to ensure data quality and reliability.

Skills

Databricks

Azure Data Factory (ADF)

ETL/ELT concepts

SQL

Python/PySpark

Azure data lake concepts

Job description

Ideal candidates have hands-on experience with Databricks, ADF, SQL, Python, and Azure data lake concepts, and can collaborate with cross-functional teams to ensure data quality and reliability in

Get your free, confidential resume review.

or drag and drop your file here.