Lead Data Engineer: Databricks, Spark & ETL (Remote)

VettedBench; United States; Remote; USD 5,300 - 19,000; Full time

Job summary

VettedBench is seeking an experienced Lead Data Engineer to design, develop, and maintain scalable data pipelines using Databricks. You will work with PySpark, Python, and Apache Spark to process large data sets and build robust ETL/ELT workflows. Delta Lake expertise and cloud familiarity are required.

Remote work options are available as part of a growing tech team. The role emphasizes data quality, performance optimization, and collaboration with engineers and analysts to deliver data-driven

Qualifications

• 8-10 years of data engineering experience.
• Strong hands-on Databricks experience.
• Proficiency in PySpark and Python.
• Solid Apache Spark knowledge.
• Experience with Delta Lake and data lake architectures.
• Strong SQL skills.
• Experience building ETL/ELT pipelines.
• Cloud exposure: Azure, AWS or GCP.
• Git and CI/CD experience is a plus.

Responsibilities

• Design, develop, and maintain scalable data pipelines with Databricks.
• Build data processing solutions using PySpark and Python.
• Leverage Apache Spark for large-scale processing.
• Create and optimize ETL/ELT pipelines.
• Work with Delta Lake for storage and processing.
• Develop workflows using Databricks Workflows.
• Transform, cleanse, and validate data.
• Optimize Spark jobs and improve pipeline performance.
• Collaborate with data engineers, analysts, and tech teams.
• Troubleshoot pipelines and ensure data quality.

Skills

Data engineering

Databricks

PySpark

Python

SQL

ETL/ELT pipelines

Cloud platforms

Git & CI/CD

Spark

Tools

Azure Databricks

Azure Data Factory

ADLS

DBT

Snowflake

Job description

VettedBench is seeking an experienced Lead Data Engineer to design, develop, and maintain scalable data pipelines using Databricks. You will work with PySpark, Python, and Apache Spark to process large data sets and build robust ETL/ELT workflows. Delta Lake expertise and cloud familiarity are required.

Remote work options are available as part of a growing tech team. The role emphasizes data quality, performance optimization, and collaboration with engineers and analysts to deliver data-driven