Lead Data Engineer {Contract}

VettedBench; United States; Remote; USD 5,300 - 19,000; Full time

Job summary

VettedBench is seeking an experienced Lead Data Engineer to design, develop, and maintain scalable data pipelines using Databricks. You will work with PySpark, Python, and Apache Spark to process large data sets and build robust ETL/ELT workflows. Delta Lake expertise and cloud familiarity are required.

Remote work options are available as part of a growing tech team. The role emphasizes data quality, performance optimization, and collaboration with engineers and analysts to deliver data-driven

Qualifications

• 8-10 years of data engineering experience.
• Strong hands-on Databricks experience.
• Proficiency in PySpark and Python.
• Solid Apache Spark knowledge.
• Experience with Delta Lake and data lake architectures.
• Strong SQL skills.
• Experience building ETL/ELT pipelines.
• Cloud exposure: Azure, AWS or GCP.
• Git and CI/CD experience is a plus.

Responsibilities

• Design, develop, and maintain scalable data pipelines with Databricks.
• Build data processing solutions using PySpark and Python.
• Leverage Apache Spark for large-scale processing.
• Create and optimize ETL/ELT pipelines.
• Work with Delta Lake for storage and processing.
• Develop workflows using Databricks Workflows.
• Transform, cleanse, and validate data.
• Optimize Spark jobs and improve pipeline performance.
• Collaborate with data engineers, analysts, and tech teams.
• Troubleshoot pipelines and ensure data quality.

Skills

Data engineering

Databricks

PySpark

Python

SQL

ETL/ELT pipelines

Cloud platforms

Git & CI/CD

Spark

Tools

Azure Databricks

Azure Data Factory

ADLS

DBT

Snowflake

Job description

We are looking for an experienced **Data Engineer (Lead)** with strong expertise in **Databricks** to design, develop, and maintain scalable data pipelines and data processing solutions.

**Key Responsibilities:**

• Design, develop, and maintain scalable data pipelines using **Databricks**.
• Develop data processing solutions using **PySpark and Python**.
• Work with **Apache Spark** for large-scale data processing.
• Build and optimize **ETL/ELT pipelines**.
• Work with **Delta Lake** for data storage and processing.
• Develop and manage workflows using **Databricks Workflows**.
• Perform data transformation, cleansing, and validation.
• Optimize Spark jobs and improve pipeline performance.
• Collaborate with data engineers, analysts, and other technical teams.
• Troubleshoot data pipeline issues and ensure data quality.

**Required Skills:**

• **8-10 years** of experience in **Data Engineering**.
• Strong hands-on experience with **Databricks**.
• Good knowledge of **PySpark and Python**.
• Strong understanding of **Apache Spark**.
• Experience with **Delta Lake** and data lake architecture.
• Good knowledge of **SQL**.
• Experience in building **ETL/ELT data pipelines**.
• Understanding of cloud platforms such as **Azure, AWS, or GCP**.
• Experience with **Git and CI/CD** is an added advantage.

**Good to Have:**

• Experience with **Azure Databricks**.
• Knowledge of **Azure Data Factory, ADLS, or similar cloud data services**.
• Experience with **DBT, Snowflake, or other modern data platforms**.
• Databricks certification is a plus.

**What We Offer:**

• Opportunity to work on real-world **Data Engineering and AI projects**.
• Exposure to modern data technologies and cloud platforms.
• Collaborative and learning-focused work environment.
• Opportunity to work with a growing technology team.

Pay: ₹500,000.00 - ₹1,800,000.00 per year

Work Location: Remote