Remote Databricks Engineer — Spark & Delta Lake Expert

Arting-Digital-Private-Limited; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

Arting Digital Private Limited is seeking a Mid-Senior Databricks Engineer for remote work. You will design and maintain scalable data pipelines using Databricks, Python, and PySpark, aligning with data governance standards.

You will optimize Spark workloads, implement Delta Lake strategies, and orchestrate ETL workflows with Airflow or Azure Data Factory, collaborating with cross-functional teams to deliver robust data solutions.

Qualifications

• 3–6 years of experience in Data Engineering.
• Strong knowledge of SQL, Python, and PySpark.
• Hands-on Databricks and Apache Spark experience.
• Understanding of Spark optimization, partitioning, and shuffling.
• Experience with Delta Lake, Z-Ordering, and VACUUM.
• Knowledge of Apache Airflow or Azure Data Factory.
• Understanding of data governance and data quality.

Responsibilities

• Develop and maintain scalable data pipelines using Databricks, Python, and PySpark.
• Build and automate ETL workflows using Delta Live Tables (DLT).
• Optimize Spark jobs through partitioning, shuffling, and performance tuning.
• Implement Delta Lake optimization techniques, including Z-Ordering and VACUUM.
• Manage data orchestration using Airflow or Azure Data Factory.
• Ensure data quality, governance, and pipeline performance.
• Collaborate with cross-functional teams to deliver efficient data engineering solutions.

Skills

SQL

Python

PySpark

Spark optimization

Data governance

Tools

Databricks

Delta Live Tables

Delta Lake

Z-Ordering

VACUUM

Airflow

Azure Data Factory

Job description

Arting Digital Private Limited is seeking a Mid-Senior Databricks Engineer for remote work. You will design and maintain scalable data pipelines using Databricks, Python, and PySpark, aligning with data governance standards.

You will optimize Spark workloads, implement Delta Lake strategies, and orchestrate ETL workflows with Airflow or Azure Data Factory, collaborating with cross-functional teams to deliver robust data solutions.