Databricks Data Engineer: Lakehouse & Spark Expert

CloudBigD

Louisville (KY)

On-site

USD 120,000 - 180,000

Full time

3 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

CloudBigD is seeking an experienced Databricks Developer to design, develop, and optimize scalable data processing solutions on the Databricks Lakehouse Platform. The role focuses on Spark, PySpark, Spark SQL, and Delta Lake to power enterprise-scale data modernization and analytics initiatives.

You will design and maintain batch and real-time data pipelines, implement data quality checks, and optimize performance while ensuring governance and security across the data estate.

Qualifications

  • Bachelor's degree required in CS/IS/Engineering or related field.
  • 8+ years hands-on Databricks development experience.
  • Strong expertise in Apache Spark and the Databricks Lakehouse Platform.
  • Hands-on experience developing data pipelines using PySpark and Spark SQL.
  • Experience with Delta Lake and its capabilities.

Responsibilities

  • Design, develop, and maintain scalable batch and real-time data pipelines using Databricks.
  • Build robust data processing solutions using PySpark, Spark SQL, and Delta Lake.
  • Implement data validation, data quality checks, and performance-tuning strategies.
  • Design and manage data architecture within the Databricks Lakehouse Platform.
  • Implement and manage Delta Lake tables, partitioning, optimization, and schema evolution.
  • Develop data models to support analytics and reporting requirements.
  • Ensure data reliability, scalability, and governance.
  • Analyze and optimize Spark jobs for improved performance and cost efficiency.
  • Monitor cluster utilization and apply resource management best practices.
  • Identify and troubleshoot bottlenecks in data processing workflows.
  • Improve processing times through caching, partitioning, and query optimization.
  • Implement role-based access controls and data security standards.
  • Support data governance initiatives, data lineage tracking, and compliance requirements.
  • Ensure adherence to enterprise data management policies.
  • Collaborate with Data Architects, Data Scientists, Business Analysts, and Application Teams.
  • Participate in Agile development processes and sprint ceremonies.
  • Prepare technical design documentation and support knowledge transfer activities.

Skills

Apache Spark
Databricks Lakehouse Platform
PySpark
Spark SQL
Delta Lake

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field

Tools

Databricks Platform

Job description

CloudBigD is seeking an experienced Databricks Developer to design, develop, and optimize scalable data processing solutions on the Databricks Lakehouse Platform. The role focuses on Spark, PySpark, Spark SQL, and Delta Lake to power enterprise-scale data modernization and analytics initiatives.

You will design and maintain batch and real-time data pipelines, implement data quality checks, and optimize performance while ensuring governance and security across the data estate.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Data Engineer: Lakehouse & Streaming Specialist
Databricks Data Engineer: Lakehouse & Streaming Specialist

CloudBigD • Houston (TX)

On-site
USD 120,000 - 180,000
Databricks Data Engineer — Lakehouse & Governance Expert
Databricks Data Engineer — Lakehouse & Governance Expert

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 170,000
Senior Data Engineer: Databricks Lakehouse Leader
Senior Data Engineer: Databricks Lakehouse Leader

Tata Consultancy Services Limited • Irvine (CA)

On-site
USD 120,000 - 180,000
Databricks Data Engineer - Lakehouse & Delta Lead
Databricks Data Engineer - Lakehouse & Delta Lead

Jobgether • United States

Remote
USD 110,000 - 160,000
Healthcare coverage
Meal vouchers
Life insurance
+4
Senior Databricks Lakehouse Architect (PySpark/ETL)
Senior Databricks Lakehouse Architect (PySpark/ETL)

VService IT Solutions • United States

Remote
USD 140,000 - 190,000
Senior Databricks Lakehouse Engineer
Senior Databricks Lakehouse Engineer

Robotics Technologies LLC • Louisville (KY)

On-site
USD 90,000 - 130,000
Remote Data Engineer – Databricks, PySpark & Lakehouse
Remote Data Engineer – Databricks, PySpark & Lakehouse

Career Sigma Software • United States

Remote
USD 120,000 - 150,000
Azure Databricks Data Engineer — Lakehouse & Spark Expert
Azure Databricks Data Engineer — Lakehouse & Spark Expert

Quantum Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 180,000
Remote Lakehouse Data Engineer | Databricks
Remote Lakehouse Data Engineer | Databricks

BairesDev • United States

Remote
USD 120,000 - 180,000
100% remote work
Competitive USD salary
Home office setup provided
+3
Remote Data Engineer: Databricks Lakehouse & MDM
Remote Data Engineer: Databricks Lakehouse & MDM

Lakefusion • United States

Remote
USD 19,000 - 32,000