Databricks Lakehouse Engineer: PySpark, SQL & Data Quality

OpenTalent; United States; On-site; USD 110,000 - 150,000; Full time

Job summary

OpenTalent is seeking a highly skilled Databricks Developer to design, build, and optimize data platforms using the Databricks Lakehouse architecture. The role focuses on scalable data pipelines, data modeling, and enabling analytics, reporting, and AI/ML initiatives.

You will work with PySpark, Python, SQL, Delta Lake, and data governance practices while collaborating with stakeholders to deliver robust, cost-efficient solutions in a cloud environment.

Qualifications

• Experience with Databricks Lakehouse Architecture and platform administration.
• Strong PySpark, Python, and SQL development skills.
• Experience building scalable ETL/ELT pipelines.
• Knowledge of Delta Lake features: ACID, time travel, schema evolution.
• Experience with data governance, metadata management.
• Ability to collaborate with stakeholders to translate requirements into technical solutions.

Responsibilities

• Design and implement scalable data solutions using Databricks Lakehouse Architecture.
• Develop and maintain data pipelines using PySpark, Python, and SQL.
• Build and optimize ETL/ELT workflows for batch and near real-time processing.
• Implement Delta Lake features including ACID transactions, time travel, schema evolution, and data versioning.
• Design and maintain enterprise data models to support reporting and analytics requirements.
• Ensure data quality through validation, monitoring, reconciliation, and governance controls.
• Develop and manage data catalogs, metadata management, and data lineage processes.
• Collaborate with business stakeholders, architects, and analytics teams to gather and translate requirements into technical solutions.
• Optimize Databricks workloads for performance, scalability, and cost efficiency.
• Implement security, access controls, and governance best practices within the Databricks ecosystem.
• Support troubleshooting, root cause analysis, and production issue resolution.
• Contribute to data platform modernization and cloud migration initiatives.

Skills

Databricks architecture

Delta Lake

PySpark

Python

SQL

Data modeling

Data quality

Data governance

Metadata management

Unity Catalog

Data pipelines

Cloud optimization

Tools

Databricks

Job description

OpenTalent is seeking a highly skilled Databricks Developer to design, build, and optimize data platforms using the Databricks Lakehouse architecture. The role focuses on scalable data pipelines, data modeling, and enabling analytics, reporting, and AI/ML initiatives.

You will work with PySpark, Python, SQL, Delta Lake, and data governance practices while collaborating with stakeholders to deliver robust, cost-efficient solutions in a cloud environment.