Databricks Lakehouse Engineer: Scalable Data Pipelines

OpenTalent; United States; On-site; USD 110,000 - 170,000; Full time

Job summary

OpenTalent is seeking a highly skilled Databricks Developer to design and manage modern data platforms using the Databricks Lakehouse architecture. You will build scalable data pipelines with PySpark, Python, and SQL, implement Delta Lake features, and drive data modeling, quality, and governance across enterprise-scale systems.

You will collaborate with architects, analytics teams, and business stakeholders to translate requirements into robust technical solutions, optimize workloads for

Qualifications

• Must have strong experience with PySpark for large-scale data processing.
• Proficient in Python programming and SQL development.
• Experience building and optimizing Databricks Lakehouse platforms.
• Knowledge of Delta Lake features (ACID, time travel, schema evolution).
• Experience with data modeling techniques (Star/Snowflake/Data Vault).
• Experience implementing data quality frameworks and governance.

Responsibilities

• Design scalable data solutions using Databricks Lakehouse Architecture.
• Develop and maintain data pipelines with PySpark, Python, and SQL.
• Build and optimize ETL/ELT workflows for batch and near real-time processing.
• Implement Delta Lake features including ACID transactions and time travel.
• Design enterprise data models to support reporting and analytics.
• Ensure data quality with validation, monitoring, reconciliation, and governance.
• Manage data catalogs, metadata management, and data lineage.
• Collaborate with stakeholders to translate requirements into technical solutions.
• Optimize Databricks workloads for performance and cost efficiency.
• Implement security and access controls within the Databricks ecosystem.

Skills

PySpark

Python

SQL

Delta Lake

Data modeling

Data quality

Data governance

Tools

Databricks

Unity Catalog

Data Catalog

Azure Data Factory

Azure Synapse

ADLS

Job description

OpenTalent is seeking a highly skilled Databricks Developer to design and manage modern data platforms using the Databricks Lakehouse architecture. You will build scalable data pipelines with PySpark, Python, and SQL, implement Delta Lake features, and drive data modeling, quality, and governance across enterprise-scale systems.

You will collaborate with architects, analytics teams, and business stakeholders to translate requirements into robust technical solutions, optimize workloads for