Databricks Lakehouse Engineer - PySpark & Data Quality

Delan Associates, Inc.; United States; On-site; USD 110,000 - 170,000; Full time

Job summary

Delan Associates, Inc. is seeking a Databricks Developer to design, build, and optimize data pipelines on the Databricks Lakehouse.

The ideal candidate will leverage PySpark, Python, and SQL to deliver scalable analytics, reporting, and AI/ML-ready data platforms, with Delta Lake for ACID and governance. You will implement data models, ensure data quality, manage catalogs and lineage, and partner with architects and analytics teams to translate requirements into robust enterprise solutions while

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.
• 4-8 years of experience in Data Engineering and Analytics.
• Minimum 3+ years of hands-on experience with Databricks and PySpark.
• Experience working in Agile development environments.

Responsibilities

• Design and implement scalable data solutions using Databricks Lakehouse Architecture.
• Develop and maintain data pipelines using PySpark, Python, and SQL.
• Build and optimize ETL/ELT workflows for batch and near real-time data processing.
• Implement Delta Lake features including ACID transactions, time travel, schema evolution, and data versioning.
• Design and maintain enterprise data models to support reporting and analytics requirements.
• Ensure data quality through validation, monitoring, reconciliation, and governance controls.
• Develop and manage data catalogs, metadata management, and data lineage processes.
• Collaborate with business stakeholders, architects, and analytics teams to gather and translate requirements into technical solutions.
• Optimize Databricks workloads for performance, scalability, and cost efficiency.
• Implement security, access controls, and governance best practices within the Databricks ecosystem.
• Support troubleshooting, root cause analysis, and production issue resolution.
• Contribute to data platform modernization and cloud migration initiatives.

Skills

PySpark

Python

SQL

ETL/ELT pipelines

Data modeling

Data quality frameworks

Data lineage

Governance practices

Collaboration with stakeholders

Education

Bachelor's/Master's in CS/IT/Data Eng

4-8 years Data Eng/Analytics

3+ years Databricks & PySpark

Agile development experience

Tools

Databricks

Delta Lake

Unity Catalog

Data Catalog

Azure Databricks

Azure Data Lake Storage

Azure Data Factory

Azure Synapse

CI/CD pipelines (Azure DevOps, GitHub Actions)

Job description

Delan Associates, Inc. is seeking a Databricks Developer to design, build, and optimize data pipelines on the Databricks Lakehouse.

The ideal candidate will leverage PySpark, Python, and SQL to deliver scalable analytics, reporting, and AI/ML-ready data platforms, with Delta Lake for ACID and governance. You will implement data models, ensure data quality, manage catalogs and lineage, and partner with architects and analytics teams to translate requirements into robust enterprise solutions while