Databricks Lakehouse Engineer: PySpark, Delta, AI/ML

Delan Associates, Inc; Louisville (KY); On-site; USD 100,000 - 150,000; Full time

Job summary

Delan Associates, Inc. is seeking a Databricks Developer to design and maintain scalable data platforms in the Databricks Lakehouse architecture.

You will build robust data pipelines using PySpark, Python, and SQL to support analytics, reporting, and AI/ML initiatives. The role requires strong Delta Lake expertise, data modeling skills, and experience implementing data quality and governance controls across enterprise systems.

Qualifications

• 4-8 years of experience in Data Engineering and Analytics.
• Minimum 3+ years of hands-on experience with Databricks and PySpark.
• Experience building scalable data platforms with Delta Lake and data governance.
• Experience in Agile development environments.

Responsibilities

• Design scalable data solutions using Databricks Lakehouse Architecture.
• Develop and maintain data pipelines with PySpark, Python, and SQL.
• Build and optimize ETL/ELT workflows for batch and near real-time processing.
• Implement Delta Lake features including ACID, time travel, schema evolution, data versioning.
• Design enterprise data models to support reporting and analytics requirements.
• Ensure data quality through validation, monitoring, reconciliation, and governance.
• Develop and manage data catalogs, metadata management, and data lineage processes.
• Collaborate with stakeholders to translate requirements into technical solutions.
• Optimize Databricks workloads for performance, scalability, and cost efficiency.
• Implement security, access controls, and governance best practices.
• Support troubleshooting, root cause analysis, and production issue resolution.
• Contribute to data platform modernization and cloud migrations.

Skills

Databricks Architecture

PySpark

Python

SQL

Delta Lake

Data Modeling

Data Quality Frameworks

Data Governance

Data Lineage

Unity Catalog

CI/CD Pipelines

Azure Databricks

Azure Data Factory

Azure Synapse Analytics

Education

Bachelor's degree in Computer Science or related field

Master's degree in Data Engineering or related field

Tools

Azure Data Factory

Azure Databricks

Azure Synapse Analytics

Delta Lake

Unity Catalog

CI/CD pipelines (Azure DevOps/GitHub Actions)

Job description

Delan Associates, Inc. is seeking a Databricks Developer to design and maintain scalable data platforms in the Databricks Lakehouse architecture.

You will build robust data pipelines using PySpark, Python, and SQL to support analytics, reporting, and AI/ML initiatives. The role requires strong Delta Lake expertise, data modeling skills, and experience implementing data quality and governance controls across enterprise systems.