Databricks Data Engineer – Production Reliability & Performance

S27a; Northern (KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

S27a is seeking an experienced Databricks Data Engineer to maintain and optimize production data pipelines and Delta Lake workloads. You will ensure reliability, performance, and cost efficiency while collaborating with data, analytics, and engineering teams.

The role emphasizes production support, cluster tuning, and observability, with opportunities to implement best practices across security and governance in a cloud environment.

Qualifications

• 6–9 years of data engineering experience with Databricks.
• Strong Spark (PySpark/Scala) and SQL skills.
• Proven production Databricks workloads experience.
• Deep Delta Lake knowledge: ACID, time travel, optimization.
• Experience with Databricks clusters and performance tuning.
• Cloud experience with Azure, AWS or GCP in Databricks context.
• Ability to troubleshoot complex Spark/Databricks issues independently.

Responsibilities

• Support and maintain existing Databricks apps, notebooks, and Delta Lake pipelines in production.
• Monitor, troubleshoot, and resolve job failures and performance issues.
• Optimize Spark jobs, SQL queries, and Delta tables for cost and performance.
• Manage workspace configurations, clusters, job scheduling, Unity Catalog.
• Implement data quality checks, logging, and observability.
• Collaborate with stakeholders on enhancements or bug fixes.
• Perform incremental improvements and reduce technical debt.
• Ensure security, governance, and cost management in Databricks.
• Document pipelines, dependencies, and runbooks.
• Participate in on-call rotations.
• Finance Data Engineering Support.

Skills

Databricks

Apache Spark

SQL

Delta Lake

Performance tuning

Cloud platforms

Troubleshooting

Data pipelines

Tools

Airflow

Azure Data Factory

Databricks Workflows

Job description

S27a is seeking an experienced Databricks Data Engineer to maintain and optimize production data pipelines and Delta Lake workloads. You will ensure reliability, performance, and cost efficiency while collaborating with data, analytics, and engineering teams.

The role emphasizes production support, cluster tuning, and observability, with opportunities to implement best practices across security and governance in a cloud environment.