DataBricks Data Engineer USA

JobCubby; Northern (KY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

JobCubby is seeking an experienced Databricks Data Engineer to support, maintain, and optimize Databricks-based data applications and pipelines. The role focuses on reliability, performance, and scalability of production Databricks workloads, working closely with data, analytics, and engineering teams to keep critical data applications stable and aligned with business needs.

You will monitor and improve pipelines, collaborate on enhancements, and ensure adherence to security and governance

Qualifications

• 6–9 years of data engineering experience.
• Strong hands-on Databricks experience.
• Proficiency in Apache Spark (PySpark/Scala) and SQL.
• Knowledge of Delta Lake concepts and optimizations.
• Experience with cloud platforms (Azure/AWS/GCP).

Responsibilities

• Maintain and optimize production Databricks workloads (jobs, notebooks, Delta Lake).
• Monitor, troubleshoot, and fix job failures and performance issues.
• Tune Spark queries and Delta tables for cost and reliability.
• Manage workspace configurations, clusters, and access controls.
• Implement data quality checks, logging, and basic observability.

Skills

Databricks

Spark

SQL

Delta Lake

ETL/ELT

Cloud platforms

Tools

Azure/AWS/GCP

Databricks Job clusters

Databricks Workflows

Unity Catalog

Job description

Job Title: Databricks Data Engineer

Location: Remote

Work Hours: EST (Eastern Standard Time)

Experience: 6–9 years

Role Overview

We are looking for an experienced Databricks Data Engineer to support, maintain, and enhance existing Databricks-based data applications and pipelines. The role focuses on ensuring reliability, performance, and scalability of production Databricks workloads rather than building net-new platforms from scratch. You will work closely with data, analytics, and engineering teams to keep critical data applications stable, optimized, and aligned with business needs.

Key Responsibilities

• Support and maintain existing Databricks applications, notebooks, jobs, and Delta Lake pipelines in production.
• Monitor, troubleshoot, and resolve issues related to job failures, performance degradation, data quality, and cluster utilization.
• Optimize existing Spark jobs, SQL queries, and Delta tables for cost, performance, and reliability.
• Manage and improve Databricks workspace configurations, including clusters, job scheduling, access controls, and Unity Catalog (where applicable).
• Implement and maintain data quality checks, logging, alerting, and basic observability for Databricks workloads.
• Collaborate with stakeholders to understand requirements for enhancements or bug fixes on existing applications.
• Perform incremental improvements, refactoring, and technical debt reduction on current Databricks solutions.
• Ensure adherence to best practices around security, governance, and cost management within the Databricks environment.
• Document existing pipelines, dependencies, and operational runbooks.
• Participate in on-call or support rotations as needed to maintain production stability (within EST working hours).

Finance Data Engineering Support

Example Activities

• Data engineering support for Finance
• Data pipeline design and development
• Data integration support
• Databricks platform development
• Automation design and implementation
• Technical testing
• Data quality validation
• Technical troubleshooting
• Reporting and analytics enablement
• Technical documentation
• Stakeholder coordination
• Deployment and release support

Example Work Products and Deliverables

• Technical specifications
• Solution designs
• Data flow documentation
• Pipeline documentation
• Data mapping documentation
• Databricks job orchestration documentation
• Databricks automation and workflow designs
• Databricks workflow configurations
• Deployment documentation
• Reporting outputs
• Technical status reports

General Transformation and Operational Support

Example activities

• Project coordination
• Business analysis
• Process improvement
• Operational readiness activities
• Reporting support
• Meeting facilitation
• Documentation support
• Stakeholder engagement
• Training and change support
• Cross-functional coordination

Example Work Products and Deliverables

• Analysis reports
• Project plans
• Process maps
• Status reports
• Presentations
• Training materials
• Transition plans
• Readiness assessments

Required Qualifications

• 6–9 years of overall experience in data engineering, with strong hands-on experience in Databricks.
• Solid proficiency in Apache Spark (PySpark and/or Scala) and SQL.
• Proven experience supporting and optimizing production Databricks workloads (jobs, notebooks, Delta Lake, workflows).
• Strong understanding of Delta Lake concepts (ACID transactions, time travel, optimization techniques such as Z-ordering, vacuum, optimize).
• Experience with Databricks Job clusters, Interactive clusters, and performance tuning (partitioning, caching, shuffle optimization, autoscaling).
• Familiarity with data modeling, ETL/ELT patterns, and production data pipeline support.
• Experience working with cloud platforms (preferably Azure, AWS, or GCP) in the context of Databricks.
• Ability to troubleshoot complex Spark and Databricks issues independently.
• Strong communication skills and ability to work effectively in a remote, EST-aligned team.

Preferred Qualifications

• Experience with Unity Catalog, Databricks SQL, or Lakehouse architecture.
• Knowledge of CI/CD practices for Databricks (e.g., Databricks Asset Bundles, Git integration, Terraform/ARM templates).
• Familiarity with orchestration tools (Airflow, Azure Data Factory, or Databricks Workflows).
• Exposure to data quality frameworks, monitoring tools, or cost optimization initiatives on Databricks.
• Experience supporting analytics or BI teams consuming Databricks data products.

Work Arrangement

• Fully remote
• Must be available and productive during EST business hours
• Collaborative remote environment with regular syncs and support responsibilities