Senior Databricks Engineer

Lumenalta; United States; Remote; USD 80,000 - 150,000; Full time

Benefits offered by this job

Health insurance (medical and dental)

401K Contribution

Professional development fund

Flexible remote hours

15 days paid time off

Job summary

Lumenalta seeks a Senior Data Engineer - Databricks to design and implement modern enterprise data platforms in the cloud, blending software engineering with data architecture.

You will build scalable PySpark, Python, and SQL pipelines and lead CI/CD, testing, and governance across the data platform.

Experience with Delta Lake, Unity Catalog, Databricks Asset Bundles, Lakeflow, Terraform, and AWS will help you drive secure, scalable solutions for US clients in a remote-ready setting.

Qualifications

• Deep expertise in Databricks, including platform architecture and best practices.
• Experience as a Solution Architect or Data Platform Owner designing end-to-end implementations.
• Strong programming experience with Python, PySpark, and SQL.
• Experience using testing frameworks such as PyTest.
• Git-based workflows, CI/CD pipelines, and DevOps practices.
• Hands-on experience with Delta Lake, Unity Catalog, Databricks Asset Bundles, and Lakeflow Jobs.
• Infrastructure as Code using Terraform.
• Strong AWS experience, particularly S3 and IAM roles.

Responsibilities

• Design and implement scalable enterprise data platforms using Databricks and AWS.
• Architect and oversee end-to-end data platform implementations.
• Develop robust data pipelines using Python, PySpark, and SQL.
• Apply strong engineering practices including testing, CI/CD, and version control.
• Implement and manage Databricks platform components including Delta Lake, Unity Catalog, Databricks Asset Bundles, and Lakeflow Jobs.
• Collaborate with engineering teams to ensure maintainable, scalable data solutions.
• Implement Infrastructure as Code practices to support repeatable environments.
• Contribute to architectural decisions and guide best practices for the data platform.

Skills

Databricks

Python

PySpark

SQL

CI/CD

DevOps

Git workflows

Terraform

AWS

Delta Lake

Unity Catalog

Databricks Asset Bundles

Lakeflow Jobs

PyTest

Tools

Terraform

Git

Databricks

Job description

We are seeking a Senior Data Engineer - Databricks with strong expertise in Databricks to help design and implement modern enterprise data platforms. This role blends strong software engineering practices with advanced data platform architecture, requiring hands-on experience building scalable pipelines and designing robust data systems in the cloud. The engineer will contribute to platform architecture, implement reliable data solutions, and ensure best practices across development, testing, deployment, and governance within the data platform ecosystem.

Responsibilities

• Design and implement scalable enterprise data platforms using Databricks and AWS.
• Architect and oversee end-to-end data platform implementations.
• Develop robust data pipelines using Python, PySpark, and SQL.
• Apply strong engineering practices including testing, CI/CD, and version control.
• Implement and manage Databricks platform components including Delta Lake, Unity Catalog, Databricks Asset Bundles, and Lakeflow Jobs.
• Collaborate with engineering teams to ensure maintainable, scalable, and secure data solutions.
• Implement Infrastructure as Code practices to support repeatable environments.
• Contribute to architectural decisions and guide best practices for the data platform.

Requirements

• Deep expertise in Databricks, including platform architecture and best practices.
• Experience as a Solution Architect or Data Platform Owner designing end-to-end implementations.
• Strong programming experience with Python, PySpark, and SQL.
• Experience using testing frameworks such as PyTest.
• Solid experience with Git-based workflows, CI/CD pipelines, and DevOps practices.
• Hands-on experience with Delta Lake, Unity Catalog, Databricks Asset Bundles (DABs), and Lakeflow Jobs.
• Experience with Infrastructure as Code using Terraform.
• Strong AWS experience, particularly S3 and IAM roles.
• Strong communication and collaboration skills in distributed teams.

Nice to Have

• Experience designing lakehouse architectures at scale.
• Experience optimizing distributed data workloads.
• Experience working in consulting or client-facing engineering environments.
• Experience implementing governance, security, and data lineage frameworks.
• Strong hands-on experience with AWS Glue (ETL, jobs, crawlers, workflows).

Salary range: $80,000 - $150,000 annually, with final compensation determined by your qualifications, expertise, experience, and the role's scope.

Location

This is a fully remote position. This role is remote within the United States only. Candidates must reside in the United States while assigned to our client and be physically located within the United States when performing project work or accessing project systems or data. Availability to work overlapping Pacific, Central, or Eastern U.S. time zones.

In addition to competitive pay, we offer a variety of benefits to support your professional and personal growth, including:

• Flexible working hours in a remote environment.
• Health insurance (medical and dental) for W2 Employees.
• 401K Contribution.
• A professional development fund to enhance your skills and knowledge.
• 15 days of paid time off annually.
• Access to soft-skill development courses to further your career.

This is a full-time position requiring a minimum of 40 hours per week, Monday through Friday.

At Lumenalta, we are committed to creating an environment that prioritizes growth, work-life balance, and the diverse needs of our team members.

Application Deadline

Applications will be accepted until October 25th, 2026. Candidates can expect feedback by November 2nd, 2026.