Senior Databricks Engineer: Scalable Data Pipelines

Compunnel, Inc.; Houston (TX); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading data solutions provider in the United States is seeking a skilled Databricks Engineer to design and build scalable cloud-based data pipelines for enterprise analytics platforms. The ideal candidate will have over 5 years of data engineering experience, including substantial work with Databricks, and will be responsible for optimizing data workflows, collaborating with cross-functional teams, and ensuring data quality. This role offers the opportunity to engage with modern data architecture and cloud technologies.

Qualifications

• 5+ years of experience in data engineering with at least 3+ years with Databricks.
• Strong hands-on experience with Apache Spark and/or Delta Live Tables.
• Proficiency in PySpark and SQL for building and optimizing large-scale data pipelines.
• Experience designing and implementing ETL/ELT workflows in cloud environments.
• Excellent problem-solving and communication skills in client-facing settings.

Responsibilities

• Design and develop scalable batch and streaming data pipelines in Databricks.
• Build ETL/ELT processes to integrate data into analytical datasets.
• Optimize pipelines for performance and cost-efficiency.
• Work with cloud platform teams on storage, networking, and security setup.
• Deliver high-quality outputs and maintain strong client relationships.

Skills

Data engineering

PySpark

SQL

ETL/ELT workflows

Apache Spark

Delta Live Tables

Cloud environments (Azure or AWS)

Data architecture concepts

CI/CD practices

Problem-solving

Education

Bachelor's degree in Computer Science

Tools

Databricks

Delta Lake

Job description

A leading data solutions provider in the United States is seeking a skilled Databricks Engineer to design and build scalable cloud-based data pipelines for enterprise analytics platforms. The ideal candidate will have over 5 years of data engineering experience, including substantial work with Databricks, and will be responsible for optimizing data workflows, collaborating with cross-functional teams, and ensuring data quality. This role offers the opportunity to engage with modern data architecture and cloud technologies.