Senior Databricks Engineer

Compunnel, Inc.; Houston (TX); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading data solutions provider in the United States is seeking a skilled Databricks Engineer to design and build scalable cloud-based data pipelines for enterprise analytics platforms. The ideal candidate will have over 5 years of data engineering experience, including substantial work with Databricks, and will be responsible for optimizing data workflows, collaborating with cross-functional teams, and ensuring data quality. This role offers the opportunity to engage with modern data architecture and cloud technologies.

Qualifications

• 5+ years of experience in data engineering with at least 3+ years with Databricks.
• Strong hands-on experience with Apache Spark and/or Delta Live Tables.
• Proficiency in PySpark and SQL for building and optimizing large-scale data pipelines.
• Experience designing and implementing ETL/ELT workflows in cloud environments.
• Excellent problem-solving and communication skills in client-facing settings.

Responsibilities

• Design and develop scalable batch and streaming data pipelines in Databricks.
• Build ETL/ELT processes to integrate data into analytical datasets.
• Optimize pipelines for performance and cost-efficiency.
• Work with cloud platform teams on storage, networking, and security setup.
• Deliver high-quality outputs and maintain strong client relationships.

Skills

Data engineering

PySpark

SQL

ETL/ELT workflows

Apache Spark

Delta Live Tables

Cloud environments (Azure or AWS)

Data architecture concepts

CI/CD practices

Problem-solving

Education

Bachelor's degree in Computer Science

Tools

Databricks

Delta Lake

Job description

Our client is seeking a skilled and passionate Databricks Engineer to design and build scalable, cloud-based data pipelines for enterprise analytics platforms. The role focuses on creating robust ETL/ELT workflows using Databricks and Spark, with opportunities to leverage advanced features such as Delta Live Tables (DLT) and cloud-native tools. The ideal candidate has strong hands-on experience in PySpark and SQL, and a proven ability to build pipelines from the ground up while collaborating with cross-functional teams.

KEY RESPONSIBILITIES

• Design and develop scalable batch and streaming data pipelines in Databricks using PySpark, SQL, and Delta Lake.
• Build ETL/ELT processes to transform and integrate data from diverse sources into unified analytical datasets.
• Leverage Delta Live Tables (DLT) and Databricks Workflows to automate and orchestrate pipelines (preferred).
• Optimize pipelines for performance, cost-efficiency, and reusability by applying Spark tuning and partitioning strategies.
• Collaborate with architects, data modelers, and analysts to implement business logic into pipelines.
• Implement data quality checks, monitoring, and observability practices across pipelines.
• Participate in Agile delivery processes, including sprint planning, code reviews, and deployments.
• Work with cloud platform teams to ensure proper storage, networking, and security setup.
• Deliver high-quality outputs, meet deadlines, and maintain strong client relationships.
• Provide consultative guidance, seek creative solutions, and proactively identify opportunities for improvement.
• Stay current with evolving technology trends and best practices in data engineering.

REQUIRED SKILLS & EXPERIENCE

• 5+ years of experience in data engineering, including at least 3+ years with Databricks.
• Strong hands-on experience with Apache Spark and/or Delta Live Tables (DLT).
• Proficiency in PySpark and SQL for building and optimizing large-scale data pipelines.
• Experience designing and implementing ETL/ELT workflows in cloud environments (Azure or AWS).
• Familiarity with Delta Lake, Unity Catalog, and Databricks Jobs/Workflows.
• Experience with structured and semi-structured data from APIs, cloud storage, and enterprise systems.
• Strong understanding of modern data architecture concepts (Medallion architecture, lakehouse patterns).
• Knowledge of Git-based version control, CI/CD practices, and infrastructure automation.
• Excellent problem-solving and communication skills, particularly in client-facing settings.
• Bachelor’s degree in Computer Science, Information Systems, or equivalent experience.

PREFERRED QUALIFICATIONS

• Experience with Delta Live Tables (DLT) for declarative pipeline development.
• Exposure to IDMC (Informatica Intelligent Data Management Cloud) or other integration platforms.
• Databricks or cloud provider certifications (Azure/AWS).
• Familiarity with data governance tools (Unity Catalog, Purview, Collibra).
• Experience with REST APIs and streaming technologies (Kafka, Event Hubs, etc.).

CERTIFICATIONS

Databricks Certified Data Engineer or equivalent cloud certifications (preferred).

EDUCATION

Bachelor’s degree in computer science, Information Systems, or a related field (Master’s degree preferred).