Databricks Data Engineer

Compunnel, Inc.; Spring (TX); On-site; USD 110,000 - 140,000; Full time

Job summary

Compunnel, Inc. is looking for a Databricks Data Engineer to design, develop, and maintain data pipelines in a Lakehouse environment. The successful candidate will have over 6 years of Data Engineering experience, with expertise in Databricks, Spark, and strong programming skills in Python and SQL.

Responsibilities include optimizing ETL/ELT pipelines, ensuring data quality, and collaborating with stakeholders to meet analytics requirements. Join us to elevate data engineering practices!

Qualifications

• 6+ years of experience in Data Engineering.
• Hands-on experience with Databricks and Apache Spark.
• Strong programming experience with Python.
• Strong SQL development and query optimization skills.

Responsibilities

• Develop and maintain data ingestion pipelines.
• Design, build, and optimize ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Implement and support Medallion Architecture layers.
• Monitor, troubleshoot, and resolve data pipeline execution issues.

Skills

Databricks

Apache Spark

Python

SQL

ETL/ELT pipelines

Delta Lake architectures

Data governance

Data quality

Job description

The Databricks Data Engineer will be responsible for designing, developing, and maintaining data ingestion and transformation pipelines within a Databricks Lakehouse environment. This role focuses on integrating data from multiple source systems, implementing scalable data architectures, and preparing high-quality datasets for downstream analytics and reporting. The ideal candidate will have strong experience with Databricks, Spark, Python, SQL, and modern data engineering practices.

Key Responsibilities

• Develop and maintain data ingestion pipelines that integrate data from multiple source systems.
• Design, build, and optimize ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Implement and support Medallion Architecture layers, including Bronze, Silver, and Gold data models.
• Ensure data quality, validation, and consistency across data pipelines and datasets.
• Monitor, troubleshoot, and resolve data pipeline execution issues.
• Optimize pipeline performance, scalability, and reliability to support enterprise data workloads.
• Work with structured and semi-structured data formats from diverse source systems.
• Support data transformation, cleansing, enrichment, and integration activities.
• Collaborate with data analysts, data scientists, and business stakeholders to support analytics requirements.
• Maintain technical documentation related to data pipelines, architectures, and processes.
• Follow data engineering best practices for performance, security, governance, and operational support.
• Participate in code reviews, testing, and continuous improvement initiatives.

Required Qualifications

• 6+ years of experience in Data Engineering.
• Hands‑on experience with Databricks and Apache Spark.
• Strong programming experience with Python.
• Strong SQL development and query optimization skills.
• Experience designing, developing, and maintaining ETL/ELT pipelines.
• Experience implementing and supporting Delta Lake architectures.
• Understanding of distributed data processing systems and large‑scale data platforms.
• Experience working with structured and semi‑structured data formats.
• Strong troubleshooting, performance tuning, and problem‑solving skills.
• Experience monitoring and supporting production data pipelines.
• Ability to work effectively in a collaborative team environment.
• Strong communication and documentation skills.

Preferred Qualifications

• Experience implementing Medallion Architecture (Bronze, Silver, Gold) within Databricks environments.
• Experience working with cloud‑based data platforms and modern data lake architectures.
• Experience supporting enterprise analytics, reporting, and business intelligence initiatives.
• Knowledge of data governance, data quality, and data lifecycle management best practices.