Senior Data Engineer | Azure Databricks, Spark & Lakehouse

Humcap, Inc.; Plano (TX); Hybrid; USD 120,000 - 170,000; Full time

Job summary

Humcap, Inc. is seeking a Senior Data Engineer in Plano, TX to design, develop and optimize a cloud-based data platform. The role centers on Azure Databricks, PySpark, Spark SQL, and scalable ETL/ELT pipelines within a modern medallion data architecture.

You will collaborate with data architects, data scientists, and cloud infrastructure teams to deliver secure, high-performance data solutions and mentor junior engineers.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related technical field.

Responsibilities

• Design and implement scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Spark SQL.
• Build robust data ingestion workflows from diverse sources, including APIs, event-streaming platforms, relational databases, and cloud storage.
• Develop and maintain Bronze, Silver, and Gold data layers using a modern medallion architecture.
• Establish best practices for Delta Lake, schema evolution, data versioning, and governance.
• Optimize large-scale data pipelines for performance, reliability, and cost efficiency through partitioning, caching, clustering, and Spark configuration tuning.
• Integrate Databricks with Azure cloud services, including cloud data lakes, data integration services, event-streaming platforms, data warehouses, secrets management, and CI/CD tooling.
• Implement secure and governed access to enterprise data using managed identities, role-based access controls, and data protection techniques.
• Support the development and evolution of a scalable cloud-native data platform.
• Partner with data analysts, data scientists, business stakeholders, and other technical teams to understand data requirements.
• Build trusted datasets, reporting structures, and business-critical KPIs.
• Provide technical leadership and mentorship to junior and mid-level engineers.
• Conduct code reviews and contribute to architectural and engineering standards.
• Collaborate with architecture, security, and infrastructure teams to ensure data solutions meet organizational security and compliance requirements.
• Implement data quality checks, validation frameworks, alerts, and lineage tracking.
• Develop monitoring and observability solutions for mission-critical data pipelines.
• Establish automated failure detection, alerting, and recovery processes.
• Continuously improve data reliability, scalability, and operational performance.

Skills

Azure Databricks

PySpark

Spark SQL

Python

SQL

Data modeling

CI/CD tooling

Scala

Education

Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or related field

Tools

Delta Lake

Unity Catalog

Databricks Notebooks

SQL-based data warehousing

Azure DevOps

Terraform

Databricks CLI

Job description

Humcap, Inc. is seeking a Senior Data Engineer in Plano, TX to design, develop and optimize a cloud-based data platform. The role centers on Azure Databricks, PySpark, Spark SQL, and scalable ETL/ELT pipelines within a modern medallion data architecture.

You will collaborate with data architects, data scientists, and cloud infrastructure teams to deliver secure, high-performance data solutions and mentor junior engineers.