Senior Data Engineer - Hybrid Azure Data & ML Pipelines

HeartCentrix Solutions; Texas; Hybrid; USD 130,000 - 190,000; Full time

Job summary

HeartCentrix Solutions seeks a Senior Data Engineer to design and build scalable data and application solutions using Python, Spark, Databricks, and Azure. This role develops complex ETL/ELT pipelines for large-scale datasets and collaborates with data scientists, analysts, and business stakeholders to translate challenges into technical solutions.

The position is hybrid with regular onsite presence, requiring on-site work at least part of the week and strong emphasis on performance, cost

Qualifications

• Advanced proficiency in Python/Scala/SQL for data processing.
• Experience with ETL/ELT and data handling.
• Spark and Databricks programming experience.
• Experience designing and building CI/CD pipelines (Azure with GitHub Actions).
• Experience with logging, monitoring, and alerting systems.
• Familiarity with AI tools to boost productivity.

Responsibilities

• Design and build scalable data/app solutions using Python and Azure ML.
• Develop ETL/ELT processes on large-scale datasets.
• Optimize pipelines for performance and cost.
• Collaborate with data scientists, analysts, and stakeholders.
• Implement data security, governance, and Unity Catalog access controls.
• Champion CI/CD pipelines to automate deployment and management.
• Design CI/CD pipelines in Azure with GitHub Actions.
• Develop monitoring, logging, and alerting for production pipelines.

Skills

Python

Scala

SQL

ETL/ELT

CI/CD

Azure

Logging/Monitoring

AI tools

Tools

Spark

Databricks

Azure DevOps

GitHub Actions

Jenkins

Job description

HeartCentrix Solutions seeks a Senior Data Engineer to design and build scalable data and application solutions using Python, Spark, Databricks, and Azure. This role develops complex ETL/ELT pipelines for large-scale datasets and collaborates with data scientists, analysts, and business stakeholders to translate challenges into technical solutions.

The position is hybrid with regular onsite presence, requiring on-site work at least part of the week and strong emphasis on performance, cost