Senior Data Engineer

HeartCentrix Solutions; Texas; Hybrid; USD 130.000 - 190.000; Vollzeit

Zusammenfassung

HeartCentrix Solutions seeks a Senior Data Engineer to design and build scalable data and application solutions using Python, Spark, Databricks, and Azure. This role develops complex ETL/ELT pipelines for large-scale datasets and collaborates with data scientists, analysts, and business stakeholders to translate challenges into technical solutions.

The position is hybrid with regular onsite presence, requiring on-site work at least part of the week and strong emphasis on performance, cost

Qualifikationen

• Advanced proficiency in Python/Scala/SQL for data processing.
• Experience with ETL/ELT and data handling.
• Spark and Databricks programming experience.
• Experience designing and building CI/CD pipelines (Azure with GitHub Actions).
• Experience with logging, monitoring, and alerting systems.
• Familiarity with AI tools to boost productivity.

Aufgaben

• Design and build scalable data/app solutions using Python and Azure ML.
• Develop ETL/ELT processes on large-scale datasets.
• Optimize pipelines for performance and cost.
• Collaborate with data scientists, analysts, and stakeholders.
• Implement data security, governance, and Unity Catalog access controls.
• Champion CI/CD pipelines to automate deployment and management.
• Design CI/CD pipelines in Azure with GitHub Actions.
• Develop monitoring, logging, and alerting for production pipelines.

Kenntnisse

Python

Scala

SQL

ETL/ELT

CI/CD

Azure

Logging/Monitoring

AI tools

Tools

Spark

Databricks

Azure DevOps

GitHub Actions

Jenkins

Jobbeschreibung

We are seeking a Senior Data Engineer to design and build scalable data and application solutions using Python, Spark, Databricks, and Azure. This role develops complex ETL/ELT processes for large-scale datasets, optimizes pipelines for performance and cost, and partners with data scientists, analysts, and business stakeholders to turn complex business problems into technical solutions. This is a hybrid position with a regular onsite presence.

Key Responsibilities

• Contribute to end-to-end design and implementation of scalable data and application solutions using Python and Azure ML
• Develop and implement complex ETL/ELT processes to ingest and transform large-scale datasets from diverse sources using Spark and Python
• Optimize Python jobs for performance and cost-efficiency using techniques such as partitioning, caching, cluster configuration tuning, and bottleneck troubleshooting
• Collaborate with data scientists, analysts, and business stakeholders to translate complex business problems into technical solutions
• Implement data security strategies, compliance standards, and governance frameworks such as Unity Catalog, including access controls, encryption, and data lineage
• Champion code quality and implement CI/CD pipelines using tools such as Azure DevOps, GitHub Actions, or Jenkins to automate deployment and management of data assets and infrastructure
• Design CI/CD pipelines that promote code across development and production environments, preferably in Azure using GitHub Actions
• Develop robust monitoring, logging, and alerting for production-grade pipelines to proactively identify and resolve issues

Required Qualifications

• Advanced proficiency in Python, Scala, and/or SQL for data processing and application development
• Experience with ETL and data handling
• Spark and Databricks programming experience
• Strong application development skills and experience across the software development lifecycle
• Experience designing and building CI/CD pipelines, preferably to Azure using GitHub Actions
• Experience with logging, monitoring, and alerting systems
• Experience using AI tools to improve productivity

Preferred Qualifications

• Java and Spring experience
• Strong knowledge of a broad range of Azure services