Senior Data Engineer — Real-Time Pipelines & ML (Azure)

JobCubby; Hinoba-an; On-site; PHP 1,200,000 - 2,400,000; Full time

Job summary

JobCubby is seeking a Senior Product Software Data Engineer to design, develop, and troubleshoot data-driven software. You will collaborate with cross-functional teams to translate business needs into scalable data solutions and enterprise-grade pipelines.

You will build real-time and batch data pipelines with PySpark, Spark, Azure Databricks, and related Azure services; mentor junior engineers; ensure data quality, governance, and documentation; partner with data scientists to operationalize ML

Qualifications

• Experience building data pipelines for real-time and batch processing.
• 2+ years building data pipelines for structured and unstructured data.
• 3+ years total experience in data engineering.
• Strong knowledge of PySpark, Hadoop, Kafka, and related big data tools.
• Experience with SQL Server data warehousing concepts.
• Experience with Azure Data Factory, Azure Data Lake, Azure SQL DB, and Azure Synapse.
• Good to have Azure ML, Python, and Scala.
• Strong communication and collaboration skills.

Responsibilities

• Collaborate with cross-functional teams to translate requirements into scalable data solutions.
• Design and develop optimized, reliable, and secure enterprise-level data pipelines for real-time and batch processing.
• Analyze existing business processes, identify gaps, and recommend data-driven improvements.
• Build modern real-time data solutions using Scala, Apache Spark, and Azure Databricks.
• Partner with data scientists and business stakeholders to operationalize machine learning models.
• Champion data governance, metadata management, and operational excellence across data platforms.
• Ensure data quality and integrity by developing validation frameworks and monitoring mechanisms.
• Adhere to coding standards, participate in peer code reviews, and contribute to technical documentation.
• Mentor junior engineers and contribute to knowledge sharing across the team.

Skills

Software Engineering

Software Development

Data Pipelines

PySpark

Azure

SQL / Relational DB

Scala

Machine Learning concepts

Communication

Mentoring

Education

Bachelor’s degree in Computer Science or related field

Tools

Azure Data Factory

Azure Databricks

Azure Data Lake

Azure Synapse Analytics

Power BI

Git

Job description

JobCubby is seeking a Senior Product Software Data Engineer to design, develop, and troubleshoot data-driven software. You will collaborate with cross-functional teams to translate business needs into scalable data solutions and enterprise-grade pipelines.

You will build real-time and batch data pipelines with PySpark, Spark, Azure Databricks, and related Azure services; mentor junior engineers; ensure data quality, governance, and documentation; partner with data scientists to operationalize ML