Sr. Data Engineer

NCS Group; Taguig; Hybrid; PHP 1,200,000 - 2,400,000; Full time

Job summary

NCS Group in the Philippines seeks a Senior Data Engineer to design and maintain automated data pipelines using Spark, Kafka, Airflow, and cloud platforms.

You will collaborate with the Data Engineering Lead to implement end-to-end ETL, ensure data quality, and support governance while operating in a hybrid setup at BGC Taguig.

Qualifications

• Total Data Engineering experience - 7-8yrs and above.
• Skilled in Python (at least 4/5 rating), SQL (at least 3/5 rating).
• Working with modern data cloud platforms (Databricks/Snowflake).
• Exposure to cloud services (preferably Azure or AWS).
• Willing to work on a hybrid work setup in BGC Taguig.

Responsibilities

• Develops automated data pipelines based on recommended solutions by Data Engineering Lead using ETL tools.
• Creates automated data pipeline design documentation and technical design specifications.
• Provides technical expertise, troubleshoots, and supports during design, development, testing, production and post-production.
• Implements end-to-end complex automated data pipeline systems using Big Data tools such as Spark, Kafka, Airflow, and Hadoop.

Skills

Python

SQL

Data integration

Analytical thinking

Tools

Databricks

Snowflake

Azure

AWS

Apache Spark

Airflow

Kafka

Talend

Hadoop

Job description

NCS is a leading AI Tech Services company. With a 15,000-strong team across the Asia Pacific, NCS scales its platforms and capabilities to provide clients with greater agility and AI expertise across a range of industries. Embracing a strong ecosystem of global partners, NCS transforms technology services delivery combining AI with digital resilience to drive real business impact. NCS is a subsidiary of the Singtel Group.

Duties & Responsibilities

• Develops automated data pipelines based on recommended solution by Data Engineering Lead using official ETL tools determined by Data Engineering Head
• Creates automated data pipeline design documentation, Source-to-Target spreadsheets, and technical design specifications.
• Provides technical expertise, troubleshoots, guides and supports during design, development, testing, production and post-production.
• Understands the source system, captures and documents business rules.
• Analyzes and proposes changes to maximize code reusability and maintain business value.
• Develops, implements and maintains highly efficient and scalable automated data pipeline processes.
• Provides expert technical knowledge and solutions pertaining to Data integration.
• Identify opportunities to optimize the automated data pipeline including environment setup, implement monitoring, quality and validation processes to ensure data accuracy and integrity.
• Provides inputs into data governance organization to drive the data governance charter.
• Ensures compliance to internal standards and guidelines.
• Conducts unit testing and component integration testing for solutions developed
• Reviews business and technical requirements and ensures the data integration platform meets requirements.
• Applies industry best practices for automated data pipeline design and development.
• Implementation of an end-to-end complex automated data pipeline system using common Big Data Tools such as (but not limited to) Apache Spark, Talend, Kafka, Airflow, NiFi i and Hadoop Tools.
• Implements monitoring and measurement of data quality as defined by business rules.
• Ensures adherence to architectural governance standards and practices.
• Develops best practices, standards of excellence, and guidelines for programming teams.
• Ensures compliance to the DevSecOps practice.
• Conducts System Testing - execute job flows, investigate system defects, resolve defects and document results
• Provides a level of effort estimates for new initiatives and change controls in order for projects to be evaluated, budgeted and tracked for success.

Minimum Qualifications

• Total Data Engineering experience - 7- 8yrs and above
• Skilled in Python (at least 4/5 rating), SQL (at least 3/5 rating)
• Working or has experience working on modern data cloud platforms (Databricks/Snowflake)
• With exposure to cloud services (preferably Azure or AWS)
• Willing to work on a hybrid work set up in BGC Taguig