Data Engineer

Pt Swapro International; Daerah Khusus Ibukota Jakarta; On-site; IDR 200.000.000 - 300.000.000; Full time

Job summary

A data management firm in Jakarta is seeking a skilled individual to develop and manage data pipelines while ensuring compliance with regulations. Candidates should have experience in SQL, Python, and ETL tools, and familiarity with cloud platforms is a plus. The role includes monitoring performance, troubleshooting incidents, and creating technical documentation.

Responsibilities

• Develop and manage data pipeline & data cleansing.
• Monitor and improve data pipeline and operational processes.
• Fulfill service requests and troubleshoot incidents.
• Create and manage technical documentation for data pipeline.
• Ensure compliance with regulatory requirements.

Skills

SQL programming

Python programming

ETL Tools (SSIS, Pentaho, Talend, Airflow)

Database management (SQL Server, PostgreSQL, Oracle, MySQL)

Machine learning

Cloud GCP (Bigquery, Dataflow, Data Fusion)

Cloud AWS

Streaming Pipeline (Kafka, RabbitMQ, Pub Sub, CDC, Debezium)

dbt

Prefect

Airflow

Job description

Jobdesc:

Develop and manage data pipeline & data cleansing : Develop and manage design and flow of data storage, collection, processing and cleansing of data in accordance with the existing data warehouse architecture so that data can be analyzed and used by users

Monitoring & improvement data pipeline : Routinely monitor, manage data pipeline and schedule so that operational processes run optimally in terms of performance and cost

Service request fulfillment & troubleshooting incident or case : Perform request fulfillment and troubleshooting of incidents that occur in the scope of work in accordance with the SLA and create PICAPA documentations

Documentation Create and manage technical documentation (Technical Specification Document) data Pipeline, Schedule and Data Catalog to ensure the documentation can be used as a standardization reference.

Ensure compliance with regulatory requirements : Establish and maintain every process to ensure compliance with privacy policies and data security.Develop and implement procedures (SOP) to ensure data retention and disposal requirements are in accordance with applicable agreements.

Qualification:

• Berpengalaman dalam penggunaan Bahasa pemrograman SQL dan Python.
• Berpengalaman dalam ETL Tools (SSIS, Pentaho, Talend, Airflow)
• Berpengalaman dalam penggunaan Database (SQL Server, PostgreSQL, Oracle, Mysql)
• Berpengalaman dalam Machine learning.
• Cloud GCP (Bigquery, Dataflow, Data Fusion), Cloud AWS.
• Streaming Pipeline (Kafka, RabbitMq, Pub Sub, CDC, Debezium).
• dbt, prefect, airflow.