Data Engineer

Super Bank Indonesia; Kebayoran Baru; On-site; IDR 250,000,000 - 420,000,000; Full time

Job summary

Super Bank Indonesia seeks a data engineer in Jakarta to build and maintain a state-of-the-art data lifecycle platform, including ingestion, storage, processing and consumption channels.

You will work with data scientists, product managers, legal and compliance across SEA to tailor offerings and adopt scalable architectures for real-time and batch workloads.

Qualifications

• 2+ years developing scalable, secure big data platforms.
• Experience with Linux and cloud platforms (AWS/Azure).
• Hands-on with open source data technologies and data pipelines.

Responsibilities

• Build and manage data assets using scalable big data technologies.
• Design and deliver data lifecycle tooling for real-time and batch use-cases.
• Expose metadata catalog for data lake exploration and lineage.
• Enable ML model testing and productionization with data support.
• Lead technical discussions, RFCs, and architecture reviews.
• Apply software engineering concepts to roadmaps and security.

Skills

Java

Python

Scala

Big Data concepts

Security/compliance awareness

Tools

Airflow

Spark

Kafka

Kubernetes

HDFS/Yarn

Elasticsearch

Presto/Trino

DBT

Job description

As a data engineer, you will be working on all aspects of data, from platform and infra build out to pipeline engineering and writing tooling/services for augmenting and fronting the core platform.

You will be responsible for building and maintaining the state-of-the-art data lifecycle management platform, including acquisition, storage, processing and consumption channels.

The team works closely with data scientists, product managers, legal, compliance and business stakeholders across the SEA in understanding and tailoring the offerings to their needs.

As a member of the data organization, you will be an early adopter and contributor to various open source big data technologies and you are encouraged to think out of the box and have fun exploring the latest patterns and designs in the fields of software and data engineering.

Work responsibilities

Build and manage the data asset using some of the most scalable and resilient open source big data technologies like Airflow, Spark, DBT, Kafka,Yarn/Kubernetes, ElasticSearch, Presto/Dremio, Visualization layer and more.

Design and deliver the next-gen data lifecycle management suite of tools/frameworks, including ingestion and consumption on the top of the data lake to support real-time, API-based and serverless use-cases, along with batch (mini/micro) as relevant

Build and expose metadata catalog for the Data Lake for easy exploration, profiling as well as lineage requirements

Enable Data Science teams to test and productionize various ML models, including propensity, risk and fraud models to better understand, serve and protect our customers

Lead and/or participate in technical discussions across the organization through collaboration, including running RFC and architecture review sessions, tech talks on new technologies as well as retrospectives

Apply core software engineering and design concepts in creating operational as well as strategic technical roadmaps for business problems that are vague/not fully understood

Obsess over security by ensuring all the components, from a platform, frameworks to the applications are fully secure and are compliant by the group’s infosec policies.

Job requirements

At least 2+ years of relevant experience in developing scalable, secured, fault tolerant, resilient & mission-critical big data platforms.

Able to maintain and monitor the ecosystem with high availability

• Must have sound understanding for all

Big Data components & Administration

Hands-on in building a complete data platform using various open source technologies.

Must have good fundamental hands-on knowledge of Linux and building a big data stack on top of AWS/Azure using Kubernetes.

Strong understanding of big data and related technologies like Spark, Presto, Airflow, HDFS Yarn etc.

Good knowledge of Complex Event Processing (CEP) systems like Spark Streaming, Kafka, Apache Flink, Beam etc.

Experience with NoSQL databases – KV/Document/Graph and similar Proven ability to contribute to the open source community and up-to-date with the latest trends in the big data space.

Able to drive best practices like CI/CD, containerization, blue-green deployments, 12-factor apps, secrets management etc in the Data ecosystem.

Able to develop an agile platform with auto scale capability up & down as well vertically and horizontally.

Must be in a position to create a monitoring ecosystem for all the components in use in the data ecosystem.

Proficiency in at least one of the programming languages Java, Scala or Python along with a fair understanding of runtime complexities.

Must have the knowledge to build Data metadata, lineage and discoverability from scratch.

Leadership competency