Senior Data Engineer: Big Data & AI Cloud Pipelines

JEET ANALYTICS PTE. LTD.; Singapore; Hybrid; SGD 180,000 - 240,000; Full time

Job summary

JEET ANALYTICS PTE. LTD. in Singapore seeks an experienced Senior Data Engineer to design, build and optimize scalable data pipelines using Spark, PySpark, SQL and Python.

You will work with AWS services to ingest, process and store data, and will implement data lake and warehouse solutions while leveraging CI/CD pipelines with Docker and Kubernetes. The role requires 8+ years in Data Engineering, strong programming with Python/Java/Scala, and hands-on experience with Hadoop, Hive, Kafka and

Qualifications

• Bachelor’s or Master’s degree in CS/IT/Engineering or related field.
• 8+ years in Data Engineering/Big Data with large-scale processing.
• Strong programming in Python, Java and/or Scala.
• Experience with Spark/PySpark, SQL, Hadoop and Hive.
• Cloud experience, preferably AWS.

Responsibilities

• Design, develop and maintain scalable batch and real-time data pipelines.
• Build ETL/ELT pipelines for large-scale data processing.
• Create data ingestion solutions from databases, APIs, files and streaming.
• Work with AWS services including S3, Glue, EMR, Redshift, Kinesis, Lambda and DynamoDB.
• Develop data models, warehouses and data lake solutions.
• Implement CI/CD pipelines with Docker, Kubernetes and OpenShift.
• Integrate Generative AI, LLMs, RAG into enterprise data apps.
• Collaborate with architects, data scientists and stakeholders.

Skills

Python

Java

Scala

PySpark

Apache Spark

SQL

Hadoop

Hive

Kafka

Databricks

Cloudera

AWS

S3

Glue

EMR

Redshift

Kinesis

Lambda

Airflow

Jenkins

Docker

Kubernetes

OpenShift

Snowflake

MongoDB

Oracle

PostgreSQL

Generative AI

Education

Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering or a related field

Tools

Databricks

Snowflake

Cloudera

Airflow

Jenkins

Git

Docker

Kubernetes

OpenShift

Job description

JEET ANALYTICS PTE. LTD. in Singapore seeks an experienced Senior Data Engineer to design, build and optimize scalable data pipelines using Spark, PySpark, SQL and Python.

You will work with AWS services to ingest, process and store data, and will implement data lake and warehouse solutions while leveraging CI/CD pipelines with Docker and Kubernetes. The role requires 8+ years in Data Engineering, strong programming with Python/Java/Scala, and hands-on experience with Hadoop, Hive, Kafka and