Data & AI Solutions Specialist

UNISONEDGE CONSULTING PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

UNISONEDGE CONSULTING PTE. LTD. is seeking a senior data engineer to design, develop, and optimize scalable data engineering solutions using Big Data, cloud, and AI technologies.

You will build high-performance data pipelines, ingestion frameworks, and processing solutions while collaborating with architects, data scientists, and business stakeholders to deliver production-ready systems.

Qualifications

• Bachelor's or master's in computer science, IT, engineering, or related field.
• 8+ years in data engineering/big data with large-scale processing.
• Strong programming in Python, Java, or Scala.
• Hands-on Spark/PySpark, SQL, Hadoop, and Hive experience.
• Experience with cloud platforms, especially AWS.
• Experience with data warehouses, relational DBs, and data lake tech.
• Experience with Kafka, Airflow, Jenkins, Git and CI/CD.
• Docker, Kubernetes or OpenShift is advantageous.
• Knowledge of Databricks, Snowflake, Cloudera is advantageous.
• Exposure to generative AI, LLMs, RAG, LangChain, vector DBs is an advantage.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Develop and optimize ETL/ELT pipelines for large-scale processing.
• Build data ingestion solutions using databases, APIs, files, and streaming platforms.
• Work with AWS services (S3, Glue, EMR, Redshift, Kinesis, Lambda, DynamoDB).
• Develop and optimize data processing using Hadoop, Hive, Spark, Kafka, Cloudera, Databricks.
• Perform SQL and Spark performance tuning for large workloads.
• Design data models, warehouses, and lake solutions.
• Develop CI/CD pipelines and support containerized deployments (Docker, Kubernetes, OpenShift).
• Integrate generative AI and NLP into enterprise data apps as needed.
• Develop solutions with LLM frameworks, RAG, vector databases, and AI APIs.
• Collaborate with solution architects, data scientists, engineers, and stakeholders to deliver production-grade solutions.
• Participate in design, development, testing, deployment and production support.

Skills

Python

Java

Scala

SQL

Spark/PySpark

Hadoop

Hive

Kafka

Airflow

CI/CD

Docker

Kubernetes

OpenShift

Databricks

Snowflake

Cloudera

AWS

Education

Bachelor's degree in CS/IT/Engineering or related field

Master's degree in CS/IT/Engineering or related field

Tools

Apache Spark/PySpark

Hadoop

Hive

Databricks

Cloudera

Snowflake

Docker

Kubernetes

OpenShift

Airflow

Jenkins

Git

AWS

S3

Glue

EMR

Redshift

Kinesis

Job description

Job Description

We are looking for a highly skilled and experienced senior data engineer to design, develop, and optimize scalable data engineering solutions using Big Data, cloud, and modern AI technologies. The successful candidate will be responsible for developing high-performance data pipelines, data ingestion frameworks, and data processing solutions while working closely with architects, business stakeholders, data scientists, and engineering teams.

Key Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using Apache Spark/PySpark, SQL, and Python.
• Develop and optimize ETL/ELT pipelines for large-scale data processing.
• Build data ingestion solutions using databases, APIs, files, and streaming platforms.
• Work with AWS services including S3, Glue, EMR, Redshift, Kinesis, Lambda, and DynamoDB.
• Develop and optimize data processing solutions using Hadoop, Hive, Spark, Kafka, Cloudera, and Databricks.
• Perform SQL and Spark performance tuning and optimize large-scale data processing workloads.
• Design and implement data models, data warehouses, and data lake solutions.
• Develop CI/CD pipelines and support containerized deployments using Docker, Kubernetes, and OpenShift.
• Integrate generative AI and NLP capabilities into enterprise data applications where required.
• Develop solutions using LLM frameworks, RAG, vector databases, and AI APIs.
• Collaborate with solution architects, data scientists, software engineers, and business stakeholders to deliver production-ready solutions.
• Participate in system design, development, testing, deployment, and production support.

Requirements

• Bachelor’s or master's degree in computer science, information technology, engineering, or a related field.
• Minimum 8 years of relevant experience in data engineering/big data, with strong experience in large-scale data processing.
• Strong programming experience in Python, Java, and/or Scala.
• Strong hands‑on experience with Apache Spark/PySpark, SQL, Hadoop, and Hive.
• Experience with cloud platforms, particularly AWS.
• Experience with data warehouses, relational databases, and data lake technologies.
• Experience with Kafka, Airflow, Jenkins, Git, and CI/CD practices.
• Experience with Docker, Kubernetes, or OpenShift is advantageous.
• Knowledge of Databricks, Snowflake, and Cloudera is advantageous.
• Exposure to generative AI, LLMs, RAG, LangChain/LangGraph, and vector databases is an advantage.
• Strong analytical, problem‑solving, and communication skills.

Technical Skills

Python, Java, Scala, PySpark, Apache Spark, SQL, Hadoop, Hive, Kafka, Databricks, Cloudera, AWS, S3, Glue, EMR, Redshift, Kinesis, Lambda, Airflow, Jenkins, Docker, Kubernetes, OpenShift, Snowflake, MongoDB, Oracle, PostgreSQL, and generative AI technologies.