Data Engineer – Data Platforms (AWS)

HypTechie; Ernakulam; On-site; INR 1,500,000 - 2,800,000; Full time

Job summary

IBM Consulting Client Innovation Centers in Kochi, Kerala, India, is seeking a Data Engineer to design and develop scalable data platforms on AWS, leveraging Spark, Python/Scala, Hive/HBase, and NoSQL. You will build robust data pipelines ingesting diverse data sources and collaborate with cross-functional teams to drive data-driven solutions.

The role requires 3–5+ years of experience with Spark, Python/Scala, and AWS Cloud Data Platforms, plus strong SQL skills.

Qualifications

• Bachelor’s degree in a relevant field; Master’s preferred.
• 3–5+ years in Data Management and Engineering (DW, Data Lakes, Lakehouse).
• 4+ years hands-on with Big Data technologies, especially Spark with Python/Scala.
• At least 3 years of experience working on AWS Cloud Data Platforms.
• Hands-on with AWS EMR, AWS Glue, DataBricks, AWS Redshift, DynamoDB.

Responsibilities

• Design, develop, and maintain data pipelines ingesting data from multiple sources.
• Build scalable data solutions using Spark, Python/Scala, and SQL.
• Optimize streaming pipelines for real-time processing.
• Work with Hadoop and AWS ecosystem to handle large-scale data volumes.
• Collaborate with cross-functional teams to integrate data into business applications.
• Apply data governance and performance tuning best practices.

Skills

Spark

Python

Scala

SQL

AWS

NoSQL

Hive

Education

Bachelor’s Degree

Master’s Degree

Tools

AWS EMR

AWS Glue

Databricks

Redshift

DynamoDB

Kafka

Job description

Data Engineer – Data Platforms (AWS)

Kochi, Kerala, India | Consulting | Early Professional

Introduction

At IBM Consulting Client Innovation Centers, we combine deep technical expertise with industry knowledge to deliver innovative solutions for global public and private sector clients. As a Data Engineer, you will design and develop scalable data platforms, pipelines, and big data solutions leveraging AWS Cloud and Spark frameworks. This is an exciting opportunity to work with modern technologies while driving innovation and adoption of data-driven solutions.

Your Role and Responsibilities

• Design, develop, and maintain data pipelines to ingest, process, and transform structured and unstructured data from multiple sources (files, streams, databases).
• Build efficient, reusable, and scalable data solutions using Apache Spark, Python, PySpark, Scala, and Hive/HBase/NoSQL databases.
• Implement and optimize streaming pipelines for real-time data processing.
• Work with Hadoop and AWS ecosystem components (e.g., EMR, Glue, DataBricks, Redshift, DynamoDB) to handle large-scale data volumes.
• Collaborate with cross-functional teams to integrate data solutions into business applications.
• Apply best practices in data engineering, performance tuning, and data governance.

Required Education

• Bachelor’s Degree
• Preferred: Master’s Degree

Required Technical and Professional Expertise

• 3–5+ years of experience in Data Management and Engineering (DW, Data Lakes, Lakehouse).
• Minimum 4 years of hands‑on experience with Big Data technologies, especially Spark with Python/Scala.
• At least 3 years of experience working on AWS Cloud Data Platforms.
• Practical expertise with AWS EMR, AWS Glue, DataBricks, AWS Redshift, DynamoDB.
• Strong SQL skills for data analysis and optimization.

Preferred Technical and Professional Experience

• Certifications in AWS or DataBricks (or Cloudera Spark Certified Developer).
• Experience with Kafka, HDFS, or other modern cloud/big data technologies.

About Business Unit

IBM Consulting is IBM’s professional services business, combining strategy, technology, and operations expertise to deliver transformation at scale. We work with some of the world’s most innovative organizations, helping them leverage the power of data and technology responsibly to accelerate business growth and create long-term value.