Data Engineer-Data Platforms-AWS

IBM; Chennai District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

IBM Consulting in India is seeking a Data Engineer specializing in data platforms on AWS. You will design, build, and operate batch and real-time data pipelines, contributing to data warehouses and data lakes across enterprise workloads.

The role emphasizes scalable, reliable data engineering within a global client context. The role requires hands-on experience with AWS services (EMR, Glue, Kinesis), Open Source tooling (Airflow, dbt), and SQL/Python, with a focus on delivering end-to-end data

Qualifications

• Experience with AWS services (EMR, Glue, Glue Catalog, Kinesis) to design and run batch and real-time data pipelines.
• Data pipelines for Data Warehouse and Data Lake using Kinesis and MSK (Kafka).
• Create data layers on RedShift, Aurora, and DynamoDB; migrate data with DMS.
• Open source tooling on AWS: Airflow, dbt, Spark (Python/Scala) on AWS platform.
• Data service management on the AWS Platform to ensure seamless integration.

Responsibilities

• Design, build, and operate batch and real-time data pipelines using AWS services.
• Create data layers on RedShift, Aurora, and DynamoDB; migrate data with DMS.
• Develop pipelines for Data Warehouse and Data Lake using Kinesis and Kafka.
• Manage data services on the AWS Platform for seamless integration.
• Apply open source technologies like Airflow and dbt, Spark with Python/Scala on AWS.

Skills

Data engineering

SQL

Python

Spark

Scala

AWS ecosystem

Education

Master's Degree

Tools

AWS EMR

AWS Glue

Glue Catalog

Kinesis

RedShift

Aurora

DynamoDB

DMS

Airflow

dbt

Python

Spark

Scala

Lambda

Databrew

RedShift Spectrum

Job description

Introduction

A career in IBM Consulting is built on long-term client relationships and close collaboration worldwide. You’ll work with leading companies across industries, helping them shape their hybrid cloud and AI journeys. With support from our strategic partners, robust IBM technology, and Red Hat, you’ll have the tools to drive meaningful change and accelerate client impact. At IBM Consulting, curiosity fuels success. You’ll be encouraged to challenge the norm, explore new ideas, and create innovative solutions that deliver real results. Our culture of growth and empathy focuses on your long-term career development while valuing your unique skills and experiences.

Introduction

A career in IBM Consulting is built on long-term client relationships and close collaboration worldwide. You’ll work with leading companies across industries, helping them shape their hybrid cloud and AI journeys. With support from our strategic partners, robust IBM technology, and Red Hat, you’ll have the tools to drive meaningful change and accelerate client impact. At IBM Consulting, curiosity fuels success. You’ll be encouraged to challenge the norm, explore new ideas, and create innovative solutions that deliver real results. Our culture of growth and empathy focuses on your long-term career development while valuing your unique skills and experiences.

Your Role And Responsibilities

As a Data Engineer specializing in Data Platforms on AWS, you will advise on, develop, and maintain data engineering solutions on the AWS Cloud ecosystem. You will design, build, and operate batch and real-time data pipelines using various AWS services.

Your Primary Responsibilities Will Include

• Design and Develop Data Pipelines: Design, build, and operate batch and real-time data pipelines using AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis, ensuring seamless integration and operation of data engineering solutions.
• Create Data Layers: Create data layers on AWS RedShift, Aurora, and DynamoDB, and migrate data using AWS DMS.
• Develop Batch and Real-Time Pipelines: Develop batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka.
• Manage Data Services: Schedule and manage data services on the AWS Platform, ensuring seamless integration and operation of data engineering solutions.
• Apply Open Source Technologies: Utilize open source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on AWS Platform to support data engineering solutions.

Preferred Education

Master\'s Degree

Required Technical And Professional Expertise

• Exposure to AWS Toolset: Experience working with AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis to design, build, and operate batch and real-time data pipelines.
• Data Pipeline Development: Exposure to developing batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka.
• Data Layer Creation: Experience working with AWS RedShift, Aurora, and DynamoDB to create data layers and migrate data using AWS DMS.
• Open Source Technologies: Exposure to utilizing open source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on AWS Platform to support data engineering solutions.
• Data Service Management: Experience working with scheduling and managing data services on the AWS Platform to ensure seamless integration and operation of data engineering solutions.

Preferred Technical And Professional Experience

• Proficiency with AWS Databrew: Experience working with AWS Glue Databrew to support data engineering solutions, including data preparation and data quality.
• Knowledge of Lambda Functions: Exposure to using Lambda functions with Python to support data engineering solutions, including data processing and data integration.
• Familiarity with RedShift Spectrum: Experience working with RedShift Spectrum to support data engineering solutions, including data warehousing and data analytics.