Data Engineer-Data Platforms

IBM Computing; Pune District; On-site; INR 1,500,000 - 2,000,000; Full time

Job summary

A leading global technology company is looking for a Big Data Engineer in Pune. You'll develop and maintain big data solutions, focusing on data engineering activities like building data pipelines and optimizing data models. The ideal candidate has over 5 years of experience with technologies such as Hadoop, Spark, and Python. This role requires strong technical skills and the ability to manage data infrastructure to support growing business needs.

Qualifications

• 5+ years of experience in Big Data technologies.
• Experience with Hadoop, Spark, Scala, and Python.
• Good understanding of data models and ETL processes.

Responsibilities

• Develop, maintain, evaluate, and test big data solutions.
• Design and optimize existing data models based on business requirements.
• Build and manage data infrastructure.

Skills

Big Data - Hadoop

Spark

Scala

Python

ETL processes

Tools

Hbase

Hive

AWS S3

Jenkins

GIT

Job description

Introduction

In this role, you\'ll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers), where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology.

Your role and responsibilities

• As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities like creating pipelines/workflows for Source to Target and implementing solutions that tackle the clients needs.
• Your primary responsibilities include:
• Design, build, optimize and support new and existing data models and ETL processes based on our clients business requirements.
• Build, deploy and manage data infrastructure that can adequately handle the needs of a rapidly growing data driven organization.
• Coordinate data access and security to enable data scientists and analysts to easily access to data whenever they need too.

Required technical and professional expertise

• Must have 5+ years exp in Big Data -Hadoop Spark -Scala ,Python
• Hbase, Hive Good to have Aws -S3,
• athena ,Dynomo DB, Lambda, Jenkins GIT
• Developed Python and pyspark programs for data analysis.. Good working experience with python to develop Custom Framework for generating of rules (just like rules engine).
• Developed Python code to gather the data from HBase and designs the solution to implement using Pyspark. Apache Spark DataFrames/RDD's were used to apply business transformations and utilized Hive Context objects to perform read/write operations.

Preferred technical and professional experience

• Understanding of Devops.
• Experience in building scalable end-to-end data ingestion and processing solutions
• Experience with object-oriented and/or functional programming languages, such as Python, Java and Scala

IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, caste, genetics, pregnancy, disability, neurodivergence, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.