Data Engineer 2/3

Manatal; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Manatal is seeking a skilled Data Engineer located in Bengaluru, Karnataka. The ideal candidate will have 5-7 years of experience in Big Data technologies, with strong proficiency in Hadoop, Spark, and AWS services. Responsibilities include designing data pipelines, developing automation processes, and collaborating with cross-functional teams to address data-related issues. Join a dynamic environment where your skills will drive impactful data infrastructure initiatives.

Qualifications

• 5-7 years of experience working with Big Data technologies.
• Extensive experience in developing and maintaining batch and real-time data pipelines.
• Hands-on experience in integrating data from multiple sources.

Responsibilities

• Design and implement scalable data pipelines.
• Develop and automate data pipelines using AWS services.
• Collaborate with stakeholders to address data infrastructure needs.

Skills

Hadoop

Spark

Python Scripting

Java

AWS Services

SQL

Kafka

Data Warehousing

Shell Scripting

Data Modeling

Education

BE/B.Tech in Computer Science/IT

Tools

SparkSQL

HiveQL

Kinesis

Druid

Aerospike

Flink

Job description

*Games24x7 is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, disability status, or any other characteristic protected by the law.*

General Accountabilities/Job Responsibilities

• Understand analytical requirement and design data pipelines around it
• Develop, test and maintain optimal and scalable end-to-end data pipelines for Batch as well as Real Time data processing.
• Leverage open source / AWS/Databricks infrastructure/services for creation and automation of data pipelines
• Work with stakeholders including the Executive, Product, Data and Development teams to assist with data-related technical issues and support their data infrastructure needs.
• Participate actively in data-marts design discussions
• Write code (queries/scripts) in Spark / Hive / Athena, etc that is both functional and elegant, following appropriate design patterns
• Build integrations for data ingestion across various types of data stores
• Leverage data APIs provided by partner platforms to fetch data on a regular basis
• Identify data quality issues and write data cleanup jobs
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metric
• Create and maintain documentation of entire data landscape.

Mandatory Requirements

• BE/B.Tech in Computer Science/IT
• 5 - 7 years experience working in Big Data technologies.
• Extensive experience in end-to-end development and maintenance of batch data pipeline and near real time Streaming data pipeline with single digit second of latency.
• Hands-on experience in Hadoop, Spark, Presto, Hive, Sqoop,MapReduce., etc
• Hands-on experience of Java and Python Scripting.
• Hands-on experience in SparkSQL, HiveQL and SQL.
• Experience with integration of data from multiple data sources (Kafka, MongoDB, Mysql, Cassandra, 3rd Party APIs etc)
• Hands-on experience on messaging queues like Kafka and Kinesis.
• Working experience on KsqlDB, Druid, Aerospike,Flink etc.
• Knowledge of Linux and shell scripting
• Should have knowledge of AWS services such as S3, EC2, EMR, Athena, Glue, Redshift etc.
• Deep understanding of the Hadoop ecosystem and strong conceptual knowledge in Hadoop architecture components
• Strong experience with Data warehousing and Data modeling
• Capable of processing large sets of structured, semi-structured and unstructured data.
• Hands-on experience in Sqoop/Spark for importing data from RDBMS to HDFS and vice-versa.
• Understand the business requirements and build the Big Data Lake based on Big Data technologies.
• Experience in end-to-end design and build process of Real Time Pipelines(Sub Second latency) will be preferred.
• Developing the data lake for real time data streaming from multiple data sources.
• Working on and leading Proof of Concept (PoC) projects in the big data space.
• Designing data pipelines to process any size and any file format.