Sr. Data Engineer

Insight Global; Hyderabad; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Insight Global in Hyderabad, India, seeks a senior data engineer to design and develop real-time data pipelines using Kafka, Spark, and AWS Kinesis. You will build scalable data processing solutions on AWS and optimize large-scale distributed systems.

Responsibilities include troubleshooting production data platforms, supporting streaming applications, and collaborating with cross-functional teams on enterprise data integration initiatives.

Qualifications

• 10+ years of experience in Data/Software Engineering or related fields.

Responsibilities

• Design and develop real-time data pipelines using Kafka, Spark, AWS Kinesis, and related technologies.

Skills

AWS Cloud Services

Apache Spark

PySpark

Scala

Java

Python

Kafka

Kinesis

Flume

Hadoop Ecosystem

Big Data

Distributed Systems

Tools

Kafka

Flume

Kinesis

Kinesis Firehose

Job description

Responsibilities

• Design and develop real-time data pipelines using Kafka, Spark, AWS Kinesis, and related technologies.
• Build and support scalable data processing solutions on AWS.
• Develop and optimize large-scale distributed data processing systems.
• Troubleshoot and support production data platforms and streaming applications.
• Collaborate with cross-functional teams on enterprise data integration initiatives.

Required Skills & Experience

• 10+ years of experience in Data Engineering, Software Engineering, or related fields.
• Strong expertise in AWS Cloud Services and cloud-native architectures.
• Hands-on experience with Apache Spark (Scala & PySpark).
• Strong knowledge of the Hadoop Ecosystem and distributed computing.
• Experience with Kafka, streaming technologies, and real-time data pipelines.
• Experience with AWS Kinesis, Kinesis Firehose, Flume, or similar ingestion platforms.
• Strong programming skills in Python, Scala, and Java.
• Experience with Big Data, Enterprise Data Warehousing, and Distributed Systems.