Big Data Developer

Sky Systems, Inc. (SkySys); Hyderabad; Hybrid; INR 1,200,000 - 2,400,000; Full time

Job summary

Sky Systems, Inc. (SkySys) in Hyderabad, India is seeking a Mid-Level Big Data Engineer with 5+ years of experience to design and deliver scalable data solutions on AWS.

The role focuses on batch and streaming pipelines using Scala, Python, Spark or PySpark, and requires strong Linux and shell scripting skills. You will work with Hadoop ecosystem tools, Kafka or similar streaming tech, and a suite of AWS services to build data platforms, APIs, and automation, while collaborating with

Qualifications

• 5+ years of experience in Big Data development and data/software engineering.
• Strong hands-on experience with Scala and Python.
• Strong experience with Apache Spark / PySpark.
• Experience with the Hadoop ecosystem, including HDFS, MapReduce, and Hive.
• Experience with Kafka or similar streaming technologies.
• Strong hands-on AWS experience, particularly with EMR, S3, EC2, ECS/EKS, MWAA/Airflow, Step Functions, API Gateway, Lambda, DynamoDB, and/or RDS/Aurora.
• Experience building large-scale batch and API-based data-processing solutions.
• Strong Linux and shell scripting skills.

Responsibilities

• Design, develop, test, and deploy scalable Big Data solutions on AWS.
• Build and maintain batch and streaming data pipelines using Scala, Python, Spark, and PySpark.
• Process and transform large volumes of structured and unstructured data.
• Develop and support distributed data-processing and data-ingestion platforms.
• Integrate new data sources and technologies into existing Big Data ecosystems.
• Optimize Spark, Hadoop, EMR, and distributed workloads.
• Build cloud-native APIs and microservices supporting data platforms.
• Automate data workflows, deployments, monitoring, and operational processes.
• Collaborate with architects, product owners, QA teams, and business stakeholders.
• Apply security, governance, reliability, and software engineering best practices.

Skills

Scala

Python

Spark

PySpark

Hadoop

Kafka

AWS

Linux

Shell scripting

Tools

EMR

S3

EC2

ECS/EKS

MWAA/Airflow

Step Functions

API Gateway

Lambda

DynamoDB

RDS/Aurora

Job description

Position Type: Full-Time Contract (40hrs/week)

Contract Duration: 6 months + extendable

Work Schedule: 8 hours/day (Mon-Fri)

Location: Hybrid - Wed/Thursday to be in onsite in Hyderabad, India

We are seeking a Mid-Level Big Data Engineer with 5+ years of experience developing scalable Big Data solutions and distributed data-processing applications in AWS environments.

The ideal candidate will have strong hands-on experience with Scala, Python, Apache Spark/PySpark, AWS, Linux, and shell scripting, along with experience building batch and streaming data pipelines and optimizing large-scale distributed workloads.

Key Responsibilities

• Design, develop, test, and deploy scalable Big Data solutions on AWS.
• Build and maintain batch and streaming data pipelines using Scala, Python, Spark, and PySpark.
• Process and transform large volumes of structured and unstructured data.
• Develop and support distributed data-processing and data-ingestion platforms.
• Integrate new data sources and technologies into existing Big Data ecosystems.
• Optimize Spark, Hadoop, EMR, and distributed computing workloads.
• Build cloud-native APIs and microservices supporting data platforms.
• Automate data workflows, deployments, monitoring, and operational processes.
• Collaborate with architects, product owners, QA teams, and business stakeholders.
• Apply security, governance, reliability, and software engineering best practices.

Requirements

• 5+ years of experience in Big Data development and data/software engineering.
• Strong hands-on experience with Scala and Python.
• Strong experience with Apache Spark / PySpark.
• Experience with the Hadoop ecosystem, including HDFS, MapReduce, and Hive.
• Experience with Kafka or similar streaming technologies.
• Strong hands-on AWS experience, particularly with EMR, S3, EC2, ECS/EKS, MWAA/Airflow, Step Functions, API Gateway, Lambda, DynamoDB, and/or RDS/Aurora.
• Experience building large-scale batch and API-based data-processing solutions.
• Strong Linux and shell scripting skills.
• Experience with data ingestion, transformation, data modeling, schema design, and data lifecycle management.
• Strong understanding of performance tuning for Spark, Hadoop, EMR, and distributed systems.
• Experience working in Agile/Scrum environments.
• Strong troubleshooting and problem-solving skills.

Preferred

• LLMs, RAG, Vector Databases, and AI/ML integrations
• AI-enabled data platforms
• Automated testing and data quality tools