Big Data Specialist

Sky Systems, Inc. (SkySys); Hyderabad; Hybrid; INR 1,500,000 - 2,100,000; Full time

Job summary

Sky Systems, Inc. (SkySys) seeks a hands-on Senior Big Data Engineer to architect, build, and support large-scale AWS data platforms from Hyderabad.

The role requires expert Python and SQL with Spark/PySpark, and experience with lakehouse architectures and production-grade data engineering. Responsibilities include designing costly and scalable batch/CDC pipelines, streaming integrations, and infrastructure automation using IaC.

Qualifications

• Expert Python and strong SQL skills.
• Deep Apache Spark/PySpark experience.
• Hands-on AWS data services experience.
• Experience with batch and CDC ingestion and data lakes/lakehouses.
• Infrastructure as Code and CI/CD experience.
• Ability to lead design discussions and review code.
• Experience with multi-terabyte data workloads.

Responsibilities

• Design, develop, and optimize large-scale batch and CDC data pipelines.
• Build and operate AWS data platforms using Glue, EMR, S3, Athena, Redshift.
• Develop streaming and event-driven integrations using Lambda, SQS, SNS, Kinesis, MSK.
• Optimize Spark/PySpark performance, partitioning, and memory usage.
• Implement lakehouse architectures using Parquet/Delta formats.
• Design dimensional models and ensure data quality, observability.
• Implement IaC and CI/CD with Terraform, CDK, CloudFormation.
• Lead technical design discussions and code reviews while coding in production.
• Process and support data workloads at multi-terabyte scale.

Skills

Python

SQL

Data modeling

Production troubleshooting

Cost optimization

Tools

AWS Glue

AWS EMR

AWS S3

Amazon Athena

Amazon Redshift

Spark/PySpark

Lambda

SQS

SNS

Kinesis

MSK

Terraform

CDK

CloudFormation

CI/CD

Git

Docker

Job description

Position Type: Full-Time Contract (40hrs/week)

Contract Duration: 6 months+

Work Schedule: 8 hours/day (Mon-Fri)

Location: Hyderabad, Hybrid(Wed and Thursday to be onsite)

We are seeking a highly hands-on Senior Big Data Engineer to architect, build, optimize, and support large-scale AWS data platforms. The ideal candidate combines expert Python and strong SQL skills with deep experience in Spark/PySpark, AWS data services, lakehouse architecture, and production-grade data engineering.

Key Responsibilities

• Design, develop, and optimize large-scale batch and CDC data pipelines.
• Build and operate data platforms using AWS Glue, EMR, S3, Athena, Redshift/Spectrum, and Step Functions.
• Develop streaming and event-driven integrations using Lambda, SQS, SNS, Kinesis, and MSK.
• Optimize Spark/PySpark execution, partitioning, data shuffling, memory utilization, and processing costs.
• Implement lakehouse and medallion architectures using formats such as Parquet, ORC, Avro, Iceberg, Hudi, or Delta Lake.
• Design dimensional data models and maintain data quality, reliability, and observability.
• Implement infrastructure and deployment automation using Terraform, CDK, CloudFormation, and CI/CD.
• Lead technical design discussions and code reviews while remaining actively involved in coding and production delivery.
• Process and support data workloads at multi-terabyte scale.

Requirements

• Expert Python and strong SQL.
• Deep Apache Spark/PySpark and AWS data engineering experience.
• Hands-on experience with AWS Glue, EMR, S3, Athena, Redshift/Spectrum, and orchestration services.
• Experience with batch and CDC ingestion, data lakes/lakehouses, and dimensional modeling.
• Strong production troubleshooting, performance tuning, and cost optimization skills.
• Experience with Infrastructure as Code, CI/CD, Git, and Docker.