Senior AWS Data Pipeline Engineer

Yash Technologies Private Limited; Bengaluru; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

YASH Technologies is seeking a Senior AWS Data Engineer with expertise in Python, PySpark, and AWS data services to design and build scalable ETL pipelines. This role focuses on developing robust data processing solutions and optimizing data workflows to support business needs.

The ideal candidate will design pipelines using AWS Glue, Lambda, Step Functions, and manage data storage in S3 with RDS (MySQL). Strong SQL skills and security best practices are required.

Qualifications

• 5+ years of data engineering / AWS development.
• Strong proficiency in Python.
• Hands-on experience with PySpark for large-scale data processing.
• Expertise in AWS services: Glue, Lambda, Step Functions, Athena, S3, RDS (MySQL).
• Strong knowledge of SQL and data querying.
• Familiarity with IAM roles and AWS security best practices.

Responsibilities

• Design and implement scalable data pipelines using AWS services.
• Develop ETL workflows using AWS Glue, PySpark, and Python.
• Build serverless data processing solutions using AWS Lambda.
• Orchestrate workflows using AWS Step Functions.
• Manage and optimize data storage in AWS S3.
• Work with relational databases like AWS RDS (MySQL).
• Ensure data quality, validation, and governance.
• Optimize performance and cost of data pipelines.
• Debug and troubleshoot data-related issues.
• Collaborate with cross-functional teams (Data Analysts, Architects, DevOps).

Skills

Python

PySpark

SQL

AWS

Tools

Docker

Kubernetes

CI/CD

Job description

Sr. Software Engineer - AWS Data Pipeline Job

YASH Technologies is a leading technology integrator specializing in helping clients reimagine operating models, enhance competitiveness, optimize costs, foster exceptional stakeholder experiences, and drive business transformation. At YASH, we’re a cluster of the brightest stars working with cutting-edge technologies. Our purpose is anchored in a single truth – bringing real positive changes in an increasingly virtual world and it drives us beyond generational gaps and disruptions of the future.

We are looking forward to hire AWS Data Pipeline Professionals in the following areas : AWS Data Developer (Python / PySpark)

Job Summary

We are seeking a highly skilled AWS Data Developer with strong expertise in Python, PySpark, and AWS data services to design and build scalable data processing solutions. The ideal candidate will be responsible for developing robust ETL pipelines, optimizing data workflows, and leveraging AWS services to support business.

Key Responsibilities

• Design and implement scalable data pipelines using AWS services
• Develop ETL workflows using AWS Glue, PySpark, and Python
• Build serverless data processing solutions using AWS Lambda
• Orchestrate workflows using AWS Step Functions
• Manage and optimize data storage in AWS S3
• Work with relational databases like AWS RDS (MySQL)
• Ensure data quality, validation, and governance
• Optimize performance and cost of data pipelines
• Debug and troubleshoot data-related issues
• Collaborate with cross-functional teams (Data Analysts, Architects, DevOps)

Required Skills & Experience

• 5+ years of experience in data engineering / AWS development
• Strong proficiency in Python
• Hands‑on experience with PySpark for large-scale data processing
• Expertise in AWS services: AWS Glue (ETL jobs), AWS Lambda, AWS Step Functions, AWS Athena, AWS S3, AWS RDS (MySQL)
• Strong knowledge of SQL and data querying
• Familiarity with IAM roles and AWS security best practices

Preferred Skills

• AWS Certification (e.g., AWS Certified Data Engineer / Developer)
• Experience with CI/CD pipelines and DevOps tools
• Familiarity with Docker / Kubernetes

At YASH, you are empowered to create a career that will take you to where you want to go while working in an inclusive team environment. We leverage career-oriented skilling models and optimize our collective intelligence aided with technology for continuous learning, unlearning, and relearning at a rapid pace and scale.

Our Hyperlearning workplace is grounded upon four principles: Flexible work arrangements, Free spirit, and emotional positivity; Agile self-determination, trust, transparency, and open collaboration; All Support needed for the realization of business goals, Stable employment with a great atmosphere and ethical corporate culture.

Experience Level Senior Level