Senior Software Engineer – AWS Data Pipeline

Jobtailor; Bengaluru; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Jobtailor is seeking a senior data engineer in Bengaluru to design and implement scalable data pipelines using AWS services. You will develop PySpark ETL workflows, build serverless data processing with Lambda, and orchestrate jobs with Step Functions.

You will ensure data quality, governance, and cost optimization while collaborating with Analysts, Architects, and DevOps in a fast-paced environment.

Qualifications

• 5+ years of experience in data engineering / AWS development.
• Strong proficiency in Python.
• Hands-on experience with PySpark for large-scale data processing.
• Expertise with AWS Glue (ETL jobs).
• Expertise with AWS Lambda.
• Expertise with AWS Step Functions.
• Expertise with AWS Athena.
• Expertise with AWS S3.
• Expertise with AWS RDS (MySQL).
• Strong knowledge of SQL and data querying.
• Familiarity with IAM roles and AWS security best practices.
• Preferred: AWS Certification (e.g., AWS Certified Data Engineer / Developer).
• Preferred: Experience with CI/CD pipelines and DevOps tools.
• Preferred: Familiarity with Docker / Kubernetes.

Responsibilities

• Design and implement scalable data pipelines using AWS services.
• Develop ETL workflows using AWS Glue, PySpark, and Python.
• Build serverless data processing solutions using AWS Lambda.
• Orchestrate workflows using AWS Step Functions.
• Manage and optimize data storage in AWS S3.
• Work with relational databases like AWS RDS (MySQL).
• Ensure data quality, validation, and governance.
• Optimize performance and cost of data pipelines.
• Debug and troubleshoot data-related issues.
• Collaborate with cross-functional teams including Data Analysts, Architects, and DevOps.

Skills

Python

PySpark

SQL

Data Engineering

AWS Security Best Practices

CI/CD

Docker

Kubernetes

AWS Glue

AWS Lambda

AWS Step Functions

AWS Athena

AWS S3

AWS RDS MySQL

Tools

AWS Glue

AWS Lambda

AWS Step Functions

AWS Athena

AWS S3

AWS RDS (MySQL)

Docker

Kubernetes

CI/CD Pipelines

Job description

Job Responsibilities

• Design and implement scalable data pipelines using AWS services
• Develop ETL workflows using AWS Glue, PySpark, and Python
• Build serverless data processing solutions using AWS Lambda
• Orchestrate workflows using AWS Step Functions
• Manage and optimize data storage in AWS S3
• Work with relational databases like AWS RDS (MySQL)
• Ensure data quality, validation, and governance
• Optimize performance and cost of data pipelines
• Debug and troubleshoot data-related issues
• Collaborate with cross-functional teams including Data Analysts, Architects, and DevOps

Requirements

• 5+ years of experience in data engineering / AWS development
• Strong proficiency in Python
• Hands-on experience with PySpark for large-scale data processing
• Expertise with AWS Glue (ETL jobs)
• Expertise with AWS Lambda
• Expertise with AWS Step Functions
• Expertise with AWS Athena
• Expertise with AWS S3
• Expertise with AWS RDS (MySQL)
• Strong knowledge of SQL and data querying
• Familiarity with IAM roles and AWS security best practices
• Preferred: AWS Certification (e.g., AWS Certified Data Engineer / Developer)
• Preferred: Experience with CI/CD pipelines and DevOps tools
• Preferred: Familiarity with Docker / Kubernetes

Core Competencies

Demonstrates expertise in designing and implementing scalable data pipelines using AWS services, with strong proficiency in Python and PySpark for ETL workflows. Ensures data quality and governance while optimizing performance and cost of data solutions.

Highest-signal resume keywords

• AWS Data Engineering
• Python Programming
• AWS Glue ETL
• AWS Lambda
• AWS Step Functions

Hard Skills

• Data Pipeline Design
• ETL Workflows
• PySpark
• SQL
• Data Quality Assurance

Soft Skills

• Collaboration
• Problem-Solving

Certifications & Qualifications

• AWS Certified Data Engineer
• AWS Certified Developer

Industry Keywords

• Data Governance
• Data Validation
• AWS Security Best Practices
• Cross-Functional Collaboration

Tools & Technologies

• AWS S3
• AWS RDS (MySQL)
• AWS Athena
• CI/CD Pipelines
• Docker