AWS Data Engineer (AWS Glue & PySpark)

Tata Consultancy Services; Bengaluru; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

Tata Consultancy Services is seeking an AWS Data Engineer in Bengaluru to design, develop, and optimize scalable ETL pipelines within an AWS ecosystem. You will work on data integration, transformation, and analytics solutions using Glue, PySpark, Python, Redshift, and Athena.

The role emphasizes data engineering over DevOps and requires hands-on cloud experience. The successful candidate will build ingestion frameworks from diverse sources, implement real-time and batch processing, and

Qualifications

• 4+ years in Data Integration or Data Engineering.
• 3+ years hands-on with AWS Glue.
• Strong PySpark and Apache Spark expertise.
• Solid Python programming experience.
• Experience with AWS Lambda, Athena, Step Functions, Redshift.
• Experience designing and developing ETL pipelines on AWS.
• Strong SQL development skills; PostgreSQL or Redshift familiarity.
• Hands-on with multiple AWS services for analytics platforms.
• Experience real-time and scalable workflow implementations.

Responsibilities

• Design, develop, and maintain scalable ETL and data integration pipelines on AWS.
• Build ingestion frameworks from databases, files, APIs, and Kafka streams.
• Develop data transformation using AWS Glue, PySpark, and Python.
• Create data pipelines with Lambda, Athena, Step Functions, Redshift, S3, and EC2.
• Maintain data models and database objects for analytics and reporting.
• Implement real-time and batch data processing workflows.
• Collaborate with architects, product owners, and stakeholders.
• Participate in agile sprint planning, development, testing, and releases.
• Provide support for data delivery across business use cases.
• Troubleshoot performance bottlenecks and optimize cloud solutions.
• Demonstrate solutions to clients and stakeholders when required.

Skills

AWS Glue

PySpark

Python

SQL

Redshift

Athena

Spark

Kafka

Lambda

Step Functions

S3

EC2

Tools

AWS Glue

AWS Lambda

Athena

Redshift

Step Functions

S3

EC2

Job description

We are seeking a highly skilled AWS Data Engineer with strong expertise in AWS Glue, PySpark, Python, Redshift, Athena, and AWS Analytics Services. The ideal candidate will be responsible for designing, developing, and optimizing scalable ETL pipelines and cloud-based data integration solutions within an AWS ecosystem. This role requires hands-on development experience and is focused on Data Engineering rather than DevOps.

Key Responsibilities

• Design, develop, and maintain scalable ETL and data integration pipelines using AWS services.
• Build data ingestion frameworks from multiple heterogeneous sources including databases, files, APIs, and Kafka streams.
• Develop data transformation and processing solutions using AWS Glue, PySpark, and Python.
• Create and optimize data pipelines leveraging AWS Lambda, Athena, Step Functions, Redshift, S3, and EC2.
• Develop and maintain data models and database objects for analytics and reporting requirements.
• Implement real-time and batch data processing workflows.
• Collaborate with architects, product owners, and business stakeholders to deliver data platform solutions.
• Perform code reviews and ensure adherence to architecture and development standards.
• Participate in Agile development processes including sprint planning, development, testing, and releases.
• Support data delivery requirements across multiple business use cases.
• Troubleshoot performance bottlenecks and optimize cloud-based data solutions.
• Demonstrate solutions to clients and internal stakeholders when required.

Required Skills

• Minimum 4 years of experience in Data Integration and Data Engineering.
• 3+ years of hands-on experience with AWS Glue.
• Strong expertise in PySpark and Apache Spark.
• Strong Python programming experience.
• Experience with AWS Lambda, Athena, Step Functions, and Redshift.
• Experience designing and developing ETL pipelines on AWS.
• Strong SQL development skills.
• Experience with PostgreSQL, Redshift, or other relational databases.
• Experience implementing data models for analytics platforms.
• Hands-on experience with AWS services including:
  • AWS Glue
  • Lambda
  • Athena
  • Redshift
  • Step Functions
  • S3
  • EC2
• Experience with real-time and scalable workflow implementations.
• Understanding of BI and Analytics solution development lifecycle.

Preferred Skills

• Informatica Cloud.
• Talend.
• SSIS.
• Matillion.
• Kafka integration and streaming technologies.
• Enterprise Data Warehousing solutions.
• Cloud-native Analytics platforms.

Desired Candidate Profile

• Strong problem-solving and analytical capabilities.
• Excellent verbal and written communication skills.
• Experience working on multiple projects simultaneously.
• Ability to work in Agile delivery environments.
• Strong collaboration and stakeholder management skills.
• Proactive approach to issue resolution and project delivery.
• Ability to work effectively under tight timelines.