Aws Data Engineer

Tata Consultancy Services; Indore District; On-site; INR 1,000,000 - 2,000,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced Data Engineer located in Indore District, Madhya Pradesh, to design and implement ETL/ELT pipelines using PySpark in distributed environments. The ideal candidate will have expertise in AWS services including Glue, EMR, Redshift, and S3 to effectively manage large-scale data processing.

Candidates should possess strong skills in Python and SQL along with a solid understanding of modern data architectures. This position offers opportunities for growth within a dynamic team.

Qualifications

• Strong proficiency in Python and SQL with query optimization skills.
• Expertise in PySpark and experience in modern data architectures.
• 6+ years of experience in data engineering roles.

Responsibilities

• Design and implement robust ETL/ELT pipelines using PySpark.
• Utilize AWS services for data storage and transformation.
• Optimize PySpark jobs and SQL queries for efficiency.

Skills

Python

SQL (window functions, query optimization)

PySpark

AWS (Glue, EMR, Redshift, S3)

Data Architectures (Snowflake, Databricks)

Tools

Kafka

Airflow

NoSQL databases (DynamoDB)

Job description

Key Responsibilities

• Design and implement robust ETL/ELT pipelines using PySpark for large-scale data processing in distributed environments.
• Utilize AWS services such as AWS Glue, EMR, Redshift, and S3 for data storage, transformation, and warehousing.
• Optimize and fine-tune PySpark jobs and SQL queries to enhance data processing speed and reduce costs.
• Design data models and implement quality checks to ensure data consistency and accuracy.
• Work with data architects, data scientists, and analysts to deliver data solutions, participating in code reviews and adhering to best practices.
• Implement streaming solutions using Kafka and automate workflows, often using tools like Airflow.

Required Skills

• Strong proficiency in Python and SQL (window functions, query optimization).
• Expertise in PySpark and Spark architecture for distributed data processing.
• Hands‑on experience with AWS cloud environment, specifically Glue, EMR, Redshift, and S3.
• Understanding of Modern Data Architectures, including Snowflake or Databricks.
• 6+ years of relevant experience in data engineering roles.

Preferred Skills

• Knowledge of NoSQL databases (e.g., DynamoDB).
• Familiarity with DevOps tools (CI/CD) and version control (Git).
• Streaming experience (Kafka).
• Relevant AWS or Data Engineering certifications.