AWS Data Engineer (Kafka & PySpark)

Tata Consultancy Services Limited; Kolkata District, Chennai, Bengaluru; Presencial; INR 1.200.000 - 2.400.000; Jornada completa

Descripción de la vacante

Tata Consultancy Services Limited in Kolkata, Bangalore and Chennai seeks an experienced AWS Data Engineer to design, build, and maintain scalable batch and real-time data pipelines on AWS using PySpark and Kafka.

The role focuses on ETL/ELT pipelines, streaming data, and data lake solutions with S3, EMR, Glue, Redshift, IAM, and workflow orchestration via Airflow or AWS Step Functions.

Formación

• Experience designing and implementing AWS-based data pipelines using PySpark and Kafka.
• Hands-on experience with batch and real-time streaming data pipelines on cloud platforms.
• Proficient in optimizing Spark jobs and building data lakes with S3 and Redshift.

Responsabilidades

• Design and develop scalable ETL/ELT pipelines using PySpark and AWS services.
• Build distributed data-processing solutions using Spark on Amazon EMR.
• Develop real-time data pipelines with Kafka and Spark/PySpark.
• Implement data lake solutions using Amazon S3 and formats like Parquet/ORC.
• Optimize Spark jobs via partitioning, caching, and tuning.
• Integrate data from files, databases, APIs, and streaming sources.
• Work with AWS services such as S3, EMR, Glue, Redshift, IAM, CloudWatch, and Step Functions.
• Implement workflow orchestration using Airflow or AWS Step Functions.
• Monitor pipelines, troubleshoot production issues, and improve reliability.

Conocimientos

AWS Data Engineering

Apache Kafka

PySpark

Apache Spark

Python

SQL

Amazon S3

AWS EMR

AWS Glue

Amazon Redshift

AWS Step Functions

Airflow

Descripción del empleo

Role- AWS Data Engineer (Kafka & PySpark)

Experience- 4-8 years

Location- Kolkata , Bangalore, Chennai

Job Description-

We are looking for an experienced AWS Data Engineer with strong hands-on expertise in PySpark and Apache Kafka to design, develop, and maintain scalable batch and real-time data pipelines on AWS.

Roles & Responsibilities

• Design and develop scalable ETL/ELT data pipelines using PySpark and AWS services.
• Build distributed data-processing solutions using PySpark/Spark on Amazon EMR. [#Personal#...Interview | Outlook]
• Develop real-time/streaming data pipelines using Kafka and Spark/PySpark.
• Implement data lake solutions using Amazon S3 and formats such as Parquet/ORC. [#Personal#...Interview | Outlook]
• Optimize Spark jobs through partitioning, caching, broadcast joins, and cluster tuning. [#Personal#...Interview | Outlook]
• Integrate data from files, databases, APIs, and streaming sources.
• Work with AWS services such as S3, EMR, Glue, Redshift, IAM, CloudWatch, and Step Functions. [#Personal#...Interview | Outlook]
• Implement workflow orchestration using Airflow / AWS Step Functions.
• Monitor pipelines, troubleshoot production issues, and improve performance and reliability.
• Implement security controls using IAM, encryption/KMS, VPC, and appropriate access controls. [#Personal#...Interview | Outlook]

Mandatory Skills

• AWS Data Engineering
• Apache Kafka
• PySpark
• Apache Spark
• Python
• SQL
• Amazon S3
• AWS EMR
• AWS Glue
• Amazon Redshift
• AWS Step Functions / Airflow

Good to Have

• Spark Structured Streaming
• Data Lake / Lakehouse concepts
• Parquet / Avro / ORC
• CI/CD
• Databricks
• Data quality and monitoring