Pyspark_6-10 Years

Tata Consultancy Services; Hyderabad, Bengaluru; On-site; INR 1,800,000 - 3,600,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced AWS Data Engineer in Hyderabad to design, build, and optimize scalable data pipelines on the AWS cloud. The role emphasizes modern data platforms, data warehousing with Py-Spark, and close collaboration with data teams to deliver high-performance analytics solutions.

You will migrate on‑prem or multi‑cloud workloads to AWS, tune Spark workloads on EMR, and implement CI/CD and IaC for automated deployment and operations, ensuring governance and

Responsibilities

• Design, develop, and optimize scalable data pipelines on AWS.
• Build batch and real-time data ingestion/processing frameworks.
• Develop enterprise data warehousing using Py-Spark.
• Convert Scala ETL and streaming pipelines to PySpark.
• Optimize PySpark jobs for performance and scalability.
• Support cloud modernization on AWS/Databricks platforms.
• Implement ETL/ELT for structured and unstructured data.
• Integrate data from databases, APIs, files, and streams.
• Ensure data quality, governance, security, and compliance.
• Automate deployment and operations with CI/CD and IaC.
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Analysts, and Data Scientists.
• Implement data models, metadata, and data lineage best practices.
• Lead migration and optimization of Spark workloads on AWS EMR.

Job description

Role & responsibilities

We are seeking an experienced AWS Data Engineer to design, develop, and maintain scalable data solutions on the Amazon Cloud Platform (AWS). The ideal candidate will have strong expertise in building modern data pipelines, data warehousing, big data processing, and cloud-native analytics solutions. The role requires close collaboration with business stakeholders, data architects, data scientists, and application teams to deliver reliable and high-performance data platforms.

• Design, develop, and optimize scalable data pipelines using AWS services.
• Build and maintain batch and real-time data ingestion and processing frameworks.
• Develop enterprise-grade data warehousing solutions using Py-Spark.
• Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.
• Optimize PySpark jobs for performance, scalability, and resource utilization.
• Support cloud modernization initiatives on AWS/Databricks platforms.
• Implement ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Ensure data quality, governance, security, and compliance across data platforms.
• Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
• Monitor data pipelines and troubleshoot production issues.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.
• Implement data models, metadata management, and data lineage best practices.
• Support migration of on-premises or multi-cloud data platforms to AWS.
• Lead the migration, modernization, and optimization of Apache Spark workloads on AWS EMR, including conversion of Scala-based Spark applications to PySpark, performance tuning, cluster optimization, dependency management, and ensuring scalable, cost-effective, and resilient data processing solutions.