Data Engineer (AWS, Databricks, PySpark)

Tata Consultancy Services; Hyderabad, Bengaluru; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Tata Consultancy Services in Hyderabad seeks an experienced Data Engineer to design scalable data platforms using AWS, Databricks, Python, PySpark, and SQL. The role focuses on cloud-native data engineering, ETL/ELT development, and large-scale data processing across AWS environments.

You will build ingestions and pipelines, optimize Spark workloads, implement data quality and governance, and collaborate with architects, analysts, and stakeholders.

Job description

We are seeking an experienced Data Engineer with strong expertise in AWS, Databricks, Python, PySpark, and SQL to build and optimize scalable data platforms and modern data pipelines. The ideal candidate will have hands‑on experience in cloud‑native data engineering, ETL/ELT development, and large‑scale data processing across AWS environments.

Key Responsibilities

• Design, develop, and maintain scalable data ingestion, transformation, and integration pipelines.
• Build and optimize ETL/ELT workflows using Python, PySpark, SQL, and Databricks.
• Develop cloud-native data solutions on AWS platforms.
• Create and maintain data pipelines for structured and unstructured data processing.
• Work with AWS services including Glue, Redshift, S3, Lambda, EMR, and Databricks.
• Collaborate with architects, analysts, and business stakeholders to design data models and integration frameworks.
• Implement data quality checks, monitoring, and governance controls.
• Develop reusable frameworks and components for orchestration, automation, and monitoring.
• Optimize Spark workloads and data processing performance.
• Troubleshoot data pipeline issues and ensure reliability, scalability, and security.
• Support Agile delivery and contribute to continuous improvement initiatives.

Required Skills

• 6-10 years of experience in Data Engineering.
• Strong expertise in Python programming.
• Hands‑on experience with PySpark and Apache Spark.
• Strong SQL development and optimization skills.
• Experience with Databricks platform.
• Experience building ETL/ELT pipelines.
• Strong AWS experience including:
• AWS Glue
• Amazon Redshift
• Amazon S3
• AWS Lambda
• Amazon EMR
• Knowledge of data modelling and data integration concepts.
• Experience working with large‑scale data platforms and cloud‑native architectures.

Preferred Skills

• Data Lake and Data Warehouse implementations.
• Workflow orchestration tools.
• Real‑time data processing experience.