Python + Spark + SQL Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a data engineer to design and maintain scalable data pipelines using Python and Apache Spark. You will write SQL for data extraction and transformation, process diverse data sources, and optimize Spark jobs for performance and cost efficiency.

The role emphasizes data validation, monitoring, and collaboration with stakeholders to meet data requirements, while ensuring security and reliability across platforms.

Qualifications

• 3+ years in data engineering with Python, Spark, and SQL.
• Strong Python for data processing and scripting.
• Hands-on experience with Apache Spark (PySpark) for large-scale processing.
• Advanced SQL: joins, window functions, and performance tuning.
• Experience with ETL/ELT concepts across multiple sources.
• Nice to have cloud platforms, data lakes, and Airflow.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and Spark.
• Write optimized SQL queries for data extraction, transformation, and analysis.
• Process structured and semi-structured data from multiple sources.
• Optimize Spark jobs for performance and cost efficiency.
• Implement data validation, error handling, and monitoring.
• Collaborate with stakeholders to understand and fulfill data requirements.
• Ensure data security, reliability, and adherence to best practices.

Skills

Python

Spark

SQL

ETL/ELT

Data processing

Airflow

Cloud platforms

Tools

Airflow

Hive

Delta Lake

Iceberg

Cloud platforms (AWS/Azure/GCP)

Job description

Job Requirements*

• 3+ years of experience in data engineering with hands-on Python, Spark, and SQL.
• Strong proficiency in Python (data processing, scripting, OOP).
• Hands-on experience with Apache Spark (PySpark) for large-scale data processing.
• Advanced knowledge of SQL - joins, window functions, performance tuning.
• Experience processing structured and semi-structured data from multiple sources.
• Strong understanding of ETL/ELT concepts and best practices.
• Good to have: experience with cloud platforms (AWS, Azure, GCP), data lakes (Hive, Delta Lake, Iceberg), and orchestration tools (Airflow).

Key Responsibilities*

• Design, develop, and maintain scalable data pipelines using Python and Apache Spark.
• Write optimized SQL queries for data extraction, transformation, and analysis.
• Process structured and semi-structured data from multiple sources.
• Optimize Spark jobs for performance and cost efficiency.
• Implement data validation, error handling, and monitoring.
• Collaborate with stakeholders to understand and fulfill data requirements.
• Ensure data security, reliability, and adherence to best practices.