AWS Data Engineer/ Data Bricks/Pyspark

Tata Consultancy Services; Bengaluru; On-site; INR 2,500,000 - 3,800,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a Technical Lead in Python/PySpark Data Engineering to guide design and delivery of scalable data pipelines. You will lead end-to-end Databricks projects, optimize streaming and batch workflows, and mentor a team of developers in a fast-paced cloud data environment.

Ideal candidates bring 5-10 years of IT experience with hands-on Python, PySpark, Databricks, AWS, Kafka, and Airflow expertise.

Qualifications

• 5-10 years of IT experience with strong data engineering expertise.
• Hands-on experience in Python, PySpark, Spark SQL, Databricks and AWS.
• Experience with Airflow, Oracle, PostgreSQL, and NoSQL databases.

Responsibilities

• Design, architect and develop scalable data processing applications using Python, PySpark, Databricks, and AWS.
• Build and optimize batch and streaming data pipelines using Apache Kafka and Airflow.
• Lead end-to-end Databricks implementations including architecture, development, deployment and performance tuning.
• Develop and maintain complex ETL/ELT solutions for large-scale data platforms.
• Collaborate with business stakeholders, vendors and cross-functional teams to deliver high-quality solutions.
• Support production systems, troubleshoot critical issues, and provide timely resolutions.
• Perform data modeling, database design and performance optimization.
• Mentor junior developers and provide technical leadership to Agile teams.
• Ensure adherence to best practices in software development, testing and deployment.

Skills

Python

Databricks

SQL / Spark SQL

Tools

Oracle Database

PL/SQL

Unix Shell Scripting

AutoSys

Test-Driven Development

GitHub

Job description

Job Title: Technical Lead - Python/PySpark Data Engineering

Experience: 5-10 Years

Job Description

We are looking for an experienced Technical Lead with strong expertise in Python, PySpark, Databricks, AWS Cloud, SQL, and Apache Kafka to lead the design, development, and implementation of scalable data engineering solutions. The ideal candidate will have extensive hands-on experience in big data processing, cloud-based data platforms, and real-time streaming architectures.

Primary Skills

• Python
• Databricks
• SQL / Spark SQL

Secondary Skills

• Oracle Database
• PL/SQL
• Unix Shell Scripting
• AutoSys
• Test-Driven Development (TDD)
• GitHub

Key Responsibilities

• Design, architect, and develop scalable data processing applications using Python, PySpark, Databricks, and AWS.
• Build and optimize batch and streaming data pipelines using Apache Kafka and Airflow.
• Lead end-to-end implementation of Databricks solutions, including architecture, development, deployment, and performance tuning.
• Develop and maintain complex ETL/ELT solutions for large-scale data platforms.
• Work closely with business stakeholders, vendors, and cross-functional teams to deliver high-quality solutions.
• Support production systems, troubleshoot critical issues, and provide timely resolutions.
• Perform data modeling, database design, and performance optimization.
• Mentor junior developers and provide technical leadership to Agile teams.
• Ensure adherence to best practices in software development, testing, and deployment.

Required Qualifications

• 5-10 years of overall IT experience with strong data engineering expertise.
• Strong hands-on experience in Python, PySpark, Spark SQL, Databricks, and AWS.
• Minimum 2+ years of experience with Apache Kafka streaming solutions.
• Experience with Airflow, Oracle, PostgreSQL, and NoSQL databases.
• Knowledge of APIs, data integration frameworks, and real-time data streaming.
• Experience working in Agile environments and version control systems.
• Excellent analytical, problem-solving, and communication skills.
• Ability to work independently in a fast-paced, deadline-driven environment.

Preferred Qualifications

• Experience in Banking or Capital Markets domain.
• Exposure to large-scale enterprise data platforms and cloud migration projects.

Note: This role requires approximately 90% hands-on development, design, and architecture work and 10% team mentoring and technical leadership responsibilities.