Python & Kafka Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced Python & Kafka Data Engineer to design, develop, and maintain scalable data pipelines and real-time streaming solutions in Bengaluru. The role emphasizes Python, Apache Kafka, data engineering, ETL processes, and cloud technologies.

Key responsibilities include building batch and real-time pipelines, developing ETL workflows, and ensuring data quality, security, and performance across data sources and cloud platforms.

Qualifications

• Strong hands-on experience in Python development.
• Expertise in Apache Kafka, Kafka Streams, and Kafka Connect.
• Experience with PySpark/Spark and Big Data technologies.
• Strong knowledge of SQL and database systems.
• Experience in building ETL/ELT pipelines.
• Hands-on experience with AWS/Azure/GCP cloud platforms.
• Knowledge of REST APIs and Microservices.
• Experience with Git, CI/CD tools, and Agile methodologies.

Responsibilities

• Design, develop, and optimize batch and real-time data pipelines.
• Build scalable streaming applications using Apache Kafka.
• Develop data processing frameworks and ETL workflows using Python.
• Integrate data from multiple sources, including APIs, databases, and cloud platforms.
• Ensure data quality, performance, reliability, and security.
• Work closely with Data Scientists, Analysts, and Business teams to deliver data solutions.
• Monitor and troubleshoot production data pipelines.
• Implement best practices for data governance and data management.

Skills

Python Development

Apache Kafka

Kafka Streams

Kafka Connect

PySpark/Spark

Big Data technologies

SQL

ETL/ELT pipelines

AWS/Azure/GCP

REST APIs

Microservices

Git

CI/CD

Agile methodologies

Analytical

Tools

Git

CI/CD

Kafka

Spark

Job description

We are looking for an experienced Python & Kafka Data Engineer to design, develop, and maintain scalable data pipelines and real-time streaming solutions. The ideal candidate should have strong expertise in Python, Apache Kafka, Data Engineering, ETL processes, and cloud technologies.

Key Responsibilities

• Design, develop, and optimize batch and real-time data pipelines.
• Build scalable streaming applications using Apache Kafka.
• Develop data processing frameworks and ETL workflows using Python.
• Integrate data from multiple sources, including APIs, databases, and cloud platforms.
• Ensure data quality, performance, reliability, and security.
• Work closely with Data Scientists, Analysts, and Business teams to deliver data solutions.
• Monitor and troubleshoot production data pipelines.
• Implement best practices for data governance and data management.

Required Skills

• Strong hands-on experience in Python Development.
• Expertise in Apache Kafka, Kafka Streams, and Kafka Connect.
• Experience with PySpark/Spark and Big Data technologies.
• Strong knowledge of SQL and database systems.
• Experience in building ETL/ELT pipelines.
• Hands-on experience with AWS/Azure/GCP cloud platforms.
• Knowledge of REST APIs and Microservices.
• Experience with Git, CI/CD tools, and Agile methodologies.
• Strong analytical and problem-solving skills.