Senior Data Engineer – PySpark, Cloud & Kafka - 5+ YoE - Immediate Joiner - Any UST Location

UST; Bengaluru; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A tech solutions company in India is seeking a professional to design, develop, and maintain scalable data pipelines using PySpark. Key responsibilities include building batch and real-time processing systems and optimizing Spark jobs. The ideal candidate should have hands-on experience with PySpark, strong SQL skills, and familiarity with cloud platforms like AWS. Experience with Kafka and CI/CD pipelines for data solutions is also preferred. This role offers an opportunity to collaborate with data teams and ensure data quality and security.

Qualifications

• Strong hands-on experience with PySpark / Apache Spark.
• Solid understanding of distributed data processing concepts.
• Experience with Apache Kafka (producers, consumers, topics, partitions).
• Hands-on experience with any one cloud platform (AWS preferred).
• Proficiency in Python.
• Strong experience with SQL and data modeling.
• Experience working with large-scale datasets.
• Familiarity with Linux/Unix environments.
• Understanding of ETL/ELT frameworks.
• Experience with CI/CD pipelines for data applications.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Build and manage batch and real-time data processing systems.
• Develop and integrate Kafka-based streaming solutions.
• Optimize Spark jobs for performance, cost, and scalability.
• Work with cloud-native services to deploy and manage data solutions.
• Ensure data quality, reliability, and security across platforms.
• Collaborate with data scientists, analysts, and application teams.
• Participate in code reviews, design discussions, and production support.

Job description

Candidates ready to join immediately can share their details via email for quick processing.

[e-mail]

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark
• Build and manage batch and real-time data processing systems
• Develop and integrate Kafka-based streaming solutions
• Optimize Spark jobs for performance, cost, and scalability
• Work with cloud-native services to deploy and manage data solutions
• Ensure data quality, reliability, and security across platforms
• Collaborate with data scientists, analysts, and application teams
• Participate in code reviews, design discussions, and production support

Must-Have Skills

• Strong hands-on experience with PySpark / Apache Spark
• Solid understanding of distributed data processing concepts
• Experience with Apache Kafka (producers, consumers, topics, partitions)
• Hands-on experience with any one cloud platform:
• AWS (S3, EMR, Glue, EC2, IAM) or
• Proficiency in Python
• Strong experience with SQL and data modeling
• Experience working with large-scale datasets
• Familiarity with Linux/Unix environments
• Understanding of ETL/ELT frameworks
• Experience with CI/CD pipelines for data applications

Good-to-Have Skills

• Experience with Spark Structured Streaming
• Knowledge of Kafka Connect and Kafka Streams
• Exposure to Databricks
• Experience with NoSQL databases (Cassandra, MongoDB, HBase)
• Familiarity with workflow orchestration tools (Airflow, Oozie)
• Knowledge of containerization (Docker, Kubernetes)
• Experience with data lake architectures
• Understanding of security, governance, and compliance in cloud environments
• Exposure to Scala or Java is a plus
• Prior experience in Agile/Scrum environments