Pyspark Developer

Hexaware Technologies; Hyderabad, Bengaluru; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Hexaware Technologies in Hyderabad is looking for a senior data engineer to design, build, and optimize scalable data pipelines using PySpark on AWS EMR. You will work with cross‑functional teams to gather requirements and deliver high‑quality solutions.

The role involves troubleshooting Kafka and Airflow workflows, automating deployments with Ansible, and ensuring security controls through proper access and encryption measures.

Qualifications

• 7-12 years of software development experience.
• Strong Python programming expertise.
• Experience with Big Data technologies: Hadoop, Spark, Kafka.
• Experience with AWS EMR or similar cloud services.
• Proficiency in scripting languages like Bash or Python for automation.

Responsibilities

• Design, develop, test, deploy, and maintain scalable data pipelines using PySpark on AWS EMR.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.
• Troubleshoot complex issues related to Kafka messaging systems and Apache Airflow workflows.
• Develop automation scripts using Ansible or equivalent tools for deployment and management of big data applications.
• Ensure compliance with security standards by implementing access controls and encryption mechanisms.

Skills

Python programming

Shell scripting

Tools

PySpark

AWS EMR

Hadoop

Spark

Kafka

Apache Airflow

Ansible

Job description

Roles and Responsibilities

• Design, develop, test, deploy, and maintain scalable data pipelines using PySpark on AWS EMR.
  
• Collaborate with cross‑functional teams to gather requirements and deliver high‑quality solutions.
  
• Troubleshoot complex issues related to Kafka messaging systems and Apache Airflow workflows.
  
• Develop automation scripts using Ansible or equivalent tools for deployment and management of big data applications.
  
• Ensure compliance with security standards by implementing access controls and encryption mechanisms.
  

Desired Candidate Profile

• 7-12 years of experience in software development with expertise in Python programming language.
  
• Strong understanding of Big Data technologies such as Hadoop, Spark, Kafka, etc. .
  
• Experience working with cloud‑based platforms like AWS EMR or similar services.
  
• Proficiency in scripting languages like Shell Scripting (Bash) or Python for automating tasks.