Developer

GSB Solutions; Hyderabad; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A technology solutions firm located in Hyderabad is seeking an experienced Data Engineer proficient in Python and PySpark to develop and maintain scalable data pipelines. The ideal candidate should have at least five years of experience in Python development, a strong grasp of data warehousing, and familiarity with cloud platforms like AWS or Azure. This role involves designing ETL processes and collaborating with teams to ensure data quality and integrity across the lifecycle.

Qualifications

• 5+ years of experience in Python and PySpark development.
• Experience with data warehousing and data lakes.
• Knowledge of machine learning libraries (e.g., MLlib) is a plus.
• Strong problem-solving and debugging skills.
• Excellent communication and collaboration abilities.

Responsibilities

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like AWS, Azure, or GCP.
• Implement data storage solutions and manage large-scale datasets.

Skills

Python programming

PySpark

Apache Spark

Big Data technologies

SQL

Cloud computing

Data engineering best practices

REST APIs

Job description

Must-Have

• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and relational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Understanding data engineering best practices.
• Experience with REST APIs, JSON/XML, and data serialization.
• Exposure to cloud computing environments.

Qualifications

• 5+ years of experience in Python and PySpark development.
• Experience with data warehousing and data lakes.
• Knowledge of machine learning libraries (e.g., MLlib) is a plus.
• Strong problem-solving and debugging skills.
• Excellent communication and collaboration abilities.

Responsibilities

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like AWS, Azure, or GCP.
• Implement data storage solutions and manage large-scale datasets.