Data Engineer- Python Pyspark GCP

Mployee.me; Hyderabad; In loco; INR 1.800.000 - 2.800.000; Tempo pieno

Descrizione del lavoro

Mployee.me is seeking an experienced Data Engineer to design and maintain scalable data pipelines using Python and PySpark. You will implement ETL processes, optimize PySpark workloads, and ensure data quality across the lifecycle.

You will collaborate with cross-functional teams, write clean code, and participate in design reviews. Expertise in Big Data technologies, SQL, and REST APIs is required, with hands-on cloud experience on GCP/AWS/Azure.

Competenze

• 4 to 8 + Years of experience using Python and Pyspark.
• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and relational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Understanding of data engineering best practices.
• Experience with REST APIs, JSON/XML, and data serialization.

Mansioni

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like GCP, AWS or Azure.

Conoscenze

Python

PySpark

Apache Spark

SQL

Big Data

Distributed computing

REST APIs

JSON/XML

Strumenti

GCP

AWS

Azure

Hadoop

Hive

Kafka

SQL databases

NoSQL databases

Spark SQL

Cloud computing

Descrizione del lavoro

Role & responsibilities

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like GCP, AWS or Azure.

Implement data storage solutions and manage large-scale datasets.

Preferred candidate profile

• 4 to 8 + Years of experience using Python and Pyspark.
• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and elational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Understanding of data engineering best practices.
• Experience with REST APIs, JSON/XML, and data serialization.

Exposure to GCP services and cloud computing environments