Python /Pyspark Developer

CIEL HR; Bengaluru; On-site; INR 1,200,000 - 2,500,000; Full time

Job summary

CIEL HR in Bengaluru, India seeks a Python and PySpark Developer to design, develop, and optimize data processing applications and pipelines. You will build scalable ETL processes using Python and PySpark, optimize existing workloads on Databricks Lakehouse, and work with Hadoop, Hive, Kafka and AWS to deliver reliable data solutions for analytics and BI.

Strong collaboration with data engineers and data scientists is required, along with a proactive approach to data quality, validation, and

Qualifications

• Strong proficiency in Python and PySpark for large-scale data processing.
• Spark SQL and DataFrames experience is a plus.
• Familiarity with Hadoop, Hive, Kafka and AWS for data pipelines.
• Excellent collaboration and communication skills for cross-functional teams.

Responsibilities

• Develop, test, and maintain scalable data pipelines and ETL processes using Python and PySpark to extract, transform, and load large datasets.
• Optimize and fine-tune existing PySpark applications and workflows for performance improvements and efficiency.
• Work with big data technologies such as Hadoop, Hive, or Kafka, and AWS cloud platforms, with Databricks Lakehouse experience.
• Ensure data quality and integrity through validation and error-handling mechanisms.
• Monitor and troubleshoot data processing jobs in production environments to ensure reliability and prompt issue resolution.

Skills

Python

PySpark

Spark SQL

DataFrames

SQL

Problem-solving

Communication

Tools

Hadoop

Hive

Kafka

Databricks Lakehouse

AWS

Job description

A Python and PySpark Developer is responsible for designing, developing, and optimizing data processing applications and pipelines using Python and the Apache Spark framework

Key Responsibilities:

Develop, test, and maintain scalable data pipelines and ETL processes using Python and PySpark to extract, transform, and load large datasetsOptimize and fine-tune existing PySpark applications and workflows for performance improvements and efficiencyWork with big data technologies such as Hadoop, Hive, or Kafka, and AWS cloud platforms. Experienced in Databricks Lakehouse platformEnsure data quality and integrity throughout the data lifecycle by performing data validation and implementing error-handling mechanismsMonitor and troubleshoot data processing jobs in production environments to ensure reliability and prompt issue resolution

Requirements:

Expertise in handling large-scale datasets and collaborating with data engineers and data scientists to build robust, scalable data solutions for analytics, machine learning, and business intelligenceStrong proficiency in Python and relevant libraries. Expertise in Apache Spark and PySpark, including Spark SQL and DataFramesSolid understanding of data processing concepts, ETL processes, Data warehousing, Quality gates, Data pipeline & reconciliationProficiency in SQL and experience with various database systemsExcellent communication and collaboration skills to work with cross-functional teams. Strong analytical and problem-solving skills for debugging complex data issues.