Pyspark Developer

Acesoft Labs (India) Pvt Ltd; Dubai; Hybrid; AED 35,000 - 50,000; Full time

Job summary

Acesoft Labs (India) Pvt Ltd in Bangalore is seeking a skilled PySpark Developer with hands-on experience in big data processing, data engineering, and distributed computing. You will develop, optimize, and maintain scalable data pipelines using PySpark, Python, and related technologies.

The role requires building ETL pipelines, tuning performance, and working with cloud-based data platforms. You will collaborate with data engineers, architects, and analysts to ensure data quality and timely

Qualifications

• Hands-on PySpark development experience.
• Strong Python programming skills.
• Good knowledge of Spark DataFrames, Spark SQL, RDDs, and transformations.
• Experience building ETL/ELT pipelines.
• SQL skills with relational databases.
• Familiarity with data warehousing concepts and large datasets.

Responsibilities

• Develop and maintain scalable ETL/ELT data pipelines using PySpark and Python.
• Design data processing solutions for large and complex datasets.
• Write optimized Spark transformations and manage DataFrames/Spark SQL.
• Perform data cleansing, validation, transformation and aggregation.
• Tune Spark jobs for performance and resource utilization.
• Work with structured and semi-structured data from multiple sources.

Skills

PySpark

Python

Big Data

ETL

SQL

Spark Architecture

Data Warehousing

Delta Lake

Airflow

CI/CD

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

Databricks

Cloud platforms (AWS/Azure/GCP)

Job description

Job Description PySpark DeveloperPosition PySpark Developer strong Experience strong 3 6 Years strong Location strong Bangalore Bengaluru strong Employment Type strong Full-Time Contract strong Work Mode strong Hybrid As per client requirement

Role OverviewWe

We are looking for a skilled strong PySpark Developer strong with strong experience in big data processing data engineering and distributed computing The candidate will be responsible for developing optimizing and maintaining scalable data pipelines using strong PySpark Python and related Big Data technologies strong The ideal candidate should have hands-on experience in processing large datasets developing ETL pipelines performance tuning and working with cloud-based data platforms

Key Responsibilities

• Develop and maintain scalable strong ETL ELT data pipelines strong using PySpark and Python
• Design and implement data processing solutions for large and complex datasets
• Write efficient and optimized strong PySpark DataFrame and Spark SQL strong transformations
• Perform data cleansing validation transformation and aggregation
• Optimize Spark jobs for performance scalability and resource utilization
• Work with structured and semi-structured data from multiple sources
• Integrate data from databases APIs files and other enterprise data sources
• Troubleshoot data pipeline failures and resolve performance-related issues
• Implement data quality checks and ensure data accuracy and consistency
• Collaborate with Data Engineers Data Architects Analysts and other stakeholders
• Participate in code reviews testing deployment and production support
• Maintain technical documentation for data pipelines and processes

Mandatory Skills

• Strong hands-on experience with strong PySpark strong
• Strong programming experience in strong Python strong
• Good knowledge of strong Apache Spark strong architecture and distributed computing
• Strong understanding of strong Spark DataFrames Spark SQL RDDs transformations and actions strong
• Experience developing strong ETL ELT pipelines strong
• Strong SQL skills and experience working with relational databases
• Good understanding of data warehousing concepts
• Experience working with large-volume datasets
• Knowledge of Spark performance tuning and optimization techniques
• Good understanding of data processing and integration concepts
• Good to HaveExperience with cloud platforms such as strong AWS Azure or GCP strong
• Experience with strong Databricks strong
• Knowledge of strong Azure Data Factory AWS Glue EMR strong or similar data engineering tools
• Experience with strong Kafka strong or other streaming technologies
• Knowledge of Hive Hadoop or HDFS
• Experience with CI CD and DevOps practices
• Knowledge of data lake and lakehouse architecture
• Experience working with Delta Lake
• Familiarity with Airflow or other workflow orchestration tools

Education

Bachelor s or Master s degree in Computer Science Information Technology Engineering or a related field

Key Competencies

• Strong analytical and problem-solving skills
• Good communication and interpersonal skills
• Ability to work independently as well as collaboratively
• Strong debugging and troubleshooting capabilities
• Ability to work in a fast-paced environment and manage multiple priorities

Preferred Experience

3–6 years of relevant experience in Data Engineering / Big Data, with strong hands-on experience in PySpark and Python.