Pyspark developer

Aligned Automation

Maharashtra

On-site

INR 1,800,000 - 2,400,000

Full time

2 days ago

Be an early applicant

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

Aligned Automation in Pune (Work from office) is seeking a Senior Data Engineer with 7-8 years of hands-on experience in PySpark, Python, SQL, Databricks, and cloud platforms. You will design, develop, and optimize large-scale data pipelines and Data Lake/Lakehouse architectures, mentor juniors, and collaborate with data analysts and stakeholders to deliver scalable data solutions.

The role requires hands-on development, architecture discussions, and adherence to best practices, including CI/CD

Qualifications

7-8 years of experience in data engineering.
Proficient in PySpark, Python, SQL and Databricks.
Experience designing and optimizing large-scale data pipelines and data lake architectures.

Responsibilities

Design, develop, and optimize ETL/ELT pipelines using PySpark.
Build high-performance pipelines for large-scale data
Design reusable data ingestion, transformation, validation, and monitoring frameworks.
Optimize Spark jobs with partitioning, joins, caching, and memory tuning.
Implement Data Lake/Lakehouse architectures using Delta Lake.
Write complex SQL queries and procedures.
Collaborate with stakeholders and data analysts to understand requirements.
Mentor junior engineers and participate in architecture discussions.
Contribute to CI/CD for data pipelines and code reviews.

Skills

PySpark

Python

SQL

Databricks

Cloud platforms

Data pipelines

Data lake

Tools

Databricks

Airflow

Git

Azure DevOps

GitHub

Job description

Senior Data Engineer - PySpark

Experience: 7-8 Years

Location: Pune (Work from office)

At Aligned Automation, we live by our "Better Together" philosophy to build a better world. As a strategic service provider to Fortune 500 companies, we help digitize enterprise operations and drive impactful business strategies. Our purpose goes beyond projects—we strive to deliver meaningful, sustainable change that shapes a more optimistic and equitable future.

Our culture is deeply rooted in our 4Cs-Care, Courage, Curiosity, and Collaboration ensuring that each employee is empowered to grow, innovate, and thrive in an inclusive workplace.

Senior Data Engineer - PySpark

Experience: 7-8 Years

Location: Pune (Work from office)

Job Summary

We are seeking a highly skilled Senior PySpark Data Engineer with 7-8 years of experience in designing, developing, and optimizing large-scale data engineering solutions. The ideal candidate should have extensive experience with PySpark, Python, SQL, Databricks, cloud platforms, and modern data architectures. The role requires hands-on development, solution design, client interaction, and mentoring junior team members.

Key Responsibilities

Design, develop, and maintain scalable ETL/ELT pipelines using PySpark.
Build high-performance data pipelines to process large volumes of structured and semi-structured data.
Develop reusable frameworks for data ingestion, transformation, validation, and monitoring.
Optimize Spark jobs by tuning partitions, joins, caching, and memory configurations.
Design and implement Data Lake/Lakehouse architectures using Delta Lake.
Write complex SQL queries, stored procedures, CTEs, and window functions.
Collaborate with business stakeholders, architects, and data analysts to understand business requirements.
Participate in architecture discussions and recommend scalable data solutions.
Perform code reviews and enforce coding standards and best practices.
Monitor production pipelines, troubleshoot issues, and implement performance improvements.
Work with DevOps teams to implement CI/CD for data pipelines.
Mentor junior engineers and provide technical leadership.
Estimate effort, prepare technical documentation, and participate in Agile ceremonies.

Required Technical Skills

Programming