Senior Software Engineer- Data

NewVision Software; Pune District; Hybrid; INR 1.200.000 - 1.800.000; Vollzeit

Zusammenfassung

New Vision Softcom & Consultancy Pvt. Ltd in Pune is seeking a Senior Databricks Engineer to design and optimize data pipelines on Databricks using PySpark, Spark SQL, and Delta Lake. You will work across cloud platforms (Azure/AWS/GCP) and collaborate with cross-functional teams.

The role requires deep Spark expertise, experience with streaming/incremental processing, and strong Python/SQL skills. Hybrid work setup and an APAC focus are offered.

Qualifikationen

• Senior Databricks Engineer with strong expertise in data engineering on Databricks.
• Deep experience with Apache Spark (PySpark), Delta Lake, and Spark optimization.
• Hands-on in building scalable Spark-based data pipelines and batch/incremental processing.
• Proficient in Python, PySpark, and SQL; experience with cloud Databricks platforms.
• Experience with Git, CI/CD, testing, and production deployments.

Aufgaben

• Develop, optimize, and maintain data pipelines on Databricks.
• Tune PySpark workloads for performance and cost efficiency.
• Architect Delta Lake tables using Bronze/Silver/Gold medallion layers.
• Collaborate with stakeholders to translate requirements into scalable solutions.
• Mentor junior developers and support solution design in sprints.

Kenntnisse

Databricks notebooks

PySpark

Spark SQL

Delta Lake

Spark performance tuning

Azure Databricks

AWS Databricks

GCP Databricks

Python

SQL

Kafka

REST APIs

Git / CI/CD

Data Modeling

Unity Catalog

Ausbildung

Graduation/Equivalent Course

Tools

Databricks

Jobbeschreibung

Company Name New Vision Softcom & Consultancy Pvt. Ltd

Function Business Units (BU)

Department/Practice AI & Analytics

Region APAC

Country India

Base office location Pune

Working model Hybrid

Weekly Off Pune office Standard

State Maharashtra

Skills

Skill

Graduation/Equivalent Course

CERTIFICATION

Working Language

Job Description

The earlier req for this position was closed with -ve BGV candidate. Thus, opening a new one.

Role Summary:

Senior Databricks Engineer with strong expertise in developing, optimizing, and maintaining data engineering solutions on the Databricks platform. The candidate should have deep experience with Apache Spark (PySpark), Databricks notebooks, Delta Lake, and Spark performance optimization. This role is focused on Spark-native data engineering rather than ETL tool development.

Must Have Skills:

• Strong hands-on experience with Databricks notebooks, Jobs, Workflows, Repos, and Unity Catalog
• Strong expertise in PySpark and Spark SQL
• Experience building scalable Spark-based data pipelines
• Strong understanding of Spark DataFrames, joins, aggregations, window functions, and distributed processing
• Experience with Delta Lake (MERGE, OPTIMIZE, ZORDER, VACUUM, Change Data Feed)
• Hands-on experience in Spark job performance tuning
• Experience optimizing partitions, shuffles, caching, memory usage, and cluster utilization
• Experience developing batch and incremental data processing pipelines
• Strong programming skills in Python, PySpark, and SQL
• Experience working on Azure Databricks, AWS Databricks, or GCP Databricks
• Experience with Git, CI/CD, code reviews, testing, and production deployments
• Implement event-driven data flows and real-time data ingestion using Apache Kafka.
• Develop and integrate REST APIs for both data extraction and downstream data serving.
• Monitor, tune, and optimize PySpark workloads for maximum performance and cost efficiency.
• Communicate effectively with cross-functional stakeholders to gather requirements and translate them into scalable technical solutions.
• Mentor junior developers, guide architectural decisions, and prepare to take on team lead responsibilities in the future. as nice to have.

Good to Have Skills:

• Experience with Lakehouse and Medallion architecture
• Knowledge of Data Modeling and Data Governance

Nice to Have Skills:

• Experience with Delta Live Tables (DLT)
• Experience mentoring teams or leading technical discussions
• Experience with Photon and Databricks Asset Bundles
• Experience with Cluster Policies and Workspace Administration
• Design, build, and maintain robust batch and streaming data pipelines using Databricks and PySpark.
• Extract complex data from a variety of upstream sources (databases, internal/external APIs, flat files) and process it into the data lake.
• Architect and optimize Delta Tables, utilizing the Medallion architecture (Bronze, Silver, Gold layers) for structured data governance.
• Must have strong written and verbal communication skills, excellent articulation skills, and experience working with client teams.

The candidate should have actively participated in and contributed to sprint planning and solution design.