Senior Specialist - Data Engineering

Vivantify Technology Solutions India Private Limited; Pune District; On-site; INR 1,800,000 - 3,200,000; Full time

Benefits offered by this job

Competitive salary and benefits

Collaborative and inclusive team

Growth and development opportunities

Job summary

Vivantify Technology Solutions India Private Limited in Pune is seeking an experienced Data Engineer with 8+ years of expertise in Big Data, Databricks, Spark, PySpark, Scala, and SQL. The role focuses on building scalable data solutions using Databricks, Lakehouse architecture, AWS, and Power BI in an Agile environment.

The candidate will design, develop, and support data engineering pipelines, work with Delta Lake, Unity Catalog, and BI reporting, and collaborate with stakeholders to deliver

Qualifications

• 8+ years of Data Engineering experience with hands-on Databricks/Spark stack.
• Strong SQL data modeling and complex query skills.
• Experience with AWS Big Data services and EMR cluster management.

Responsibilities

• Design and develop scalable data engineering solutions using Databricks, Spark, PySpark, Scala, and Python.
• Build and maintain data pipelines with Delta Lake and Lakehouse architecture.
• Develop complex SQL queries and data models across relational databases.
• Work with AWS Big Data services; manage EMR clusters and related services.
• Develop Power BI reporting and data visualization solutions.
• Implement data orchestration, streaming, CI/CD, and source control practices.
• Utilize Unity Catalog for data cataloguing and governance.
• Collaborate with stakeholders in Agile environments and deliver data engineering成果.

Skills

Databricks

Apache Spark

PySpark

Scala

Python

SQL

Power BI

Delta Lake

Lakehouse

AWS

S3

Glue

EMR

Kinesis

PostgreSQL

Oracle

SQL Server

Git

CI/CD

Unity Catalog

Kafka

Azure DevOps

Tools

Databricks

Spark

Job description

Location: Pune

Experience: 7–11 Years

About the Role

We are seeking an experienced Data Engineer with 8+ years of expertise in Big Data Engineering, Databricks, Apache Spark, PySpark, Scala, and SQL. The role focuses on building and delivering scalable data solutions using Databricks, Delta Lake, Lakehouse architecture, AWS, and Power BI in a fast-paced Agile environment.

Job Description

The role involves designing, developing, and supporting modern data engineering solutions using Databricks and Apache Spark, with strong emphasis on PySpark, Scala, Python, SQL, and data modeling. The candidate will work with Delta Lake and Lakehouse architecture to build scalable distributed data processing solutions.

The position requires hands‑on experience with AWS Big Data services, including S3, Glue, EMR, DMS, Athena, RDS, Redshift, and Kinesis, as well as experience managing EMR clusters and working with relational databases such as PostgreSQL, Oracle, and SQL Server.

The candidate will also contribute to data orchestration, streaming, CI/CD, source control, data cataloguing, and Power BI solutions. Experience with Agile at Scale, Continuous Integration, Test‑Driven Development, and Infrastructure as Code practices will support successful delivery.

Key Responsibilities

• Design and develop scalable data engineering solutions using Databricks, Apache Spark, PySpark, Scala, and Python.
• Build and maintain data pipelines using Delta Lake and Lakehouse architecture.
• Develop complex SQL queries and implement effective data models across relational databases.
• Work with AWS Big Data services and manage EMR clusters and associated services.
• Develop data visualization and reporting solutions using Power BI.
• Implement data orchestration, streaming, CI/CD, and source control practices.
• Work with data cataloguing solutions such as Unity Catalog.
• Collaborate with stakeholders in Agile environments and take ownership of data engineering deliverables.

Required Skills

• Databricks
• Apache Spark, PySpark, and Scala
• Python
• SQL and Data Modelling
• Power BI
• Delta Lake and Lakehouse architecture
• AWS cloud platform
• AWS services: S3, Glue, EMR, DMS, Athena, RDS, Redshift, Kinesis
• EMR cluster management
• Data orchestration tools
• Kafka / streaming frameworks
• RDBMS: PostgreSQL, Oracle, SQL Server
• Complex SQL queries, particularly in PostgreSQL
• Git and CI/CD pipelines
• Azure DevOps / GitHub
• Unity Catalog
• Distributed computing principles
• Agile, Continuous Integration, Test‑Driven Development, and Infrastructure as Code practices

Preferred / Good to Have

• Data warehousing concepts
• Experience supporting Machine Learning pipelines in Databricks
• MLflow experiment tracking and model registry
• Databricks Mosaic AI, Vector Search, and embeddings pipelines
• Generative AI concepts including LLMs, embeddings, and RAG
• Terraform or CloudFormation
• Debezium or other CDC tools
• Jenkins‑based workflow orchestration

Mandatory Skills

• Databricks

Candidate Profile

The ideal candidate will have 8+ years of Data Engineering experience, including strong hands‑on expertise in Databricks, Spark, Python, SQL, AWS, and modern data architecture. The candidate should possess strong analytical and problem‑solving abilities, communicate effectively with stakeholders, and demonstrate ownership while working in a fast‑paced Agile environment.

Benefits Package

• Competitive salary and benefits package
• Opportunities for professional growth and development
• A collaborative and inclusive work environment
• The opportunity to work on exciting and challenging projects with leading clients