Senior Specialist - Data Engineering

Vivantify Technology Solutions India Pvt. Ltd.; Maharashtra; On-site; INR 1,500,000 - 2,100,000; Full time

Benefits offered by this job

Competitive salary

Growth opportunities

Collaborative environment

Challenging projects

Job summary

Vivantify Technology Solutions India Pvt. Ltd. in Pune invites an experienced Data Engineer to build scalable data platforms using Databricks, Delta Lake, and Lakehouse on AWS.

You will work with PySpark, Scala, Python, and SQL to deliver ETL pipelines, data modeling, and BI reporting in an Agile environment. Strong AWS Big Data services experience is required.

Qualifications

• 8+ years of Data Engineering experience with Databricks and Spark stacks.
• Experience building scalable data pipelines and data models.

Responsibilities

• Design and develop scalable data engineering solutions using Databricks, Spark, PySpark, Scala, and Python.
• Build and maintain data pipelines with Delta Lake in Lakehouse architectures.
• Develop complex SQL queries and data models across relational databases.

Skills

Databricks

Apache Spark

PySpark

Scala

Python

SQL

Power BI

Delta Lake

AWS

S3

EMR

Kinesis

PostgreSQL

Git

CI/CD

Unity Catalog

Tools

Azure DevOps

GitHub

Job description

About The Role

We are seeking an experienced

Location:

Pune

Experience:

7–11 Years

About The Role

We are seeking an experienced Data Engineer with 8+ years of expertise in Big Data Engineering, Databricks, Apache Spark, PySpark, Scala, and SQL. The role focuses on building and delivering scalable data solutions using Databricks, Delta Lake, Lakehouse architecture, AWS, and Power BI in a fast-paced Agile environment.

Job Description

The role involves designing, developing, and supporting modern data engineering solutions using Databricks and Apache Spark, with strong emphasis on PySpark, Scala, Python, SQL, and data modeling. The candidate will work with Delta Lake and Lakehouse architecture to build scalable distributed data processing solutions.

The position requires hands-on experience with AWS Big Data services, including S3, Glue, EMR, DMS, Athena, RDS, Redshift, and Kinesis, as well as experience managing EMR clusters and working with relational databases such as PostgreSQL, Oracle, and SQL Server.

The candidate will also contribute to data orchestration, streaming, CI/CD, source control, data cataloguing, and Power BI solutions. Experience with Agile at Scale, Continuous Integration, Test-Driven Development, and Infrastructure as Code practices will support successful delivery.

Key Responsibilities

• Design and develop scalable data engineering solutions using Databricks, Apache Spark, PySpark, Scala, and Python.
• Build and maintain data pipelines using Delta Lake and Lakehouse architecture.
• Develop complex SQL queries and implement effective data models across relational databases.
• Work with AWS Big Data services and manage EMR clusters and associated services.
• Develop data visualization and reporting solutions using Power BI.
• Implement data orchestration, streaming, CI/CD, and source control practices.
• Work with data cataloguing solutions such as Unity Catalog.
• Collaborate with stakeholders in Agile environments and take ownership of data engineering deliverables.

Required Skills

• Databricks
• Apache Spark, PySpark, and Scala
• Python
• SQL and Data Modelling
• Power BI
• Delta Lake and Lakehouse architecture
• AWS cloud platform
• AWS services: S3, Glue, EMR, DMS, Athena, RDS, Redshift, Kinesis
• EMR cluster management
• Data orchestration tools
• Kafka / streaming frameworks
• RDBMS: PostgreSQL, Oracle, SQL Server
• Complex SQL queries, particularly in PostgreSQL
• Git and CI/CD pipelines
• Azure DevOps / GitHub
• Unity Catalog
• Distributed computing principles
• Agile, Continuous Integration, Test-Driven Development, and Infrastructure as Code practices

Preferred / Good To Have

• Data warehousing concepts
• Experience supporting Machine Learning pipelines in Databricks
• MLflow experiment tracking and model registry
• Databricks Mosaic AI, Vector Search, and embeddings pipelines
• Generative AI concepts including LLMs, embeddings, and RAG
• Terraform or CloudFormation
• Debezium or other CDC tools
• Jenkins-based workflow orchestration

Mandatory Skills

• Databricks

Candidate Profile

The ideal candidate will have 8+ years of Data Engineering experience, including strong hands-on expertise in Databricks, Spark, Python, SQL, AWS, and modern data architecture. The candidate should possess strong analytical and problem-solving abilities, communicate effectively with stakeholders, and demonstrate ownership while working in a fast-paced Agile environment.

Benefits Package

• Competitive salary and benefits package
• Opportunities for professional growth and development
• A collaborative and inclusive work environment
• The opportunity to work on exciting and challenging projects with leading clients