Senior Data Engineer (Databricks & AWS Migration) - 573433

NITYO; India; Hybrid; INR 700,000 - 1,200,000; Full time

Job summary

NITYO is seeking a Senior Data Engineer to lead migration of data pipelines to a Databricks/AWS platform. You will analyze current ETL processes, design scalable pipelines, and re-engineer legacy workflows for efficiency and reliability.

The role requires 6+ years of experience with PySpark/SQL, Databricks, and AWS data services, and offers a hybrid work arrangement across India. Close collaboration with data teams and stakeholders is essential.

Qualifications

• 6+ years of data engineering experience.
• Strong SQL and relational DB skills.
• Hands-on ETL/ELT development and migrations.
• Proficient in Python and/or Scala.
• Hands-on Spark experience (PySpark) and Databricks.
• Experience with AWS data services (Glue/Lambda/S3/Redshift).
• Knowledge of data modeling, warehousing, and lake architectures.
• Experience with large-scale datasets and distributed processing.
• Experience with orchestration/workflow tools.

Responsibilities

• Migrate data pipelines to a new platform using Databricks.
• Analyze existing ETL pipelines, data sources, and transformation logic.
• Design, build and maintain scalable data pipelines for batch and near real-time processing.
• Re-engineer legacy ETL/ELT workflows into modular pipelines using Spark/Databricks.
• Integrate data from multiple sources ensuring data flow across systems.
• Implement data transformation logic using PySpark/SQL on Databricks.
• Validate migrated pipelines for data accuracy and reconciliation.
• Optimize pipelines for performance, scalability, and cost in cloud environments.
• Collaborate with business stakeholders and data teams to understand data needs.
• Support reporting and analytics by providing high-quality datasets.
• Monitor pipelines, troubleshoot failures, and ensure observability (CloudWatch/Databricks).
• Follow data governance, security, and compliance during migration.
• Implement CI/CD pipelines for data workflows and document processes.

Skills

SQL

Python

Scala

Apache Spark

Databricks

AWS data services

Data modeling

Data warehousing

Data lake

ETL/ELT

Pipeline orchestration

CI/CD

Logging/Monitoring

Tools

Databricks

Delta Lake

Notebooks

Jobs/Workflows

AWS Glue

AWS Lambda

S3

EventBridge

SQS

Redshift

CloudWatch

Job description

Senior Data Engineer (Databricks & AWS Migration)

This is a Senior Data Engineer role that involves migrating data pipelines to a new platform using Databricks and AWS. The candidate will analyze and understand existing ETL pipelines, data sources, and transformation logic across current platforms.

Key Responsibilities

• Analyze and understand existing ETL pipelines, data sources, and transformation logic across current platforms
• Drive end-to-end migration of data pipelines to the new platform using Databricks
• Design, build, and maintain scalable data pipelines for batch and near real-time processing in the new platform
• Re-engineer legacy ETL/ELT workflows into optimized, scalable, and modular pipelines using Spark/Databricks
• Integrate data from multiple heterogeneous sources and ensure seamless data flow across systems
• Implement data transformation logic using PySpark/SQL on Databricks and ensure adherence to best practices
• Validate migrated pipelines to ensure data accuracy, reconciliation, and consistency with source systems
• Optimize pipelines for performance, scalability, and cost efficiency within cloud environments
• Work closely with business stakeholders, analysts, and data teams to understand data requirements and use cases
• Support reporting, analytics, and downstream applications by providing reliable and high-quality datasets
• Monitor production pipelines, troubleshoot failures, and proactively resolve data-related issues
• Ensure proper logging, alerting, and observability for all pipelines (using tools like CloudWatch / Databricks monitoring)
• Follow data governance, security, and compliance standards during data handling and migration
• Implement CI/CD pipelines for data workflows and maintain proper documentation of pipelines and processes

Requirements

• Strong proficiency in SQL and relational databases (Oracle, PostgreSQL, etc.)
• Hands-on experience with ETL/ELT pipeline development and migration projects
• Strong programming skills in Python and/or Scala
• Good experience with Apache Spark (preferably PySpark)
• Hands-on experience with Databricks (Delta Lake, notebooks, jobs, workflows)
• Strong knowledge of AWS data engineering services including:
• AWS Glue, Lambda, S3, EventBridge, SQS
• Amazon Redshift, Firehose, CloudWatch
• Strong understanding of data modeling, data warehousing concepts, and data lake architectures
• Experience working with large-scale datasets and distributed processing
• Knowledge of pipeline orchestration and workflow management

Work Arrangements

• Hybrid work arrangement: Face-to-face interactions required (Pan India)
• 6+ years of experience
• Contract type