Senior Data Engineer

Tekskills

Pune District, Bengaluru, Delhi

Hybrid

INR 1,500,000 - 2,100,000

Full time

7 days ago

Be an early applicant

Application generator

A complete application in a minute — tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Tekskills is seeking a Senior Data Engineer to lead migration of ETL/ELT pipelines to Databricks, with strong PySpark, SQL, AWS experience. The role involves designing scalable batch and near-real-time data workflows, integrating diverse data sources, and ensuring governance and observability.

You will collaborate with analytics and data teams, optimize pipelines for performance and cost, and implement CI/CD for data workflows while maintaining documentation.

Qualifications

Strong proficiency in SQL and relational databases (e.g., Oracle, PostgreSQL).
ETL/ELT development and migration experience.
Strong programming in Python and/or Scala.
Hands-on experience with Apache Spark / PySpark.
Hands-on with Databricks – Delta Lake, Notebooks, Jobs, Workflows.
Knowledge of AWS data engineering services: Glue, Lambda, S3, EventBridge, Redshift, Kinesis, CloudWatch.

Responsibilities

Analyze existing ETL pipelines, data sources, and transformation logic.
Drive end-to-end migration of data pipelines to Databricks.
Design, develop, and maintain scalable batch and near-real-time data pipelines.
Re-engineer legacy ETL/ELT workflows using Spark, PySpark, SQL, and Databricks.
Integrate data from multiple heterogeneous sources.
Implement robust data transformation and processing logic.
Perform data validation, reconciliation, and consistency checks after migration.
Optimize pipelines for performance, scalability, and cost efficiency.
Collaborate with business stakeholders, analysts, and data teams to understand requirements.
Support reporting, analytics, and downstream applications with reliable datasets.
Monitor production pipelines, troubleshoot failures, and resolve data issues.
Implement logging, alerting, and observability using CloudWatch / Databricks monitoring.
Follow data governance, security, and compliance standards.
Implement CI/CD for data workflows and maintain technical documentation.

Skills

SQL

PySpark

Databricks

AWS

ETL/ELT

Python

Scala

Spark

Data Modeling

Tools

Delta Lake

Notebooks

Jobs/Workflows

Job description

Role & responsibilities

Hiring: Senior Data Engineer Databricks / AWS / PySpark

Location: Pan India Experience: 6+ Years Relevant Experience: 4+ Years

We are looking for an experienced Data Engineer with strong expertise in Databricks, PySpark, SQL, AWS, and ETL/ELT pipeline migration to join our team.

Key Responsibilities

Analyze existing ETL pipelines, data sources, and transformation logic across platforms
Drive end-to-end migration of data pipelines to Databricks
Design, develop, and maintain scalable batch and near-real-time data pipelines
Re-engineer legacy ETL/ELT workflows using Spark, PySpark, SQL, and Databricks
Integrate data from multiple heterogeneous sources
Implement robust data transformation and processing logic
Perform data validation, reconciliation, and consistency checks after migration
Optimize pipelines for performance, scalability, and cost efficiency
Collaborate with business stakeholders, analysts, and data teams to understand requirements
Support reporting, analytics, and downstream applications with reliable datasets
Monitor production pipelines, troubleshoot failures, and resolve data issues
Implement logging, alerting, and observability using CloudWatch / Databricks monitoring
Follow data governance, security, and compliance standards
Implement CI/CD for data workflows and maintain technical documentation