Senior Data Engineering

Mphasis; Pune District; On-site; INR 2,000,000 - 3,600,000; Full time

Job summary

Mphasis is seeking an Experienced Data Engineer to design, build, and maintain scalable data pipelines and analytics-ready data products on Databricks and AWS. You will own end-to-end data ingestion, modelling, transformation, quality, and performance optimization.

You will work with Databricks Lakehouse, Delta Lake, and Unity Catalog, writing PySpark and Spark SQL, and integrating AWS services. Strong focus on data governance, security, and agile delivery is expected.

Qualifications

• Bachelor's degree in Computer Science, Engineering, IT, or equivalent.
• 4–8 years in data engineering, ETL/ELT, and cloud data platforms.
• Production-grade pipeline delivery on AWS or similar cloud platforms.

Responsibilities

• Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.
• Ingest data from databases, APIs, event streams, and flat files.
• Design and implement Medallion Architecture (Bronze / Silver / Gold layers).
• Write data transformation logic in PySpark, Spark SQL, and Python.
• Implement data quality checks, validation rules, and reconciliation processes.
• Optimise Spark workloads for performance, cost, and scalability.
• Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.
• Support CI/CD for pipelines using Git, DevOps tooling, and release processes.
• Monitor pipelines, triage failures, and restore services within agreed SLAs.
• Maintain technical documentation: data models, pipeline flows, runbooks.
• Adhere to data security, privacy, and regulatory compliance standards.

Skills

Databricks

Apache Spark

Delta Lake

Medallion Architecture

Python

AWS

Data modelling

CI/CD

Education

Bachelor's in Computer Science / Engineering or equivalent

Tools

Git

Terraform

AWS CloudFormation

dbt

Job description

Job Description For Data Engineer – Databricks Data PlatformRole Overview

Experienced Data Engineer to design, build, and maintain scalable data pipelines and analytics-ready data products on Databricks and AWS. End-to-end ownership of data ingestion, modelling, transformation, quality, and performance optimisation.

Key Responsibilities

• Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.
• Ingest data from databases, APIs, event streams, and flat files.
• Design and implement Medallion Architecture (Bronze / Silver / Gold layers).
• Write data transformation logic in PySpark, Spark SQL, and Python.
• Implement data quality checks, validation rules, and reconciliation processes.
• Optimise Spark workloads for performance, cost, and scalability.
• Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.
• Support CI/CD for pipelines using Git, DevOps tooling, and release processes.
• Monitor pipelines, triage failures, and restore services within agreed SLAs.
• Maintain technical documentation: data models, pipeline flows, runbooks.
• Adhere to data security, privacy, and regulatory compliance standards.

Must-Have Skills

• Databricks: Workflows, Delta Live Tables, Unity Catalog, Databricks SQL.
• Apache Spark: PySpark and Spark SQL for large-scale data processing.
• Delta Lake: ACID transactions, schema evolution, time travel, merge.
• Medallion Architecture or equivalent layered data platform design.
• Python for pipeline development and transformation logic.
• AWS: S3, Glue, IAM, EC2, Lambda.
• Batch, micro-batch, and streaming ingestion (Structured Streaming / Kafka).
• Data modelling: star schema, snowflake schema, Data Vault.
• CI/CD, Git, version control, and Agile delivery practices.

Good-to-Have Skills

• Unity Catalog for data governance and lineage.
• AWS Kinesis for real-time streaming.
• Data Warehouse and Informatica PWC Knowledge
• dbt for SQL-based transformation workflows.
• AWS Redshift or Athena for cloud data warehousing.
• Terraform or AWS CloudFormation for infrastructure-as-code.
• AWS Step Functions for workflow orchestration.
• Data observability tools (Monte Carlo, AWS Glue Catalog, Atlas).
• Industry experience in insurance or financial services.
• Databricks or AWS Data Analytics certification.

Technical Environment

• Platform: Databricks Lakehouse, Delta Lake, Unity Catalog.
• Languages: Python, PySpark, SQL, Spark SQL.
• Ingestion: Auto Loader, APIs, JDBC, CDC tools.
• Processing: Apache Spark, Structured Streaming, Delta Live Tables.
• Cloud: AWS.
• Orchestration: Databricks Workflows, AWS Step Functions.
• DevOps: Git, CI/CD, Terraform, GitHub Actions.
• Analytics: Power BI, Tableau, Looker.

Experience And Qualifications

• Bachelor's in Computer Science, Engineering, IT, or equivalent.
• 4–8 years in data engineering, ETL/ELT, and cloud data platforms.
• Production-grade pipeline delivery on AWS.
• Agile or DevOps-driven delivery environment.