Senior Data Engineer - Databricks & AWS

Mphasis Ltd; Pune District; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Mphasis Ltd is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines on Databricks and AWS, owning ingestion, modelling, transformation, quality, and performance.

You'll work with Python, PySpark, Spark SQL, Delta Lake, and medallion architecture; optimize workloads, implement data quality, and collaborate with DevOps for CI/CD, data security, and regulatory compliance.

Qualifications

• Bachelor's degree in CS/Engineering or equivalent.
• 4–8 years of data engineering experience on cloud data platforms.
• Production-grade pipelines on AWS.
• Experience with Agile/DevOps delivery.

Responsibilities

• Build and maintain data pipelines using Databricks, Spark, Delta Lake on AWS.
• Ingest data from databases, APIs, and flat files.
• Design Medallion Architecture (Bronze / Silver / Gold).
• Write PySpark, Spark SQL, and Python for transformations.
• Implement data quality checks and data validation.
• Optimize Spark workloads for performance and cost.
• Integrate AWS services (S3, Glue, Redshift, IAM) into solutions.
• Support CI/CD for pipelines using Git and DevOps tooling.
• Monitor pipelines and resolve failures against SLAs.
• Maintain data models and runbooks; ensure security and compliance.

Skills

Databricks

Spark

Delta Lake

Medallion Architecture

Python

AWS

CI/CD

Git

SQL

Data modelling

Education

Bachelor's in Computer Science or Engineering

Tools

Databricks Lakehouse

Unity Catalog

Terraform

AWS Glue Catalog

Job description

Senior Data Engineering Job Description for Data Engineer – Databricks Data Platform

Role Overview Experienced Data Engineer to design, build, and maintain scalable data pipelines and analytics-ready data products on Databricks and AWS. End-to-end ownership of data ingestion, modelling, transformation, quality, and performance optimisation.

Key Responsibilities

• Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.
• Ingest data from databases, APIs, event streams, and flat files.
• Design and implement Medallion Architecture (Bronze / Silver / Gold layers).
• Write data transformation logic in PySpark, Spark SQL, and Python.
• Implement data quality checks, validation rules, and reconciliation processes.
• Optimise Spark workloads for performance, cost, and scalability.
• Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.
• Support CI/CD for pipelines using Git, DevOps tooling, and release processes.
• Monitor pipelines, triage failures, and restore services within agreed SLAs.
• Maintain technical documentation: data models, pipeline flows, runbooks.
• Adhere to data security, privacy, and regulatory compliance standards.

Must-Have Skills

• Databricks: Workflows, Delta Live Tables, Unity Catalog, Databricks SQL.
• Apache Spark: PySpark and Spark SQL for large-scale data processing.
• Delta Lake: ACID transactions, schema evolution, time travel, merge.
• Medallion Architecture or equivalent layered data platform design.
• Python for pipeline development and transformation logic.
• AWS: S3, Glue, IAM, EC2, Lambda.
• Batch, micro-batch, and streaming ingestion (Structured Streaming / Kafka).
• Data modelling: star schema, snowflake schema, Data Vault.
• CI/CD, Git, version control, and Agile delivery practices.

Good-to-Have Skills

• Unity Catalog for data governance and lineage.
• AWS Kinesis for real-time streaming.
• Data Warehouse and Informatica PWC Knowledge.
• dbt for SQL-based transformation workflows.
• AWS Redshift or Athena for cloud data warehousing.
• Terraform or AWS CloudFormation for infrastructure-as-code.
• AWS Step Functions for workflow orchestration.
• Data observability tools (Monte Carlo, AWS Glue Catalog, Atlas).
• Industry experience in insurance or financial services.
• Databricks or AWS Data Analytics certification.

Technical Environment

• Platform: Databricks Lakehouse, Delta Lake, Unity Catalog.
• Languages: Python, PySpark, SQL, Spark SQL.
• Ingestion: Auto Loader, APIs, JDBC, CDC tools.
• Processing: Apache Spark, Structured Streaming, Delta Live Tables.
• Cloud: AWS.
• Orchestration: Databricks Workflows, AWS Step Functions.
• DevOps: Git, CI/CD, Terraform, GitHub Actions.
• Analytics: Power BI, Tableau, Looker.

Experience and Qualifications

• Bachelor's in Computer Science, Engineering, IT, or equivalent.
• 4–8 years in data engineering, ETL/ELT, and cloud data platforms.
• Production-grade pipeline delivery on AWS.
• Agile or DevOps-driven delivery environment.