Senior Data Engineer – Databricks & AWS

RefRelay Private Limited; Pune District; On-site; INR 3,000,000 - 4,500,000; Full time

Job summary

RefRelay Private Limited in Pune is seeking an experienced Senior Data Engineer to design, develop, and optimise scalable data ingestion and processing pipelines using Databricks, PySpark, and AWS services. You will implement Medallion Architecture and Lakehouse solutions, leverage Unity Catalog for governance, and collaborate with data science and analytics teams to deliver reliable data platforms.

Strong hands-on problem-solving and ability to build cost-efficient, secure data platforms are

Qualifications

• Databricks and PySpark skills relevant to building scalable data pipelines.
• AWS Streaming and ETL services expertise.
• Unity Catalog and MLflow knowledge for governance and lifecycle management.
• Medallion Architecture and Lakehouse knowledge.
• SQL, data modelling, and data lake concepts.
• Experience with enterprise-scale data ingestion and processing pipelines.

Responsibilities

• Design and implement scalable batch and streaming data pipelines using Databricks and PySpark.
• Develop cloud-native data ingestion and ETL solutions using AWS streaming and data processing services.
• Implement Medallion Architecture (Bronze, Silver, Gold) and modern Lakehouse solutions.
• Utilise Unity Catalog for data governance, security, lineage, and metadata management.
• Integrate and support MLflow for machine learning lifecycle management and analytics use cases.
• Focus on pipeline performance, reliability, scalability, data quality, and cost efficiency.
• Collaborate with data science, analytics, and business teams to deliver reliable data solutions.
• Participate in technical design, code reviews, CI/CD, and engineering best practices.

Skills

Databricks

PySpark

SQL

Data Modelling

Data Lake concepts

Medallion Architecture

Lakehouse Architecture

AWS Streaming

MLflow

Unity Catalog

Tools

Databricks

PySpark

AWS

Unity Catalog

MLflow

Medallion Architecture

Lakehouse Architecture

Job description

We are seeking an experienced Senior Data Engineer with strong hands-on expertise in designing, developing, and optimising scalable data ingestion and processing pipelines using Databricks, PySpark, and AWS services.

Key Responsibilities

• Design and implement scalable batch and streaming data pipelines using Databricks and PySpark.
• Develop cloud-native data ingestion and ETL solutions using AWS streaming and data processing services.
• Implement Medallion Architecture (Bronze, Silver, and Gold layers) and modern Lakehouse solutions.
• Leverage Unity Catalogue for data governance, security, lineage, and metadata management.
• Integrate and support MLflow for machine learning lifecycle management and analytics use cases.
• pipeline performance, reliability, scalability, data quality, and cost efficiency.
• Collaborate with data science, analytics, and business teams to deliver reliable data solutions.
• Participate in technical design, code reviews, CI/CD, and engineering best practices.

Required Skills & Expertise

• Databricks and PySpark
• AWS Streaming and ETL Services
• Unity Catalog and MLflow
• Medallion Architecture and Lakehouse Architecture
• SQL, Data Modelling, and Data Lake concepts
• Strong experience in enterprise-scale data ingestion and processing pipelines

Good to Have

• Python scripting and automation
• Pandas for data transformation and analysis
• Workflow orchestration using Airflow or Databricks Workflows
• Exposure to DataOps/MLOps practices and CI/CD implementation

Ideal Candidate

A hands-on Data Engineer with 6–8 years of relevant experience, strong problem-solving skills, and the ability to build secure, governed, scalable, and cost-efficient data platforms.