Big Data Engineer

TymblHub; Bengaluru; On-site; INR 1,500,000 - 3,000,000; Full time

Job summary

EPAM Systems in Bengaluru, India is seeking a Big Data Engineer with 6–11 years of experience to design robust batch and real-time data pipelines using Spark/PySpark and Databricks. The role covers ETL/ELT design, data modeling, and building scalable data platforms with a focus on governance and cost optimization.

You will architect Databricks Lakehouse solutions, optimize Spark workloads, and integrate with AWS services to support analytics, ML, and AI workloads.

Qualifications

• Experience designing batch and real-time data pipelines.
• Proficient in Spark/PySpark and Databricks.
• Knowledge of cloud data architectures on AWS.

Responsibilities

• Design robust batch and real-time data processing pipelines using Spark/PySpark and Databricks.
• Design and implement scalable ETL/ELT pipelines for varied data types.
• Establish data architecture standards for scalability, reliability, and security.
• Architect Databricks Lakehouse solutions and optimize Spark workloads.
• Design data ingestion, transformation, orchestration, storage and serving patterns.

Skills

Spark PySpark

ETL/ELT

Data pipelines

Data modeling

Data orchestration

Tools

Databricks

Delta Lake

AWS S3

AWS Lambda

Amazon EKS

DynamoDB

API Gateway

IAM

Job description

# Big Data EngineerEPAM SystemsPosted on October 5, 2026---Experience6 - 11 yrsSalary (CTC)₹15L - ₹30LJob LocationBengaluru, IndiaVacancy1DesignationBig Data EngineerJob TypeONSITE---## Job DescriptionRoles and Responsibilities* Design robust **batch and real-time data processing pipelines** using Apache Spark/PySpark and Databricks.* Design and implement scalable **ETL/ELT pipelines** for structured, semi-structured, and unstructured data.* Define data ingestion, transformation, orchestration, storage, and serving patterns.* Establish data architecture standards covering scalability, reliability, security, performance, and maintainability.* Design data models, data processing frameworks, and reusable data engineering components.* Optimize large-scale data processing workloads for performance and cost.Databricks & Lakehouse* Architect and implement **Databricks Lakehouse** solutions.* Develop production-grade Spark/PySpark workloads using Databricks.* Design data pipelines supporting analytics, ML, and AI workloads.* Work with Delta Lake and modern Lakehouse architecture patterns.* Implement data quality, validation, lineage, monitoring, and governance mechanisms.* Optimize Spark jobs, cluster configurations, partitioning, caching, and data storage strategies.* Integrate Databricks with AWS services and enterprise data platforms.AWS Cloud* Design cloud-native data solutions using AWS.* Work extensively with services such as:* Amazon S3* AWS Lambda* Amazon EKS* DynamoDB* API Gateway* IAM* Event-driven AWS services* Design secure and highly available data architectures.* Implement cloud-native patterns for scalability, fault tolerance, and disaster recovery.* Drive **AWS cost optimization / FinOps** for data workloads.Data Pipelines & Integration* Build reliable and reusable data ingestion and transformation frameworks.* Integrate data from APIs, databases, files, event streams, and enterprise applications.* Implement incremental processing, CDC, schema evolution, error handling, retries, and reconciliation.* Establish pipeline monitoring and operational processes.* Build data pipelines capable of supporting both analytical and ML workloads.Candidates with experience in the following areas will be preferred:* MLflow and model lifecycle management.* Data pipelines supporting model training and inference.* Retrieval-Augmented Generation (**RAG**) data pipelines.* Vector databases such as **pgVector, Pinecone, or Weaviate**.* Semantic search and embedding pipelines.* GenAI data preparation