Senior Data Engineer

Orcapod Consulting Services; Gurugram District; On-site; INR 1,500,000 - 3,200,000; Full time

Job summary

EPAM is seeking an experienced Data Engineer with strong expertise in Spark, Python, Databricks, AWS, Snowflake, and Kafka to design, develop, and optimize scalable data pipelines for enterprise applications.

You will work on data ingestion, transformation, and real-time streaming, building robust data solutions and ensuring data quality across systems. Collaboration with cross-functional teams is required to deliver scalable results.

Qualifications

• Strong Python development experience.
• Extensive hands-on with Apache Spark / PySpark.
• Experience with Databricks platforms and Delta Lake.
• Solid knowledge of AWS cloud services and data storage.
• Good SQL and data warehousing understanding.
• Experience with Snowflake and Kafka is a plus.
• Strong ETL/ELT design and data integration capabilities.

Responsibilities

• Design scalable data pipelines using Python and Spark.
• Develop and maintain Databricks-based solutions.
• Build cloud data solutions with AWS services.
• Design storage and processing with Snowflake.
• Implement streaming data pipelines with Kafka.
• Ingest, transform, cleanse, and validate data from sources.
• Optimize Spark jobs and SQL queries for performance.
• Create reusable ETL/ELT frameworks with data quality.
• Collaborate with cross-functional teams to meet data needs.
• Troubleshoot pipeline issues and perform root-cause analysis.
• Follow best practices for security, monitoring, and governance.

Skills

Python

Apache Spark / PySpark

Databricks

AWS

SQL / data warehousing

Snowflake

Kafka

ETL / ELT

Tools

Databricks Delta Lake

AWS Cloud Services

Job description

Skills : Spark, Python, Databricks, AWS, Snowflake OR Kafka

Job Summary

EPAM is looking for an experienced Data Engineer with strong expertise in Spark, Python, Databricks, AWS, Snowflake, and/or Kafka. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data processing solutions for enterprise applications.

Key Responsibilities

• Design and develop scalable data pipelines using Python and Apache Spark/PySpark.
• Develop and maintain data engineering solutions on Databricks.
• Build and manage cloud-based data solutions using AWS services.
• Design and optimize data storage and processing solutions using Snowflake.
• Implement real-time/streaming data pipelines using Apache Kafka.
• Perform data ingestion, transformation, cleansing, and validation from multiple sources.
• Optimize Spark jobs, Databricks workflows, SQL queries, and data pipelines for performance.
• Develop reusable ETL/ELT frameworks and ensure data quality and reliability.
• Work with cross-functional teams to understand data requirements and deliver scalable solutions.
• Troubleshoot production data pipeline issues and perform root-cause analysis.
• Follow best practices for security, monitoring, documentation, and data governance.

Must-Have Skills

• Strong hands‑on experience with Python.
• Strong experience with Apache Spark / PySpark.
• Hands‑on experience with Databricks.
• Experience with AWS cloud services.
• Good knowledge of SQL and data warehousing concepts.
• Experience with Snowflake and/or Kafka.
• Strong understanding of ETL/ELT, data pipelines, data transformation, and data integration.

Good to Have

• AWS services such as S3, Glue, EMR, Lambda, Redshift, and CloudWatch.
• Databricks Delta Lake, Workflows, and Unity Catalog.
• Kafka topics, producers, consumers, and streaming.
• CI/CD and DevOps practices.
• Agile/Scrum experience.