Data Engineer

Epam Systems; Pune District, Chennai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Epam Systems is seeking a skilled Data Engineer to design and maintain scalable data pipelines using Apache Spark and Python. The role involves cloud-based data platforms on AWS, Databricks, and Google Cloud Platform, enabling BI, analytics, and data-driven applications.

You will collaborate with data scientists, analysts, and stakeholders to deliver robust ETL/ELT workflows, ensure data quality, and optimize performances and costs across the pipeline lifecycle.

Qualifications

• Experience with Apache Spark, Python, and cloud-based data platforms.
• Hands-on development of scalable data pipelines and ETL/ELT workflows.
• Familiarity with AWS, Databricks, and Google Cloud Platform.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Spark and Python.
• Build and optimize batch and distributed data processing applications.
• Develop ETL/ELT workflows for data ingestion, transformation, cleansing, and integration.
• Implement data engineering solutions using AWS cloud services and Databricks.
• Develop and manage data processing workflows on Google Cloud Platform (GCP).
• Optimize Spark jobs for performance, scalability, resource utilization, and cost efficiency.
• Work with structured, semi-structured, and unstructured datasets.
• Implement data quality checks, error handling, logging, and monitoring.
• Collaborate with data scientists, analysts, architects, and business stakeholders.
• Follow best practices for data security, governance, access control, and cloud resource management.
• Troubleshoot production issues and ensure the reliability and availability of data pipelines.

Skills

Apache Spark

Python

AWS

Databricks

GCP

Tools

Databricks

Spark

Python

Job description

Job Summary

We are looking for a skilled Data Engineer with strong expertise in Apache Spark, Python, and cloud-based data engineering platforms. The ideal candidate will have hands-on experience developing scalable data pipelines, processing large datasets, and implementing data solutions on AWS, AWS Databricks, and Google Cloud Platform (GCP).

The candidate will be responsible for designing, developing, optimizing, and maintaining robust data processing workflows to support business intelligence, analytics, and data-driven applications.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark and Python.


• Build and optimize batch and distributed data processing applications.


• Develop ETL/ELT workflows for data ingestion, transformation, cleansing, and integration.


• Implement data engineering solutions using AWS cloud services and Databricks.


• Develop and manage data processing workflows on Google Cloud Platform (GCP).


• Optimize Spark jobs for performance, scalability, resource utilization, and cost efficiency.


• Work with structured, semi-structured, and unstructured datasets.


• Implement data quality checks, error handling, logging, and monitoring.


• Collaborate with data scientists, analysts, architects, and business stakeholders.


• Follow best practices for data security, governance, access control, and cloud resource management.


• Troubleshoot production issues and ensure the reliability and availability of data pipelines.