Lead Data Engineer

V2 Solutions; Hyderabad; On-site; INR 4,020,000 - 4,440,000; Full time

Job summary

EPAM Systems in Hyderabad, India is seeking a Lead Data Engineer with 5+ years of Big Data experience. The role focuses on building cloud-based data pipelines, ODS architectures, and scalable data platforms using Spark, Spark Streaming, Python or Scala. You will lead a team and work with analytics and IS architecture teams to enable data-driven decisions.

The position is onsite with hybrid flexibility once established, offering vast opportunities for growth and global projects across geographies.

Qualifications

• 5+ years of experience in Big Data and data-related tech.
• Expert in Apache Spark, including RDDs, DataFrames, and AQE.
• Proficient in Python or Scala for data engineering.
• Experience with Spark-Streaming and cloud data services (Azure/AWS).

Responsibilities

• Create and maintain cloud-based data pipelines and an ODS.
• Ingest structured and unstructured data via batch and Kafka streaming.
• Assemble large, complex datasets meeting functional and technical requirements.
• Lead a team and collaborate with analytics and IS architecture teams.

Skills

Spark

Python

Scala

SQL

Data pipeline

AGILE

Streaming

Cloud (Azure/AWS)

Leadership

Tools

Spark Streaming

Job description

# Lead Data EngineerEPAM SystemsPosted on August 19, 2026---Experience8 - 13 yrsSalary (CTC)₹40.2L - ₹44.4LJob LocationHyderabad, IndiaVacancy6DesignationLead Data EngineerJob TypeONSITE---## Job Description**Exp:** 5-14 years**Location:** Gurgaon/Pune/Hyderabad/Bengaluru/Chennai/Coimbatore**Work Mode:** Hybrid (2-3 days office in a week)**Job Description:**1. 5+ Years of in Big Data & Data related technology experience2. Expert level knowledge and experience in Apache **Spark** (Advance Concepts i.e. RDDs, DataFrames, fault tolerance, Spark internals, Catalyst optimizer and AQE)3. Hands on programming with **Python (**Advance Concepts i.e. comprehensions, decorators, generators including their use cases and benefits and data structures**)**4. Understands exception handling and its practical applications.5. Proficient in using Python or Scala for data engineering and automation.6. Experience with building stream-processing systems, using technologies such as Spark-Streaming7. Experience with integration of data from multiple data sources such as RDBMS (SQL Server, Oracle), ERP, Files8. Good understanding of SQL queries, joins, stored procedures, relational schemas9. Knowledge of ETL techniques and frameworks10. Performance tuning of Spark Jobs11. Experience with native Cloud data services **Azure/AWS**12. Ability to lead a team efficiently13. Experience with designing and implementing Big data solutions14. Practitioner of AGILE methodology **What you'll do*** Create and maintain optimal cloud-based Operational Datasource (ODS) & data pipeline architectures capable of ingesting structured and unstructured data using both batch & kafka streaming methods* Assemble large, complex data sets that meet functional and technical requirements.* Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.* Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources.* Develop data flows that can leverage both on premise and cloud architectures.* Work with stakeholders including the business analytics teams and IS architecture teams to assist with data-related technical issues and support their data infrastructure needs. **WE OFFER**1. Opportunity to work on technical challenges that may impact across geographies2. Vast opportunities for self-development: online university, knowledge sharing opportunities globally, learning opportunities through external certifications3. Opportunity to share your ideas on international platforms4. Sponsored Tech Talks & Hackathons5. Possibility to relocate to any EPAM office for short and long-term projects6. Focused individual development7. Benefit package:• Health benefits, Medical Benefits• Retirement benefits• Paid time off• Flexible benefits1. Forums to explore beyond work passion (CSR, photography, painting, sports, etc.**About EPAM:** https://vimeo.com/729865941