Job Title: Senior / Lead Big Data Engineer
Experience: 9 - 15 Years
Location: Bangalore (Hybrid)
Mode of interview: 2 Virtual+1 in-person
# of roles: 2
Notice period: Immediate to 30 Days joiners only
About the Role
We are looking for a seasoned Senior/Lead Data Engineer with 9 to 15 years of experience to architect, build, and optimize large-scale data platforms. In this role, you will bridge the gap between complex big data infrastructure, functional business domains, and enterprise AI initiatives. Beyond taking end-to-end ownership of batch data pipelines that process massive customer and behavioral datasets, you will play a key leadership role in building, mentoring, and scaling a high-performing data engineering team.
Key Responsibilities
- Platform Optimization & Governance
- Performance Tuning: Proactively troubleshoot and fine-tune complex Spark and Hive/HQL jobs to reduce latency, optimize memory usage, and manage cluster resource allocation (YARN).
- Data Quality & Reliability: Establish automated data validation frameworks, lineage tracking, and strict SLA monitoring to guarantee data accuracy and platform reliability.
- Platform Operations: Oversee on-prem big data platform stability, capacity planning, and workflow orchestration using Airflow.
- Team Building & Technical Leadership
- Build & Scale the Team: Partner with leadership to hire, onboard, and build a strong data engineering team from the ground up.
- Mentorship & Guidance: Provide hands-on technical leadership, conduct architecture and code reviews, and mentor junior and mid-level engineers.
- Engineering Standards: Establish and enforce best practices around Git version control, code quality, documentation, and Agile delivery methodologies.
Architecture & Pipeline Development
- End-to-End ETL Ownership: Architect, develop, and maintain scalable, fault-tolerant batch ETL/ELT pipelines across on-premises Hadoop-based data platforms.
- Large-Scale Data Processing: Design high-throughput data ingestion and transformation workflows for massive customer and behavioral datasets.
- Data Modeling: Lead dimensional and relational data modeling efforts to support enterprise analytics, reporting, and downstream AI solutions.
- Functional Alignment & Cross-Team Collaboration
- Business & Functional Translation: Translate complex functional business requirements into scalable technical data architectures, ensuring data assets directly solve business challenges.
- Cross-Functional Partnership: Collaborate closely with analytics, product, and AI teams to understand data needs and deliver reliable, analytics-ready datasets.
Qualifications & Skills
- Educational Qualification: B.E. or B.Tech in Computer Science, Information Technology, or a related engineering discipline.
- Experience: 9 to 15 years of progressive experience in Data Engineering, with deep hands-on expertise in Hadoop-based big data ecosystems (HDFS, YARN).
- Leadership: Proven experience leading engineering initiatives and building or mentoring technical teams.
- Big Data Stack: Expert-level proficiency in Apache Spark (Core, Spark SQL, PySpark) and Apache Hive / HQL.
- Programming & Querying: Strong command of Python and advanced SQL, paired with solid experience in enterprise data modeling.
- Pipeline Engineering: Proven track record of designing and maintaining production-grade batch ETL pipelines.
- AI & Functional Acumen: Practical experience working on data initiatives supporting AI applications, combined with strong functional business understanding.
- Version Control: Proficient in Git for collaborative software development.
- Domain Experience: Prior experience working with e-commerce platforms and processing high-volume customer/transactional datasets is an added advantage.
- Workflow Orchestration: Hands-on experience managing complex DAGs and schedules using Apache Airflow or Oozie.
- Infrastructure & Tuning: Deep understanding of on-premise big data platform operations and advanced performance tuning for Hive and Spark workloads.
- Methodology: Familiarity with Agile/Scrum development methodologies.
Required Skills
Mandatory Skill Set:
- Hadoop Ecosystem (HDFS, YARN)
- Hive & HQL
- Apache Spark (Core, Spark SQL, PySpark)
- SQL & Data Modeling
- AI Data Engineering
- Batch ETL/Data Pipeline Development
Functional Acumen: Team Building & Leadership
Preferred Domain expertise : Retail / E-commerce / Online Retailing / Online Shopping / Digital Commerce / E-tailing