Senior/Lead Data Engineer

Sourcebae; Chennai District; On-site; INR 2,800,000 - 4,800,000; Full time

Job summary

Sourcebae is seeking a Lead Data Engineer to design and deploy scalable data platforms in a cloud environment. You will lead a team of engineers, drive architectural decisions, and collaborate with cross-functional teams to deliver robust data solutions.

The role emphasizes hands-on development, data governance, and ongoing improvement of data pipelines, with a focus on AI/ML readiness and observability across systems.

Qualifications

• 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable data pipelines.
• Proficiency in Python, Scala, and Spark for large-scale data processing.
• Expertise with Spark, Data Lake, Delta Lake, and Hive.
• Strong SQL skills across large data volumes.

Responsibilities

• Lead end-to-end design, development, and delivery of software systems and data pipelines.
• Define technical strategy, architecture, and best practices for data engineering.
• Build optimized data pipelines in a cloud environment.
• Create infrastructure for ETL processes from various sources and storage systems.
• Architect and maintain database pipelines for AI/ML readiness.
• Mentor engineers through code reviews and technical discussions.
• Ensure timely delivery with high engineering standards.
• Drive continuous improvement in data quality and reliability.

Skills

Python

Scala

Spark

SQL

Snowflake

Hive

Data Lake

Delta Lake

Batch processing

Streaming processing

Data Lifecycle Management

Agile

Observability

Distributed systems

Education

Bachelor's degree in Computer Science or related field

Tools

Snowflake

Hive

Delta Lake

Data Lake

Spark

Job description

Roles and Responsibilities:

Role Overview

• The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands-on technical work, architectural decision-making, mentoring junior developers, and collaborating with cross-functional teams to ensure successful delivery of scalable data platforms and analytical solutions.

Key Responsibilities

• Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.
• Define technical strategy, architecture, and best practices for development and data engineering.
• Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.
• Construct infrastructure for efficient ETL processes from various sources and storage systems.
• Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.
• Lead the implementation of algorithms and prototypes to transform raw data into useful information.
• Review code for quality, scalability, and performance.
• Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.
• Develop analytical tools, programs, and reporting mechanisms.
• Create data validation methods and data analysis tools.
• Interpret data trends and patterns to establish operational alerts.
• Conduct complex data analysis and present results effectively.
• Prepare data for prescriptive and predictive modeling.
• Ensure compliance with data governance and security policies.
• Troubleshoot, debug, and resolve complex technical issues.
• Drive continuous improvement in software and data development processes, tools, and methodologies.
• Mentor and guide engineers through code reviews, technical discussions, and training.
• Ensure timely delivery of projects while maintaining high engineering standards.
• Continuously explore opportunities to enhance data quality and reliability.
• Apply strong programming and problem-solving skills to develop scalable solutions.
• Demonstrate passion for testing strategy, problem-solving, and continuous learning.
• Willingness to acquire new skills and knowledge.
• Possess a product/engineering mindset to drive impactful data solutions.
• Experience working in distributed environments with global teams.
• Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills and Experience Requirements

• Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.
• High proficiency in Python, Scala, and Spark for applied large-scale data processing.
• Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.
• Solid understanding of batch and streaming data processing techniques.
• Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.
• Expert-level ability to write complex, optimized SQL queries across extensive data volumes.
• Experience with RDBMS and OLAP databases such as MySQL and Snowflake.
• Familiarity with Agile methodologies.
• Obsession for service observability, instrumentation, monitoring, and alerting.
• Knowledge or experience in architectural best practices for building data lakes.
• Qualifications -Bachelor’s degree in computer science, Engineering, Information Systems, or related field