Lead Data Engineer

MathCo; Bengaluru; On-site; INR 3,500,000 - 6,500,000; Full time

Job summary

MathCo in Bengaluru is seeking a senior data engineering lead to head a team responsible for designing, building, and maintaining scalable data pipelines and data infrastructure.

You will work with cross-functional teams to ensure data systems meet high standards of quality and performance, while mentoring engineers and aligning projects with business objectives.

Qualifications

• Experience leading data engineering teams and delivering scalable pipelines.
• Strong knowledge of Data Warehousing, ETL, and data governance.
• Proficiency with cloud data platforms (Azure/AWS/GCP).
• Excellent communication and stakeholder management.
• Ability to translate business requirements into technical outcomes.

Responsibilities

• Lead and mentor a team of data engineers to design and maintain scalable data pipelines.
• Collaborate with cross-functional teams to ensure data quality and performance.
• Plan, execute, and deliver data engineering projects on time and within scope.
• Define and implement CI/CD strategies for data pipelines.
• Oversee data security and governance across data operations.

Skills

Team Leadership

Data Engineering

Cross-functional Collaboration

Performance Reviews

Project Management

Data Quality

Stakeholder Management

Mentoring

Cloud Computing

ETL/ELT

Education

B.E./B.Tech

MCA

M.Sc. in Mathematics/Statistics

Tools

Databricks

Spark

Azure

AWS

GCP

Terraform

Azure DevOps

GitHub Actions

Unity Catalog

Collibra

Alation

Docker

Kubernetes

Job description

• Responsible for leading a team of talented data engineers responsible for designing, building, and maintaining scalable data pipelines and infrastructure
• Work closely with cross-functional teams to ensure client data systems meet the highest standards of quality and performance
• Lead, mentor, and develop a team of data engineers, fostering a collaborative and inclusive team environment
• Conduct regular performance reviews, provide feedback, and set goals for team members
• Identify and address skill gaps, and provide opportunities for professional development
• Plan, execute, and deliver data engineering projects on time and within scope
• Coordinate with stakeholders to gather requirements, set priorities, and define project timelines.
• Ensure projects align with overall business objectives and data strategy
• Oversee the design, development, and maintenance of data pipelines, ETL processes, and data warehouse
• Ensure data quality, integrity, and security across all data engineering projects.
• Identify opportunities for process improvements and drive initiatives to enhance the efficiency and effectiveness of data operations.
• Has strong conceptual understanding of Data Warehousing and ETL, Data Governance and Security, Cloud Computing, and Batch & Real Time data processing
• Ability to build/drive reusable frameworks that can drive efficiency of the overall data system
• Has executed and lead multiple projects including on - streaming, batch, large data pipelines, etc.
• Manages conversation with the client stakeholders to understand the requirement and translate it into technical outcomes.

Required Tech Stack

• Strong experience with Databricks, Spark, and cloud platforms (Azure, AWS, GCP).
• Architect and deploy cloud-based data solutions (Azure, AWS, GCP).
• Define CI/CD strategies for data pipelines using Terraform, Azure DevOps, or GitHub Actions.
• Implement data cataloging, lineage tracking, and access control (Unity Catalog,Collibra, Alation).
• Ensure compliance with GDPR, CCPA, and industry-specific data security policies.
• Develop strategies for distributed computing, parallel processing, and caching mechanisms.

Required Non-Tech Stack

• Partner with data architects, product managers, and business leaders to define data requirements and align engineering efforts with business objectives.
• Define data engineering standards and playbooks to streamline development.
• Oversee end-to-end project execution, from scoping to delivery.
• Stay updated with emerging trends in data engineering, AI, and analytics to continuously improve architectures.
• Evaluate and recommend new data tools, frameworks, and best practices.
• Ability to translate complex technical concepts into business-friendly language.
• Excellent communication, leadership, and stakeholder management.

Good to Have Tech Stack

• Experience with machine learning and advanced analytics technologies
• Familiarity with data visualization tools and techniques
• Knowledge of data security and privacy practices
• Understanding of data governance and compliance frameworks
• Experience with containerization and orchestration technologies (e.g., Docker, Kubernetes)
• Experience with graph databases and graph processing frameworks
• Experience with data virtualization and data federation techniques
• Proficiency in data profiling and data quality management

Preferred Educational Qualifications

B.E/B.Tech, MCA, M.Sc. (Mathematics, Statistics)