Lead Data Engineer

Tredence Inc; Bengaluru; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

Tredence Inc is seeking a senior data engineering lead to guide a team responsible for building scalable data pipelines and data infrastructure. You will collaborate with cross-functional teams to ensure high-quality, high-performance data systems and drive data governance across the organization.

You will mentor engineers, define development roadmaps, and ensure timely delivery of complex data projects while aligning with client business objectives and security policies.

Qualifications

• Strong experience with Databricks, Spark, and cloud platforms (Azure, AWS, GCP).
• Architect and deploy cloud-based data solutions (Azure, AWS, GCP).
• Define CI/CD strategies for data pipelines using Terraform, Azure DevOps, or GitHub Actions.
• Implement data cataloguing, lineage tracking, and access control (Unity Catalogue, Collibra, Alation).
• Ensure compliance with GDPR, CCPA, and industry-specific data security policies.
• Develop strategies for distributed computing, parallel processing, and caching mechanisms.

Responsibilities

• Lead a team of data engineers to design, build, and maintain scalable data pipelines and infrastructure.
• Collaborate with cross-functional teams to meet client data quality and performance standards.
• Mentor and develop team members; conduct performance reviews and set goals.
• Plan and deliver data engineering projects on time and within scope.
• Coordinate with stakeholders to gather requirements and define timelines.
• Oversee data pipelines, ETL processes, and data warehouse design and maintenance.

Skills

Team leadership

Data engineering strategy

CI/CD for data pipelines

Cloud computing fundamentals

Tools

Databricks

Spark

Azure

AWS

GCP

Terraform

Azure DevOps

GitHub Actions

Unity Catalogue

Collibra

Alation

Job description

Responsibilities:

• Responsible for leading a team of talented data engineers responsible for designing, building, and maintaining scalable data pipelines and infrastructure.
• Work closely with cross-functional teams to ensure client data systems meet the highest standards of quality and performance.
• Lead, mentor, and develop a team of data engineers, fostering a collaborative and inclusive team environment.
• Conduct regular performance reviews, provide feedback, and set goals for team members.
• Identify and address skill gaps, and provide opportunities for professional development.
• Plan, execute, and deliver data engineering projects on time and within scope.
• Coordinate with stakeholders to gather requirements, set priorities, and define project timelines.
• Ensure projects align with overall business objectives and data strategy.
• Oversee the design, development, and maintenance of data pipelines, ETL processes, and data warehouse.
• Ensure data quality, integrity, and security across all data engineering projects.
• Identify opportunities for process improvements and drive initiatives to enhance the efficiency and effectiveness of data operations.
• Has a strong conceptual understanding of Data Warehousing and ETL, Data Governance and Security, Cloud Computing, and Batch and Real-Time data processing.
• Ability to build/drive reusable frameworks that can drive efficiency of the overall data system
• Has executed and lead multiple projects, including streaming, batch, large data pipelines, etc.
• Manages conversations with the client stakeholders to understand the Internal Use Only requirement and translate it into technical outcomes.

Requirements:

• Strong experience with Databricks, Spark, and cloud platforms (Azure, AWS, GCP).
• Architect and deploy cloud-based data solutions (Azure, AWS, GCP). Define CI/CD strategies for data pipelines using Terraform, Azure DevOps, or GitHub Actions.
• Implement data cataloguing, lineage tracking, and access control (Unity Catalogue, Collibra, Alation).
• Ensure compliance with GDPR, CCPA, and industry-specific data security policies.
• Develop strategies for distributed computing, parallel processing, and caching mechanisms.