Eucloid - Lead Data Engineer

Eucloid Data Solutions Pvt Ltd; Chennai District; On-site; INR 2,200,000 - 3,800,000; Full time

Job summary

Eucloid Data Solutions Pvt Ltd is seeking a Lead Data Engineer to own the design, development, and scaling of modern data platforms for enterprise clients. You will lead data engineering initiatives across architecture, pipeline design, data integration, performance optimization, and platform modernization.

The ideal candidate will translate complex business and technical requirements into scalable data solutions, provide technical direction to data teams, and collaborate with architects,

Qualifications

• 7+ years of experience in Data Engineering or related roles.
• Strong hands-on experience designing enterprise-scale data solutions.
• Expertise with AWS data services (S3, Redshift, EMR, Athena, Kinesis, Lambda).
• Experience with Databricks and/or Snowflake preferred.
• Experience with workflow orchestration tools (Airflow, Azkaban, Luigi).
• Experience with ETL/ELT tools like Talend or Informatica.

Responsibilities

• Lead architecture and design of scalable data engineering solutions for complex requirements.
• Own end-to-end data platform initiatives from design to production support.
• Define ingestion, transformation, processing, storage, and consumption strategies across ecosystems.
• Design and optimize large-scale batch and streaming pipelines from multiple sources.
• Establish data quality, validation, monitoring, and observability frameworks.

Skills

Data engineering

AWS

Databricks

Snowflake

Airflow

ETL/ELT

CI/CD

NoSQL databases

Data modeling

Data quality

Tools

Airflow

Azkaban

Luigi

Talend

Informatica

Job description

Job Description

We are looking for a Lead Data Engineer to join our Data Platform team and take ownership of designing, developing, and scaling modern data platforms for enterprise clients.

The role involves leading data engineering initiatives across technology evaluation, architecture, pipeline design, data integration, performance optimization, and platform modernization.

The ideal candidate will be responsible for translating complex business and technical requirements into scalable data solutions while providing technical direction to data engineering teams.

The role requires strong expertise in cloud data platforms, distributed data processing, data architecture, ETL/ELT, and data quality, along with the ability to work closely with architects, analysts, application teams, and business stakeholders.

Key Responsibilities

• Lead the architecture and design of scalable, secure, and high-performance data engineering solutions for complex business requirements.
• Own end-to-end data platform initiatives, from technology evaluation and solution design to implementation, deployment, and production support.
• Define data ingestion, transformation, processing, storage, and consumption strategies across enterprise data ecosystems.
• Design and optimize large-scale batch and streaming data pipelines integrating data from multiple internal and external sources.
• Establish reusable data engineering frameworks, standards, design patterns, and best practices across projects.
• Lead the development of cloud-based data platforms leveraging AWS services such as S3, Redshift, EMR, Athena, Kinesis, and Lambda.
• Drive the design and implementation of modern data architectures using technologies such as Databricks and Snowflake.
• Provide technical leadership for ETL/ELT development, data warehousing, data lake, and data lakehouse initiatives.
• Design robust mechanisms for handling schema evolution, schema drift, data dependencies, and source-system changes.
• Establish data quality, validation, reconciliation, monitoring, and observability frameworks to ensure reliability of critical data pipelines.
• Define strategies for pipeline performance optimization, scalability, availability, and cost efficiency.
• Lead integration of data from supplier systems, landing zones, APIs, databases, files, and other heterogeneous sources.
• Evaluate and recommend appropriate technologies, tools, and frameworks based on scalability, performance, maintainability, and business requirements.
• Design workflow orchestration and dependency management using tools such as Airflow, Azkaban, or Luigi.
• Guide implementation of ETL solutions using platforms such as Talend, Informatica, or equivalent technologies.
• Work with NoSQL and relational databases to design appropriate data storage and processing solutions.
• Establish engineering practices around version control, CI/CD, testing, deployment, monitoring, and production support.
• Lead root-cause analysis for complex data platform and pipeline issues and drive permanent solutions.
• Collaborate with software engineers, solution architects, database architects, data scientists, analysts, and business stakeholders.
• Mentor data engineers and provide technical guidance on architecture, coding standards, design reviews, and engineering best practices.
• Review technical designs and code to ensure adherence to scalability, reliability, security, and maintainability standards.
• Drive continuous improvement and modernization of existing data platforms and engineering processes.
• Ensure technical documentation, architecture decisions, data flows, and operational processes are maintained effectively.

Technical Expertise

• 7+ years of experience in Data Engineering, Data Platforms, Big Data, or related technology roles.
• Strong hands-on experience designing and implementing enterprise-scale data engineering solutions.
• Strong expertise in AWS data services such as S3, Redshift, EMR, Athena, Kinesis, and Lambda.
• Strong understanding of data warehousing, data lakes, data lakehouse architecture, ETL/ELT, and distributed data processing.
• Experience working with Databricks and/or Snowflake is highly preferred.
• Strong experience building and optimizing large-scale data pipelines and data processing frameworks.
• Experience with workflow orchestration tools such as Airflow, Azkaban, or Luigi.
• Good experience with ETL tools such as Talend, Informatica, or equivalent platforms.
• Experience working with relational and NoSQL databases.
• Strong understanding of data modelling, metadata management, data lineage, dependency management, and data quality.
• Experience handling schema evolution, schema drift, data reconciliation, and data validation.
• Strong understanding of pipeline monitoring, production support, performance tuning, and operational reliability.
• Experience with CI/CD, Git, automated testing, and deployment practices for data engineering environments.
• Strong understanding of cloud-native data architecture and scalable distributed systems.

Leadership & Stakeholder Management

• Ability to lead technical discussions and drive architecture and design decisions.
• Strong problem-solving skills with the ability to resolve complex data engineering challenges.
• Experience mentoring and guiding data engineering teams.
• Ability to conduct design and code reviews and establish engineering standards.
• Strong communication skills with the ability to explain complex technical concepts to both technical and non-technical stakeholders.
• Ability to work independently and take ownership of large-scale data engineering initiatives.
• Strong stakeholder management skills with experience working directly with clients and cross-functional teams

(ref:hirist.tech)