Lead Data Engineer

Weekday AI (YC W21); Coimbatore District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Weekday AI (YC W21) is seeking a highly skilled Lead Data Engineer, based in India, to design and optimize scalable cloud-based data platforms. The ideal candidate should have extensive experience in Google Cloud Platform (GCP), Apache Beam, Java, and BigQuery, with proven capabilities in building enterprise-grade data solutions.

The role involves collaborating with various teams to create efficient data processing systems, ensuring data quality and performance while driving engineering best practices.

Qualifications

• 7+ years of experience in data engineering or related field.
• Strong hands-on experience with modern cloud data architectures.
• Experience working with large-scale datasets and complex integrations.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using GCP services.
• Build and maintain real-time and batch data processing pipelines.
• Collaborate with cross-functional teams to deliver data-driven solutions.

Skills

Google Cloud Platform (GCP)

Apache Beam

Java

BigQuery

SQL

Kafka

Data modeling

CI/CD

Job description

Salary range: Rs 1500000 - Rs 2500000 (INR 15-25 LPA)

Experience: 7+ years

Location: Chennai, Coimbatore, Bangalore, Pune

Job type: full-time

We are looking for a highly skilled Lead Data Engineer to design, build, and optimize scalable cloud-based data platforms and processing pipelines for enterprise-grade applications. This role is ideal for someone who enjoys solving large-scale data engineering challenges and has deep expertise in Google Cloud Platform (GCP), Apache Beam/Dataflow, Java, and BigQuery.

As a Lead Data Engineer, you will play a critical role in building robust batch and real-time data processing systems that enable analytics, reporting, and data-driven decision-making across the organization. You will work closely with architects, analysts, DevOps teams, and business stakeholders to develop reliable, scalable, and high-performance data solutions.

The ideal candidate is passionate about modern cloud data architectures, distributed systems, and building efficient ETL/ELT pipelines capable of handling large-scale enterprise workloads. This role requires strong technical expertise, problem-solving ability, and the capability to drive engineering best practices across data platforms.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using Google Cloud Platform services
• Build and maintain real-time and batch data processing pipelines using Apache Beam and Google Dataflow
• Develop backend processing components and data transformation services using Java
• Work extensively with BigQuery for data warehousing, analytics, querying, and performance optimization
• Integrate data from multiple sources including APIs, relational databases, streaming systems, and cloud platforms
• Build reliable and scalable streaming data pipelines using technologies such as Kafka and cloud-native services
• Optimize pipeline performance, scalability, reliability, and cloud infrastructure costs
• Ensure high standards of data quality, governance, monitoring, security, and operational excellence
• Collaborate with cross-functional teams including Architects, Analysts, DevOps, and business stakeholders to deliver data-driven solutions
• Troubleshoot production issues, perform root-cause analysis, and implement long-term scalable fixes
• Implement CI/CD practices, version control workflows, and automated deployment pipelines for data engineering solutions
• Design and maintain scalable data models, schemas, and warehouse structures
• Participate in architecture discussions and contribute to improving overall cloud data platform capabilities
• Drive best practices in distributed data processing, pipeline optimization, and cloud-native engineering

What Makes You a Great Fit

• Strong hands-on experience with Google Cloud Platform (GCP) services
• Deep expertise in Apache Beam and Google Dataflow for large-scale data processing
• Strong programming skills in Java with experience building backend processing systems
• Hands-on experience with BigQuery and cloud-based data warehousing solutions
• Experience building and maintaining batch and real-time streaming data pipelines
• Strong understanding of SQL, data modeling, and distributed data processing concepts
• Experience working with Kafka or similar streaming technologies
• Familiarity with CI/CD pipelines, Git workflows, and Agile development methodologies
• Strong analytical, troubleshooting, and debugging capabilities
• Understanding of scalable cloud architecture, performance optimization, and reliability engineering
• Ability to work collaboratively with cross-functional engineering and business teams
• Experience handling enterprise-scale datasets and complex integration requirements
• Strong communication skills with the ability to translate technical solutions into business impact
• Passion for building modern cloud-native data platforms and solving large-scale engineering challenges