Lead AWS Data Engineer

Weekday (YC W21); Bengaluru; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

Weekday (YC W21) in Bengaluru is seeking a highly experienced Senior Data Engineer to design and optimize scalable cloud-based data platforms. In this role, you will work closely with business stakeholders, data scientists, and engineering teams to create reliable and high-performance data solutions.

The ideal candidate has 8-10 years of experience in data engineering and strong expertise in AWS services like Glue and Redshift. This position demands a proactive mindset and a focus on best practices in data management.

Qualifications

• 8–10 years of experience in Data Engineering and Cloud Data Platform development.
• Strong expertise in AWS cloud services, especially Glue and Redshift.
• Advanced proficiency in Python and PySpark for data processing.
• Strong command of SQL and PL/SQL.

Responsibilities

• Design and maintain scalable data pipelines and ETL/ELT processes.
• Build cloud-based data solutions using AWS services.
• Ensure data quality, governance, security, and compliance.
• Mentor junior engineers and drive continuous improvement.

Skills

Data Engineering

AWS services (Glue, Lambda, Redshift)

Python

ETL/ELT processes

SQL

Data Warehousing

PySpark

Data governance

Analytical thinking

Job description

Industry: IT Services and IT Consulting

Seniority level: Mid‑Senior level

Experience: 7+ years

Location: Bengaluru, Pune

Job Type: full‑time

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.

We are looking for a highly experienced Senior Data Engineer to design, build, and optimize scalable cloud‑based data platforms that power enterprise analytics, reporting, and AI‑driven initiatives. This role is ideal for a hands‑on data engineering professional who combines deep technical expertise with strong business understanding to create reliable, secure, and high‑performance data solutions. You will be responsible for developing modern data architectures, building robust ETL/ELT frameworks, and leveraging cloud‑native technologies to support large‑scale data processing and analytics.

Working closely with business stakeholders, data scientists, analysts, and engineering teams, you will transform complex business requirements into scalable technical solutions while driving best practices in data engineering, governance, performance optimization, and platform reliability.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines, ETL/ELT processes, and data integration frameworks.
• Build cloud‑native data solutions using AWS services such as Glue, Lambda, Redshift, Aurora, OpenSearch, Step Functions, SNS, and S3.
• Develop and optimize large‑scale data processing workflows using Python, PySpark, SQL, and PL/SQL.
• Design and implement enterprise data warehouses, data marts, dimensional models, and reporting structures.
• Collaborate with business stakeholders, product teams, analysts, and data scientists to deliver data‑driven solutions.
• Ensure data quality, governance, security, compliance, and lifecycle management across the data ecosystem.
• Optimize database performance, query execution, and high‑volume data processing workloads.
• Implement monitoring, logging, alerting, and troubleshooting mechanisms to maintain platform reliability.
• Participate in architecture reviews, cloud modernization initiatives, and technical design discussions.
• Support AI, machine learning, and advanced analytics use cases through scalable data infrastructure.
• Drive continuous improvement initiatives focused on performance, scalability, and operational excellence.
• Mentor junior engineers and promote engineering best practices, code quality standards, and knowledge sharing.
• Support deployment activities, production issue resolution, and ongoing platform enhancements.

What Makes You a Great Fit

• 8–10 years of experience in Data Engineering, Data Warehousing, and Cloud Data Platform development.
• Strong expertise in AWS cloud services, including Glue, Lambda, Redshift, Aurora, OpenSearch, Step Functions, SNS, and S3.
• Advanced proficiency in Python and PySpark for large‑scale data processing and transformation.
• Strong command of SQL and PL/SQL development and optimization.
• Extensive experience building ETL/ELT pipelines, data ingestion frameworks, and integration solutions.
• Deep understanding of data warehousing concepts, dimensional modeling, and database architecture.
• Experience working with structured, semi‑structured, and unstructured datasets.
• Strong knowledge of modern data architectures, including Data Lakes, Data Warehouses, and Lakehouse environments.
• Familiarity with workflow orchestration, event‑driven architectures, and distributed data processing systems.
• Experience implementing data governance, quality frameworks, and security best practices.
• Strong analytical thinking, problem‑solving skills, and attention to detail.
• Excellent communication and stakeholder management abilities.
• Experience supporting AI, Machine Learning, Data Science, or Advanced Analytics initiatives is highly desirable.
• Exposure to CI/CD pipelines, DevOps practices, Infrastructure as Code (IaC), and Agile delivery methodologies is a plus.
• AWS certifications or equivalent cloud credentials are highly valued.
• A proactive, ownership‑driven mindset with a passion for building scalable and reliable data platforms.