Data Engineer

Weekday; Bengaluru; On-site; INR 3,000,000 - 5,000,000; Full time

Job summary

Weekday in Bengaluru, Karnataka, India is seeking a Senior Data Engineer with 10-16 years of experience and expertise in Python to design, develop, and manage scalable data solutions. This role requires deep knowledge of data architecture, ETL/ELT pipelines, data modeling, and distributed data systems, with the ability to solve complex technical and business problems.

You will own critical data initiatives, collaborate with engineering, analytics, product, and business teams, and contribute to

Qualifications

• 10–16 years of hands-on experience in Data Engineering or a closely related field.
• Strong and demonstrable expertise in Python, including production-grade coding.
• Strong understanding of ETL/ELT, data integration, data transformation, data modeling, and pipeline orchestration.
• Experience designing and implementing scalable data pipelines and distributed data processing solutions.
• Strong SQL skills and experience with relational and analytical databases.
• Good understanding of data warehousing, data lakes, and modern data architecture concepts.
• Experience with software development practices such as Git, code reviews, testing, CI/CD, and debugging.
• Strong analytical and problem-solving abilities with a focus on data quality and reliability.

Responsibilities

• Design, develop, and maintain scalable and high-performance data pipelines for batch and real-time data processing.
• Use Python extensively to build data ingestion frameworks, transformation workflows, automation utilities, and data processing applications.
• Develop robust ETL/ELT processes to collect, transform, validate, and integrate data from multiple sources.
• Design efficient data models, schemas, and architectures that support analytics, reporting, and operational use cases.
• Optimize data pipelines and processing workloads for performance, scalability, reliability, and cost efficiency.
• Implement data quality, validation, monitoring, and error-handling mechanisms across data workflows.
• Work with large and complex datasets and troubleshoot performance, data integrity, and pipeline reliability issues.
• Collaborate with software engineers, data scientists, analysts, product managers, and other stakeholders to translate requirements into data solutions.
• Establish engineering best practices around coding standards, testing, documentation, version control, deployment, and operational support.
• Mentor junior and mid-level data engineers and provide technical guidance on architecture and troubleshooting.
• Participate in architectural discussions to evolve the data platform and engineering practices.

Skills

Python

ETL/ELT

Data Modeling

SQL

Distributed Data Processing

CI/CD

Mentoring

Tools

Apache Spark

Kafka

Airflow

Databricks

Snowflake

Job description

This role is for one of Weekday’s clients

Min Experience: 10+ years
Location: Bengaluru, Karnataka, India
JobType: full-time

We are seeking an experienced Senior Data Engineer with 10–16 years of professional experience and strong expertise in Python to design, develop, and manage scalable data engineering solutions. The ideal candidate will have a deep understanding of data architecture, data processing, ETL/ELT pipelines, data modeling, and distributed data systems, along with the ability to solve complex technical and business problems.

As a Senior Data Engineer, you will take ownership of critical data initiatives, work closely with engineering, analytics, product, and business teams, and contribute to the design of robust and reliable data platforms.

Requirements

Key Responsibilities

• Design, develop, and maintain scalable and high-performance data pipelines for batch and real-time data processing.
• Use Python extensively to build data ingestion frameworks, transformation workflows, automation utilities, and data processing applications.
• Develop robust ETL/ELT processes to collect, transform, validate, and integrate data from multiple structured and unstructured sources.
• Design efficient data models, schemas, and architectures that support analytics, reporting, and operational use cases.
• Optimize data pipelines and processing workloads for performance, scalability, reliability, and cost efficiency.
• Implement data quality, validation, monitoring, and error-handling mechanisms across data workflows.
• Work with large and complex datasets and troubleshoot performance, data integrity, and pipeline reliability issues.
• Collaborate with software engineers, data scientists, analysts, product managers, and other stakeholders to understand requirements and translate them into effective data solutions.
• Establish engineering best practices around coding standards, testing, documentation, version control, deployment, and operational support.
• Mentor junior and mid-level data engineers and provide technical guidance on architecture, development, and troubleshooting.
• Participate in architectural discussions and contribute to the evolution of the organization's data platform and engineering practices.

Must-Have Skills

• 10–16 years of hands-on experience in Data Engineering or a closely related field.
• Strong and demonstrable expertise in Python, including writing production-grade, modular, maintainable, and optimized code.
• Strong understanding of data engineering principles, including ETL/ELT, data integration, data transformation, data modeling, and pipeline orchestration.
• Experience designing and implementing scalable data pipelines and distributed data processing solutions.
• Strong SQL skills and experience working with relational and analytical databases.
• Good understanding of data warehousing, data lakes, and modern data architecture concepts.
• Experience with software development practices such as Git, code reviews, testing, CI/CD, and debugging.
• Strong analytical and problem-solving abilities with excellent attention to data quality and reliability.
• Ability to independently drive complex data engineering projects from requirements through implementation and production deployment.

Preferred Qualifications

• Experience working with cloud-based data platforms and services.
• Exposure to technologies such as Apache Spark, Kafka, Airflow, Databricks, Snowflake, BigQuery, or similar platforms.
• Experience with real-time data processing and streaming architectures.
• Familiarity with containerization, orchestration, and cloud-native application development.
• Understanding of data governance, security, lineage, and privacy best practices.