Data Engineer

PW (PhysicsWallah); Dadri; Vor Ort; INR 900.000 - 1.300.000; Vollzeit

Zusammenfassung

PW (PhysicsWallah) is seeking an experienced Data Engineer to design, build, and optimize scalable data infrastructure and pipelines. You will work on ETL/ELT processes, distributed data processing, and data lake and warehouse architectures to support business intelligence and analytics.

Collaborate with data science, analytics, and product teams to gather requirements and implement data-driven solutions. You will mentor juniors, drive performance tuning, cost optimization, and ensure data

Qualifikationen

• 1+ years of professional experience in data engineering or backend systems with a focus on scalable systems.
• Strong hands-on experience with Python or Scala, and writing efficient, production- grade code.
• Deep understanding of data engineering concepts: data modeling, data warehousing, data lakes, streaming vs. batch processing, and metadata management.
• Solid experience with AWS (S3, Redshift, EMR, Glue, Lambda) or equivalent cloud platforms.
• Experience working with orchestration tools like Apache Airflow (preferred) or similar (Azkaban, Luigi).

Aufgaben

• Design and implement scalable, efficient ETL/ELT pipelines to ingest, transform, and process data from multiple sources.
• Architect and maintain robust data lake and warehouse solutions, aligning with business and analytical needs.
• Own the development and optimization of distributed data processing systems using Spark, AWS EMR, or similar technologies.
• Collaborate with cross-functional teams (data science, analytics, product) to gather requirements and implement data-driven solutions.
• Ensure high levels of data quality, security, and availability across systems.
• Evaluate emerging technologies and tools for data processing and workflow orchestration.
• Build reusable components, libraries, and frameworks to enhance engineering efficiency and reliability.
• Drive performance tuning, cost optimization, and automation of data infrastructure.
• Mentor junior engineers, review code, and set standards for development practices.

Jobbeschreibung

We are looking for an experienced Data Engineer to design, build, and optimize scalable data

infrastructure and pipelines. The ideal candidate will have strong expertise in ETL/ELT

processes, distributed data processing, and data lake and warehouse architectures. They will

work closely with cross-functional teams to build reliable, high-performance data solutions that

support business intelligence, analytics, and data-driven decision-making.

Key Responsibilities:

• Design and implement scalable, efficient ETL/ELT pipelines to ingest, transform, and process data from multiple sources.


• Architect and maintain robust data lake and warehouse solutions, aligning with business and analytical needs.


• Own the development and optimization of distributed data processing systems using Spark, AWS EMR, or similar technologies.


• Collaborate with cross-functional teams (data science, analytics, product) to gather requirements and implement data-driven solutions.


• Ensure high levels of data quality, security, and availability across systems.


• Evaluate emerging technologies and tools for data processing and workflow orchestration.


• Build reusable components, libraries, and frameworks to enhance engineering efficiency and reliability.


• Drive performance tuning, cost optimization, and automation of data infrastructure.


• Mentor junior engineers, review code, and set standards for development practices.

Required Skills & Qualifications:

• 1+ years of professional experience in data engineering or backend systems with a focus on scalable systems.


• Strong hands-on experience with Python or Scala, and writing efficient, production- grade code.


• Deep understanding of data engineering concepts: data modeling, data warehousing, data lakes, streaming vs. batch processing, and metadata management.


• Solid experience with AWS (S3, Redshift, EMR, Glue, Lambda) or equivalent cloud platforms.


• Experience working with orchestration tools like Apache Airflow (preferred) or similar (Azkaban, Luigi).


• Proven expertise in working with big data tools such as Apache Spark, and managing Kubernetes clusters.


• Proficient in SQL and working with both relational (Postgres, Redshift) and NoSQL (MongoDB) databases.


• Ability to understand API-driven architecture and integrate with backend services as part of data pipelines.


• Strong problem-solving skills, with a proactive attitude towards ownership and continuous improvement.