Data Engineer III

Phenom People; Hyderabad; On-site; INR 2,500,000 - 4,000,000; Full time

Benefits offered by this job

Health benefits

Flexible hours

Parental leave

Career development opportunities

Perks in every location

Job summary

Phenom People is looking for an experienced Data Engineer to optimize and enhance our data pipelines that power insights across the organization. This role focuses on improving performance, ensuring data quality, automating pipeline development, and enabling data sharing through scalable and secure mechanisms.

The ideal candidate will have strong expertise in PySpark, SQL optimization, and experience with data sharing platforms such as DataHub.

Qualifications

• 6–10 years of experience as a Data Engineer with large-scale systems.
• Strong Python and PySpark expertise with Spark optimization.
• Deep SQL tuning and data quality practices.
• Experience with Airflow, Livy, EMR and data sharing platforms.

Responsibilities

• Optimize existing data pipelines for performance and scalability.
• Review and tune SQL queries to improve execution time.
• Design and develop batch data pipelines meeting business needs.
• Automate pipeline generation and metadata management.
• Implement data quality and validation frameworks.
• Share data via DataHub using connectors like SFTP and Snowflake Data Share.
• Enhance observability with Grafana and telemetry tools.

Skills

Data engineering

Problem solving

Communication

Tools

Python

PySpark

SQL

Airflow

Livy

EMR

Flink

Iceberg

Snowflake

DataHub

Grafana

Cursor

Copilot

Windsurf

SFTP

Job description

Job Description

Phenom Intro:

Our purpose is to help a billion people find the right job! Phenom is an AI-Powered talent experience platform that is redefining the HR tech space. We have grown into a global organization with offices in 6 countries and over 2,000 employees. As an HR tech unicorn organization, innovation and creativity is within our DNA. Come help us make every talent moment Phenomenal!

Job Summary

Phenom People is looking for an experiencedData Engineerto optimize and enhance our data pipelines that power insights across the organization. This role focuses on improving performance, ensuring data quality, automating pipeline development, and enabling data sharing through scalable and secure mechanisms.

The ideal candidate will have strong expertise inPySpark,SQL optimization, and experience with data sharing platforms such asDataHub. You'll collaborate closely with data engineering, platform, and analytics teams to modernize our data ecosystem built onAirflow, Livy, EMR, Flink, Iceberg, and Snowflake.

If you're passionate about performance tuning, automation, and building resilient data systems — we'd love to hear from you.

What You'll Do

• Optimize existing data pipelines built usingPySparkfor performance and scalability.
• Review and tuneSQL queriesto improve execution time and resource utilization.
• Design and developnew batch data pipelinesthat meet evolving business needs (real-time decisioning is not required; systems can tolerate up to 6 hours of delay).
• Automate pipeline generationand metadata management to streamline development.
• Implementdata quality and validation frameworksto ensure reliability and trust in data assets.
• Collaborate with teams to integrate and share data viaDataHub, leveraging connectors such asSFTPandSnowflake Data Share.
• Enhance observability by integrating and monitoring pipelines throughGrafanaand other telemetry tools.
• Work closely with other engineers to improve coding practices and leverageAI-powered developer assistantslikeCursor, Copilot, or Windsurffor productivity.
• Participate in design reviews, documentation, and operational support for production systems.

Work Experience

What You've Done

• 6-10 years of experience as aData Engineerworking with large-scale distributed data systems.
• Strong proficiency inPython(especiallyPySpark) and hands-on experience optimizing Spark jobs onEMR or similar environments.
• Deep knowledge ofSQL tuningand query optimization across analytical databases (preferablySnowflake).
• Experience working withAirflowfor workflow orchestration andLivyfor Spark job management.
• Exposure toFlinkandApache Icebergfor batch or incremental data processing.
• Strong understanding ofdata quality frameworks,testing, andvalidation techniques.
• Familiarity withDataHubor similar data catalog and sharing platforms.
• Experience setting up dashboards and alerts inGrafanaor equivalent monitoring tools.
• Exposure toAI coding assistants(Cursor, GitHub Copilot, Windsurf) for accelerating development is a plus.
• Passionate about automation, continuous improvement, and building scalable, maintainable systems.

Benefits

We want you to be your best self and to pursue your passions!

• Health and wellness benefits/programs to support holistic employee health.
• Flexible hours and working schedules, as well as parental leave for new parents.
• Growing organization with career pathing and development opportunities.
• Tons of perks and extras in every location for all Phenoms!

Diversity, Equity, & Inclusion

Our commitment to diversity runs deep! Diversity is essential to building phenomenal teams, products, and customer experiences. Phenom is proud to be an equal opportunity employer taking collective action to build a more inclusive environment where every candidate and employee feels welcomed.

We recognize there is more to be done. Our teams are committed to continuous improvement until these powerful ideas are ingrained in our culture for Phenom and employers everywhere!