Data Engineer

Agivant Technologies; Hyderabad; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Agivant Technologies seeks a data engineering professional to execute massive-scale asset rationalization, migrations, and programmatic metadata implementations using enterprise query, visualization, and telemetry tools.

You will manage data pipelines, trace data lineage, and optimize queries across Hive/Presto/Trino, PySpark, and Databricks, ensuring observability and SLA adherence.

Qualifications

• Proficient in Hive and SQL for large-scale data manipulation (≤140 chars).
• Experience with Tableau and enterprise visualization tools for dashboards.
• Knowledge of data lineage and telemetry in large data pipelines.

Responsibilities

• Execute asset rationalization for dashboards and data pipelines.
• Write programmatic metric definitions and mappings into central catalogs via APIs.
• Monitor observability dashboards and resolve escalations within SLAs.

Tools

Hive

Presto/Trino

Python

Airflow

Tableau

Power BI/Enterprise Visualization

Databricks

PySpark

dbt

AI coding assistants

Job description

Job Description:

Mission: Execute massive-scale asset rationalization, pipeline migrations, and programmatic metadata implementations using enterprise query, visualization, and telemetry tools.

Core Responsibilities

• Asset Rationalization: Execute the migration, consolidation, or retirement of hundreds of legacy enterprise dashboards (Tableau, PowerBI) and data pipelines based on strict usage telemetry.
• Programmatic Definitions: Write code that pushes metric definitions, dimensions, and owner mapping programmatically into central semantic catalogs via APIs.
• Data Health Enforcement: Monitor real-time observability dashboards and resolve automated escalations regarding pipeline freshness lags or volume drops within strict SLAs.

Requirements

Tech Stack & Depth

• Core: Hive, Advanced SQL, Python for data manipulation, Airflow (or similar orchestration), Tableau / Enterprise Visualization tools.
• Depth: Hands-on querying of massive distributed datasets (Hive/Presto/Trino) and telemetry logging. Must know how to trace complex data lineage manually through code.

Good to Have / Bonus Tech Stack

• Experience with Databricks workflows and PySpark optimization.
• Integrating automated testing in dbt.
• Utilizing AI coding assistants for rapid SQL script generation and optimization.

Requirements: