Senior Data Engineer - Teradata/Databricks

ZingMind Technologies; Dadri; Hybrid; INR 3,000,000 - 6,000,000; Full time

Job summary

NTT DATA Americas is seeking a Senior Data Engineer to support a Teradata Utilization Analysis engagement, focused on cost-reduction opportunities and a repeatable, log-driven analysis solution.

You will ingest Teradata DBQL logs, integrate Autosys/DataStage/MagicWand metadata, and build end-to-end pipelines in Databricks to identify unused datasets and analyze workload statistics for optimization.

Qualifications

• Teradata DBA and DBQL knowledge with focus on system internals.
• Experience with Teradata DBQL and system views for inventory and last read/write tracking.
• Strong SQL and performance tuning across DBQL and catalog tables.
• Experience with Teradata ResUsage/workload statistics for cost impact.
• Data classification of datasets (hot/warm/cold) with tiered storage recommendations.
• ETL pipeline assessment using DataStage and Autosys.
• Experience with Teradata BTEQ/FastExport/TPT utilities.
• Databricks platform, Spark/PySpark, Delta Lake, notebooks, and workflows.
• Python data wrangling (pandas), SQL, and data visualization tools.

Responsibilities

• Ingest and validate 18+ months of Teradata DBQL logs (SQL text, usage, timestamps, IDs, row counts).
• Integrate autosys, DataStage, and MagicWand metadata to enhance DBQL analysis.
• Build end-to-end, log-driven pipelines in Databricks to identify unused datasets/partitions.
• Analyze CPU/IO usage and workload statistics to quantify cost-reduction opportunities.
• Classify data into cold/warm/hot tiers and generate heatmaps of access patterns.
• Develop a prioritized recommendation backlog with savings and risk levels.
• Apply AI/ML models or LLM-assisted analysis to detect anomalies and predict cold data.
• Produce deliverables: Observation Report, Workshop Notes & Action Log, Final Readout.

Skills

Teradata DBA

DBQL

System Views

Performance Tuning

ResUsage/Workload

Data Classification

ETL Pipeline Assessment

BTEQ/FastExport/TPT

Databricks Platform

Spark/PySpark

Delta Lake

Databricks Notebooks

Python (pandas)

SQL (Spark/ANSI)

Unity Catalog

Dashboarding

ML/AI (scikit-learn)

LLM Integration

NLP

Plotly/Matplotlib

Tools

Databricks

Python

Plotly

Job description

Position Overview

NTT DATA Americas is seeking a highly skilled Senior Data Engineer to support a Teradata Utilization Analysis engagement, which focuses on analyzing Teradata platform utilization to identify cost-reduction opportunities and deliver a repeatable, log-driven analysis solution.

Work Locations: Any NTT location

Experience (Relevant): 10+ Years

Shift Timings: 2pm to 11pm IST

Hybrid: 3 days/ week

Start date: ASAP

Key Responsibilities

• Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
• Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
• Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
• Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
• Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
• Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
• Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
• Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.

Required Skills & Experience

Teradata DBA Skills:

• Teradata DBA / Administration: Primary platform under analysis; deep knowledge of system internals required.
• Teradata DBQL (Database Query Logging): Core data source - SQL text, object usage, timestamps, user/app IDs, row counts, steps.
• Teradata System Views & Space Metadata: 5+ years - Object inventory, space analysis, last read/write tracking.
• Teradata SQL & Performance Tuning: Writing complex analytical queries across DBQL and system catalog tables.
• Teradata ResUsage / Workload Statistics: Quantifying CPU/IO cost impact for dataset removal/archival recommendations.
• Data Classification (hot/warm/cold): 3+ years - Categorizing datasets for tiered storage management recommendations.
• ETL Pipeline Assessment (DataStage, Autosys): 4+ years - Identifying and recommending decommission of stale ingestion pipelines.
• Teradata BTEQ / FastExport / TPT: 3+ years - Data extraction, log export, and metadata collection utilities.

Databricks Engineering Skills:

• Databricks Platform (Workspaces, Clusters, Jobs): Core analysis and solution delivery platform for the engagement.
• Apache Spark (PySpark / Spark SQL): Large-scale log ingestion, transformation, and analytical processing.
• Delta Lake: Building reliable, ACID-compliant data pipelines for log-driven analysis.
• Databricks Notebooks & Workflows: Developing repeatable, shareable analytical notebooks for customer handoff.
• Python (pandas, numpy, matplotlib): Data wrangling, statistical analysis, and visualization of usage heatmaps.
• SQL (Spark SQL / ANSI SQL): Analytical querying across large DBQL datasets in Databricks.
• Unity Catalog / Data Governance in Databricks: Organizing deliverable assets and maintaining data lineage within the solution.
• Dashboarding (Databricks SQL / Power BI): Usage heatmaps, partition access patterns, and recommendation visualizations.

AI / ML Skills:

• Machine Learning (scikit-learn, MLflow): Anomaly detection on access patterns, predictive cold-data classification.
• LLM / Generative AI Integration: LLM-assisted log interpretation, automated recommendation narrative generation.
• Feature Engineering on Time-Series Log Data: 3+ years - Extracting access frequency, recency, and seasonality signals from DBQL logs.
• Clustering & Classification Algorithms: 3+ years - Unsupervised grouping of datasets by usage behavior for cold/warm/hot tiers.
• MLflow / Experiment Tracking: 2+ years - Tracking model runs for reproducible analysis and customer handoff artifacts.
• Natural Language Processing (NLP): Parsing and classifying SQL text from DBQL for workload pattern analysis.
• Data Visualization (Plotly, Matplotlib, Seaborn): Generating usage heatmaps and partition access charts for stakeholder readouts.

Preferred Qualifications

• Familiarity with DataStage orchestration log parsing.
• Prior work on data platform cost optimization or cloud migration readiness assessments.
• Experience working in healthcare payer data environments (HIPAA awareness).
• Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs).