Senior Databricks & PySpark Production Engineer

NTT DATA, Inc.; Addison (TX); In loco; USD 76.000 - 85.000; Tempo pieno

Vantaggi offerti da questo lavoro

Medical insurance

Dental insurance

Vision insurance

401(k) match

Paid time off

Descrizione del lavoro

NTT DATA, Inc. in Addison, TX seeks a Senior Data Engineering Support Engineer with expert Databricks and PySpark skills.

You will anchor enterprise data pipelines, drive high availability, performance, and cost optimization while leading RCA, building self-healing scripts, and mentoring junior staff. Responsibilities include incident management, performance tuning, cost optimization, governance, and SOP creation.

Competenze

• 7+ years of total experience in Data Engineering.
• 3+ years dedicated to supporting or optimizing large-scale Databricks and PySpark systems in production.

Mansioni

• Act as the highest tier of internal escalation for complex, high-priority production failures in massive-scale batch and streaming PySpark jobs.
• Drive the resolution of complex distributed computing bottlenecks, including data skew, OOM errors, and costly shuffle operations.
• Analyze Databricks cluster utilization and run strategies to reduce cloud compute spend.
• Oversee data access control, schema evolution, and data lineage tracking within Databricks Unity Catalog.
• Build robust automated frameworks and scripts in Python/PySpark for data reconciliation and self-healing pipelines.
• Author technical runbooks and post-mortem RCA documents; establish SOPs to reduce MTTR.
• Provide mentorship and code reviews to junior support engineers and collaborate with core development teams.

Conoscenze

Databricks

PySpark

Python

SQL

Delta Lake

Unity Catalog

RCA

SRE

CI/CD

Cloud compute optimization

Strumenti

Databricks

Unity Catalog

Delta Live Tables

Descrizione del lavoro

NTT DATA, Inc. in Addison, TX seeks a Senior Data Engineering Support Engineer with expert Databricks and PySpark skills.

You will anchor enterprise data pipelines, drive high availability, performance, and cost optimization while leading RCA, building self-healing scripts, and mentoring junior staff. Responsibilities include incident management, performance tuning, cost optimization, governance, and SOP creation.