Remote Senior Databricks Data Engineer & SRE Lead

NTT DATA, Inc.; Oregon; Hybrid; USD 92,000 - 138,000; Full time

Benefits offered by this job

Health insurance

HSA/FSA

Life insurance

Disability insurance

Paid time off

401k with match

Employee assistance program

Job summary

NTT DATA is seeking a Senior Data Engineering Support Engineer to join our team. The role focuses on Databricks and PySpark expertise, serving as the technical anchor for enterprise big data operations, ensuring high availability, performance optimization, and cost efficiency across production pipelines.

You will lead root-cause analyses, design automated self-healing scripts, and mentor junior staff to maintain Delta Lakehouse stability.

Qualifications

• 7+ years of total experience in Data Engineering.
• 3+ years dedicated to supporting or optimizing large-scale Databricks and PySpark systems in production.

Responsibilities

• Act as the highest tier of internal escalation for complex, high-priority production failures in massive-scale batch and streaming PySpark jobs.
• Drive the resolution of complex distributed computing bottlenecks, including data skew, OOM errors, partition pruning issues and costly shuffle operations.
• Analyze Databricks cluster utilization and run strategies to reduce cloud compute spend (auto-scaling, spot instances, driver/executor types).
• Oversee and troubleshoot data access control, schema evolution, and data lineage tracking within Databricks Unity Catalog.
• Build automated frameworks and scripts in Python/PySpark to automate data reconciliation, system health checks, and self-healing pipelines.
• Author comprehensive runbooks and post-mortem RCA documents; establish SOPs to reduce MTTR.
• Provide technical guidance to junior engineers, conduct code reviews for hotfixes, collaborate with core development teams.

Skills

Databricks

PySpark

Python

SQL

Delta Lake

Unity Catalog

Cloud platforms

CI/CD

Troubleshooting

RCA

Tools

Databricks UI

Spark

Ganglia

Spark UI

Version control

Job description

NTT DATA is seeking a Senior Data Engineering Support Engineer to join our team. The role focuses on Databricks and PySpark expertise, serving as the technical anchor for enterprise big data operations, ensuring high availability, performance optimization, and cost efficiency across production pipelines.

You will lead root-cause analyses, design automated self-healing scripts, and mentor junior staff to maintain Delta Lakehouse stability.