Senior Data Engineer: Scale, Reliability & Spark

D8 LooP; Santa Clara (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

NVIDIA is seeking a data engineer to optimize distributed data processing, build robust ingestion pipelines, and ensure reliable analytics at scale. You will collaborate with teams to model data, implement incremental loads, and manage schema evolution using Databricks and Spark.

You will implement data freshness checks, monitoring, and automated recovery, while writing maintainable Python and SQL code. This role emphasizes production reliability and scalable architectures across lakehouse

Qualifications

• Experience with data modeling, incremental processing, partitioning, schema evolution and analytical query performance.
• Strong Python and SQL skills, with experience writing maintainable, tested code and debugging complex data transformations.
• Experience with automated tests, data quality checks, monitoring, incident diagnosis and recovery.
• Build reusable ingestion frameworks and data contracts for APIs and file-based sources, handling schema changes, late-arriving data and safe backfills.
• Experience with Databricks, Spark and lakehouse technologies, or comparable distributed data platforms.
• Experience with data catalogs, workflow orchestration, observability platforms or AI-assisted anomaly detection and root-cause analysis.

Responsibilities

• Own ingestion pipelines and data contracts for APIs and file-based sources, handling schema changes and late-arriving data.
• Implement data freshness, completeness and accuracy checks, lineage and monitoring.
• Diagnose incidents and automate retries, replay and recovery.

Skills

Cost & performance

Data modeling

Incremental processing

Partitioning

Schema evolution

Analytical query performance

Python

SQL

Ingestion frameworks

Databricks

Apache Spark

Airflow

Data governance

Backfills

Tools

Databricks

Apache Spark

Job description

NVIDIA is seeking a data engineer to optimize distributed data processing, build robust ingestion pipelines, and ensure reliable analytics at scale. You will collaborate with teams to model data, implement incremental loads, and manage schema evolution using Databricks and Spark.

You will implement data freshness checks, monitoring, and automated recovery, while writing maintainable Python and SQL code. This role emphasizes production reliability and scalable architectures across lakehouse