Senior Databricks Data Engineer — Data Validation & Testing (Remote)

Lumenalta; United States; Remote; USD 90,000 - 132,000; Full time

Benefits offered by this job

Health insurance

401K Contribution

Professional development fund

15 days paid time off

Job summary

Lumenalta is seeking a Senior Databricks Data Engineer to design and implement automated validation tests for large-scale data pipelines. You will ensure data parity with legacy systems, validate incremental loads, and prevent duplicates across sources while building a robust test framework in PySpark and SQL.

You will lead unit tests for transformation logic, verify merges and idempotency, and integrate tests into Databricks Workflows and CI/CD.

Qualifications

• 5+ years in data engineering with test/validation frameworks for production pipelines.
• Strong PySpark and SQL skills for reconciliation and assertion logic at scale.
• Hands-on experience reconciling a new platform against a legacy system or reports and explaining variances.
• Experience with Python test frameworks (pytest) and data testing tools like chispa, Great Expectations, or Delta Live Tables.

Responsibilities

• Build the automated test framework for the platform's pipelines using PySpark and SQL and run it across environments.
• Write reconciliation tests comparing Databricks tables to legacy systems and reports with clear variance outputs.
• Test incremental loading (CDC, appends) and ensure idempotency; verify merges, upserts, and late data handling.
• Develop unit tests for transformation logic to validate keying, joins, and business rules in isolation.
• Define data quality expectations within pipelines so bad data is quarantined or fails runs; document results and evidence.

Skills

PySpark

SQL

Data validation

pytest

Education

Bachelor's degree in Computer Science or related field

Tools

chispa

Great Expectations

Delta Live Tables

Lakeflow

Job description

Lumenalta is seeking a Senior Databricks Data Engineer to design and implement automated validation tests for large-scale data pipelines. You will ensure data parity with legacy systems, validate incremental loads, and prevent duplicates across sources while building a robust test framework in PySpark and SQL.

You will lead unit tests for transformation logic, verify merges and idempotency, and integrate tests into Databricks Workflows and CI/CD.