Remote Senior Databricks Data Engineer: Validation & Testing

Lumenalta; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

Lumenalta is seeking a data engineer to design and implement automated tests for its large-scale data platforms. You will write reconciliation tests between Databricks tables and legacy reports, validate incremental loads, and ensure idempotent re-runs while preventing duplicates across sources.

Responsibilities include building PySpark/SQL test frameworks, integrating tests into Databricks Workflows and CI/CD, and defining data quality gates.

Qualifications

• 5+ years in data engineering, including building test/validation frameworks for production pipelines.
• Strong PySpark and SQL, used to write reconciliation and assertion logic at scale.
• Hands-on experience reconciling a new platform against a legacy system or its reports.
• Deep understanding of incremental load correctness: CDC, idempotency, MERGE, late-arriving data.
• Experience with Python test frameworks like pytest and data testing tools (chispa, Great Expectations).
• Experience with Databricks Workflows and CI/CD across environments.
• Clear written communication; plan and report test evidence and variances.

Responsibilities

• Build the automated test framework for pipelines in PySpark and SQL and run it across environments.
• Write reconciliation tests comparing Databricks tables to legacy systems and reports with clear variance output.
• Test incremental loading, ensure idempotency, correct merges/upserts, and no double-loads.
• Check duplicates on composite keys and cross-source relationships.
• Write unit tests for transformation logic to validate keys, joins, and business rules.
• Define data quality checks within pipelines to quarantine bad data.
• Test behavior when source files change shape (new/renamed/missing columns).
• Integrate tests with Databricks Workflows and the CI/CD pipeline for gatekeeping deployments.
• Verify Unity Catalog access controls, row filters, and data masking for restricted data.
• Document test results and defects, and present weekly quality status.

Skills

Data engineering

PySpark

SQL

Data validation

Unit testing

Delta Lake

CI/CD

Databricks Workflows

US citizenship

Clear written communication

Tools

Databricks

Great Expectations

chispa

Delta Live Tables

Lakewood?

Job description

Lumenalta is seeking a data engineer to design and implement automated tests for its large-scale data platforms. You will write reconciliation tests between Databricks tables and legacy reports, validate incremental loads, and ensure idempotent re-runs while preventing duplicates across sources.

Responsibilities include building PySpark/SQL test frameworks, integrating tests into Databricks Workflows and CI/CD, and defining data quality gates.