Data Test Engineer – Real-Time Pipelines

HTD Talent; Dallas (TX); On-site; USD 90,000 - 120,000; Full time

Job summary

HTD Talent is seeking experienced Data Testers to validate complex data pipelines across Databricks, Kafka, and Cosmos DB. You will ensure data accuracy in near-real-time architectures and coordinate testing activities with data engineers, developers, and business teams.

Responsibilities include end-to-end testing of data flows, API and SQL validation, and defect management through resolution. Strong SQL, testing automation, and analytical skills are essential.

Qualifications

• 3+ years of data testing or data integration testing experience.
• Hands-on with Databricks for data pipeline testing and validation.
• Hands-on with Apache Kafka including event consumption and streaming validation.
• Hands-on with Azure Cosmos DB including data validation and integration testing.
• Experience testing near-real-time data processing and event-driven architectures.
• Experience validating end-to-end data flows involving Kafka, Azure Functions, and Cosmos DB.
• Strong SQL skills for source-to-target validation and data reconciliation.
• Experience with API testing and automated test scripts.
• Defect management, troubleshooting, and production deployment validation.
• Strong communication and analytical skills.

Responsibilities

• Design and execute test plans, cases, and automated scripts based on requirements.
• Perform end-to-end data integration testing across Databricks, Kafka, Azure Functions, Cosmos DB.
• Validate Kafka event consumption, processing, and data loading into Cosmos DB.
• Test near-real-time data pipelines for accuracy and completeness.
• Perform API, functional, and integration testing.
• Write and execute SQL queries to validate data transformations.
• Prepare test data, execute tests, and document results.
• Identify, log, and track defects through resolution.
• Support defect triage and post-release validation.
• Collaborate with data engineers, developers, and business teams to resolve issues.

Skills

Data testing

Databricks

Apache Kafka

Azure Cosmos DB

End-to-end data validation

SQL

API testing

Defect management

Communication & analysis

Tools

Azure Event Hubs

ADLS Gen2

PySpark

Job description

HTD Talent is seeking experienced Data Testers to validate complex data pipelines across Databricks, Kafka, and Cosmos DB. You will ensure data accuracy in near-real-time architectures and coordinate testing activities with data engineers, developers, and business teams.

Responsibilities include end-to-end testing of data flows, API and SQL validation, and defect management through resolution. Strong SQL, testing automation, and analytical skills are essential.