Data Quality Engineer - Spark, Kafka & AWS (Hybrid Dallas)

Select Minds LLC; Dallas (TX); Hybrid; USD 120,000 - 150,000; Full time

Benefits offered by this job

W2 Opportunity

Competitive salary

Opportunity for advancement

Job summary

Select Minds LLC is seeking a Data Quality Engineer to ensure the reliability and correctness of data pipelines. This hands-on role focuses on validating data across platforms like Databricks and AWS. Candidates must have 7+ years of experience, with specific expertise in data validation, SQL, Python, and streaming platforms like Kafka. The position offers a competitive salary and the opportunity for advancement in a hybrid work environment in Dallas, TX.

Qualifications

• 7+ years of total experience in QA, SDET, or Data Quality Engineering.
• Minimum 4–6 years of hands-on experience with data platforms.
• 3+ years of hands-on experience with Databricks and Apache Spark.
• Strong SQL skills for data validation.
• Proficiency in Python for automation.

Responsibilities

• Validate data pipelines for accuracy, completeness, consistency, and timeliness.
• Build SQL-based validations for business rules.
• Test pipelines built on AWS services.
• Validate data integrity and delivery guarantees in Kafka.
• Develop reusable validation utilities and synthetic datasets.

Skills

Data Quality Engineering

Validation Testing

Databricks

AWS

SQL

Python

Kafka

ETL/ELT

Apache Spark

Job description

Benefits:

• W2 OPPORTUNITY
• Competitive salary
• Opportunity for advancement

Job Title: Data Quality Engineer (Databricks, Kafka, AWS)
Location: Dallas, TX (Hybrid – 3 days onsite)
Job Type: Long-term Contract
Work Authorization: Open - W2 opportunity
Interview Process: In-person (Client interview- Mandatory)

We are looking for a Data Quality Engineer to own validation across batch and streaming data pipelines. This role focuses on ensuring data correctness, reliability, and performance across platforms built on Databricks, Kafka, AWS, SQL, and Python.
This is a hands-on role focused on building scalable data validation frameworks and ensuring production-grade data systems.

Key Responsibilities
End-to-End Data Validation
* Validate data pipelines for accuracy, completeness, consistency, and timeliness
* Build SQL-based validations for business rules and transformations
* Implement reconciliation between source and downstream systems
* Ensure data lineage and traceability

ETL / ELT & Spark Testing
* Test pipelines built on AWS (Glue, Lambda, EMR, Step Functions)
* Validate transformations using SQL and Python
* Test ingestion, transformation, aggregation, and serving layers
* Handle backfills, reprocessing, and historical data loads
* Validate Spark pipelines (PySpark/Scala) on Databricks

Streaming (Kafka)
* Validate data integrity, ordering, and delivery guarantees
* Test producer and consumer logic and serialization formats (Avro, JSON, Protobuf)
* Validate topics, partitions, offsets, retention, and schema evolution
* Simulate late events, duplicates, and failure scenarios

Automation & Frameworks
* Build Python-based data testing frameworks
* Develop reusable validation utilities and synthetic datasets
* Integrate data tests into CI/CD pipelines
* Enable automated alerts for data quality issues

Performance & Reliability
* Validate throughput, latency, and concurrency at scale
* Test retry logic, idempotency, and recovery mechanisms
* Perform regression, soak, and failover testing

Observability
* Validate logs, metrics, and alerts using tools such as CloudWatch, Prometheus, and Grafana
* Define and monitor data SLAs and SLOs
* Support incident response, root cause analysis, and postmortems

Required Qualifications & Experience
* 7+ years of total experience in QA, SDET, or Data Quality Engineering
* Minimum 4–6 years of hands-on experience working with data platforms, data pipelines, or data engineering ecosystems
* 3+ years of hands-on experience with Databricks and Apache Spark
* Strong SQL skills for data validation, reconciliation, and complex analysis
* Proficiency in Python for automation and data validation
* Experience testing ETL/ELT pipelines (batch and streaming)
* Hands-on experience with Kafka or similar streaming platforms
* Strong understanding of AWS data services (S3, Glue, Lambda, Redshift, Athena)
* Experience working with large-scale distributed data systems
* Strong debugging, analytical, and problem-solving skills

Nice to Have
* Experience with data quality or observability tools such as Great Expectations or Monte Carlo
* Knowledge of schema registry and data contracts
* Experience with CI/CD tools such as GitHub Actions or Jenkins

Flexible work from home options available.