Automated Data Quality Engineer

Habemco; United States; Remote; USD 80,000 - 110,000; Full time

Job summary

A leading company is seeking an experienced Automated Data QA Engineer to join their data engineering team. This role involves ensuring data accuracy and reliability through automated validation scripts and testing processes. The ideal candidate will collaborate with various teams to support data workflows and quality benchmarks, while also optimizing data assurance processes. Competitive pay, benefits, and a collaborative work environment are offered.

Benefits

Competitive pay

Quarterly bonuses

401(k) with 4% match

Qualifications

• 4+ years experience as an SDET or Data QA Engineer focusing on data pipelines.
• Strong understanding of data testing methodologies and tools.

Responsibilities

• Develop and maintain automated data test frameworks for AWS Glue ETL processes.
• Monitor pipeline stability, performance, and error handling.

Skills

Problem-Solving

Data Optimization

Communication

Adaptability

Education

Bachelor’s degree in Computer Science

Master’s degree in relevant fields

Tools

PyTest

AWS Glue

GitHub Actions

Jenkins

Job description

Lead Technical Recruiter at DHI Group, Inc.

Habemco is a shared services company wholly owned and operated by the Habematolel Pomo of Upper Lake, a federally recognized Native American tribe located in Northern California. Habemco’s support services such as product development and technology are needed for business growth, powering the Tribe’s economy, and enabling education, health care, and elder support programs for the Tribal community. Our talented team provides cross-functional support services to various tribal business and government entities. The Habemco team plays a critical role in ensuring a successful future for our customers, employees, and the Tribe.

Headquartered in a remote part of California, the Tribe recognizes that to compete in industries like FinTech, it must access expertise nationwide. Employees work remotely or at our headquarters in Upper Lake, California, and at a campus in Lenexa, Kansas.

Employees receive competitive pay and benefits, quarterly bonuses, and a 401(k) with a 4% match. Our team is creative, forward-thinking, passionate, and fast-moving. Are you ready to grow with us?

Purpose of the Position:

We seek an experienced Automated Data QA Engineer to join our data engineering team. You will ensure data accuracy, consistency, and reliability by designing and maintaining automated validation scripts, executing data quality tests, and identifying anomalies early in the data pipeline. Your expertise in data testing, scripting, and cloud technologies will be vital in optimizing our data quality assurance processes and ensuring high standards of data integrity and usability. The role involves both independent and collaborative work, contributing to complex project aspects.

Key Responsibilities:

1. Develop and maintain automated data test frameworks for AWS Glue ETL processes, Lambda functions, and PySpark SQL workflows using tools like PyTest and Spark’s testing capabilities.
2. Design and execute test cases to ensure data accuracy, completeness, and consistency across pipelines and Data Lake components.
3. Monitor pipeline stability, performance, and error handling; collaborate to improve pipeline efficiency.
4. Partner with data engineers, scientists, and business teams to understand requirements and support data workflows and quality benchmarks.
5. Create and run unit and integration tests for ETL jobs, Lambda handlers, and PySpark transformations.
6. Test PySpark SQL logic with in-memory datasets for accuracy and resilience.
7. Integrate testing into CI/CD pipelines using tools like AWS CodePipeline, GitHub Actions, and CodeDeploy.
8. Manage test data and environments, creating synthetic datasets that mimic production conditions.
9. Implement data quality rules in AWS Glue using DQDL to validate outputs.
10. Assess and improve test coverage with tools like pytest-cov, identifying untested pipeline segments.
11. Troubleshoot and debug test failures, ensuring data accuracy and performance.
12. Collaborate with development teams to embed TDD practices and improve testability.
13. Document testing frameworks and strategies, sharing best practices and training materials.

Education and Experience:

Required:

• Bachelor’s degree in Computer Science, Data Engineering, or related field, or 4+ years in Automated QA roles.
• 4+ years experience as an SDET, Data QA Engineer, or similar, focusing on data pipelines.
• Knowledge of CI tools like Jenkins, GitHub Actions, AWS CodePipeline.
• Experience with version control (Git) and collaborative development.
• Strong understanding of data testing methodologies and tools like PyTest.
• Ability to produce clear test documentation and reports.
• Excellent debugging and problem-solving skills for data anomalies and failures.
• Strong communication skills for collaboration with technical and non-technical teams.
• Legal work authorization in the U.S. without sponsorship requirements.

Preferred:

• Master’s degree in relevant fields.
• Experience with cloud data services and testing in cloud environments.
• Familiarity with Agile methodologies.

Skills & Abilities:

• Advanced problem-solving and data optimization skills.
• Effective prioritization and project management.
• Willingness to learn, share skills, and work in a team.
• Adaptability, change management, and team rallying abilities.
• Decision-making and debugging expertise.
• Ability to communicate technical ideas clearly to varied audiences.
• Strong interpersonal and professional communication skills.
• Ability to work in a fast-paced, confidential environment.

Physical Requirements:

• Prolonged sitting and working at a computer.
• Ability to differentiate wire and cable colors and tones.

Seniority level

• Associate

Employment type

• Full-time

Job function

• Information Technology