Data Engineer

DataVue LLC; United States; On-site; USD 110,000 - 180,000; Full time

Job summary

DataVue LLC seeks a hands-on Data Engineer to build and operate large-scale data workflows, ingesting, cleansing, matching, profiling, and integrating datasets. You will optimize pipelines, automate scheduling, and ensure data quality for analytics and modeling.

Responsibilities include engineering Python/Spark ETL processes in AWS, developing match keys, maintaining data dictionaries, and supporting production data requests. Bachelor’s degree and US work authorization required.

Qualifications

• Advanced Python engineering for large-scale processing and automation.
• Advanced SQL with complex joins and window functions.
• Hands-on Spark/PySpark for distributed data jobs.
• Production AWS experience (S3, Glue/EMR) and secure data access patterns.
• Experience orchestrating multi-step data workflows (Airflow, Dagster, Step Functions, or Glue) including retries.
• Ability to interpret data dictionaries and communicate in a remote team.
• Bachelor's degree; authorized to work in the United States without visa sponsorship.

Responsibilities

• Build and maintain data pipelines and automation.
• Perform data cleansing, matching, and record-linkage across sources.
• Develop attribute profiling and data quality checks.
• Design and optimize staging and downstream tables; manage SQL performance.
• Support historical data capabilities and time-series attributes.

Skills

Python

SQL

Spark/PySpark

AWS

Data pipelines

Data quality

Education

Bachelor's degree

Tools

S3

EMR

Glue

Dagster

Airflow

Step Functions

Parquet

Iceberg

Job description

About DataVue

DataVue provides consumer credit data, audience solutions, and predictive analytics to lenders and other businesses. Our data environment spans more than one billion records, with individual source files exceeding 200 million records and some datasets containing more than 4,000 attributes.

About DataVue

DataVue provides consumer credit data, audience solutions, and predictive analytics to lenders and other businesses. Our data environment spans more than one billion records, with individual source files exceeding 200 million records and some datasets containing more than 4,000 attributes.

The role

We are looking for a hands‑on Data Engineer. You will build and operate workflows that ingest, cleanse, match, profile, and integrate large datasets, making them reliable and usable for customer delivery, analytics, and modeling.

Major feeds refresh approximately every four to six weeks. Between refreshes, you will improve automation, onboard sources, maintain data dictionaries, investigate quality issues, and support production data requests. Database responsibilities focus on tables, processing logic, and performance optimization; infrastructure administration is handled separately.

What You Will Do

Data pipelines and automation

• Build and maintain Python and Spark ETL/ELT workflows in AWS. Manage source mappings, schema changes, full and incremental loads, reconciliation, and recovery from failed processing
• Automate scheduling, dependencies, logging, alerts, retries, and safe reruns. Optimize runtime, memory use, and processing costs; maintain version-controlled code, tests, and runbooks

Data cleansing and matching

• Integrate CASS address processing and other cleansing steps. Develop match keys, crosswalks, and record‑linkage logic connecting consumers, households, addresses, and properties across sources
• Handle duplicates, conflicting values, missing identifiers, and ambiguous matches. Monitor match quality and coverage, prevent unintended record multiplication, and maintain consistent identities across refreshes

Attribute profiling and data quality

• Build configurable profiling across thousands of attributes. Measure meaningful population rates excluding blanks and placeholder codes, validity, distinct values, distributions, ranges, percentiles, and concentration as appropriate
• Monitor drift between deliveries and assess coverage within relevant populations. Establish quality checks and exception reports, investigate discrepancies, and validate refreshed data before production release
• Maintain data dictionaries, source lineage, definitions, and quality findings. Help technical, modeling, sales, and operations teams understand attribute usability and limitations

Database development and data operations

• Design and maintain staging, integrated, and downstream tables. Optimize SQL, large joins, schemas, indexing, partitioning, and data distribution as appropriate to the platform
• Support customer orders, audience extractions, suppressions, match‑and‑append requests, and modeling datasets as priorities require. Apply established controls for sensitive consumer data

Historical data and derived attributes

• Contribute to historical data capabilities using Apache Iceberg or comparable technology, with potential petabyte‑scale requirements. Support historical extracts for modeling and potential data products
• Preserve source versions, effective and ingestion dates, and transformation lineage. Build reproducible point‑in‑time datasets that reflect when information became available
• Implement and validate time‑series attributes, including changes, trends, rolling measures, and event counts, using definitions developed with business and ML teams. Account for missing periods, corrections, and changing source or matching rules

Requirements

Required experience

• Advanced Python engineering: production‑grade, reusable code for large‑scale processing and automation, with strong debugging, memory management, testing, and error‑handling skills
• Advanced SQL and database development: complex joins, window functions, execution‑plan analysis, data modeling, and performance optimization across large tables
• Hands‑on Spark/PySpark: building and optimizing distributed jobs, including partitioning, large joins, shuffles, data skew, and job troubleshooting
• Production AWS experience: S3, processing services such as Glue or EMR, and secure data access patterns (IAM, encryption, least‑privilege)
• Demonstrated processing experience with hundreds of millions of records, efficient storage formats such as Parquet, and practical methods for avoiding unnecessary scans and recomputation
• Experience orchestrating multi‑step data workflows with a production scheduler such as Airflow, Dagster, Step Functions, or Glue workflows, including dependency management, retries, and alerting
• Cross‑source matching, deduplication, attribute profiling, data validation, and investigation of quality issues
• Ownership of reliable production workflows, including Git, code review, automated testing, controlled releases, monitoring, and recovery
• Ability to interpret complex data dictionaries, translate business requirements into accurate processing logic, and communicate clearly in a remote team
• Bachelor's degree
• Authorized to work in the United States without current or future visa sponsorship

Preferred experience

• SingleStore or comparable distributed SQL databases
• Apache Iceberg and historical data lake or lakehouse development
• point‑in‑time modeling datasets and time‑series feature pipelines
• Consumer credit, property, mortgage, or direct marketing data
• CASS processing and postal data hygiene