Senior Data Engineer: Real-Time AI-Driven Data Platform

Valid8 Financial, Inc.; New York (NY); On-site; USD 130,000 - 190,000; Full time

Job summary

BDIPlus, a data and AI consulting firm, seeks a Senior Data Engineer to support a Fortune 100 financial services client’s real-time intelligence initiatives.

In a virtualization-first architecture using Denodo, you’ll build governed virtual data models, ensure high quality data, and establish trusted data assets with formal contracts and full lineage.

This US-based role in New York emphasizes data lakehouse storage on AWS, API development, and AI-assisted delivery to accelerate outcomes.

Qualifications

• 5+ years of data engineering experience in enterprise-scale environments.
• Strong experience with Denodo and VQL development, query optimization, caching, governance.
• Experience with Apache Iceberg on AWS (S3, Glue, Athena), including schema evolution and partitioning.
• Hands-on experience with Informatica.
• Experience implementing data quality frameworks.
• Strong SQL and programming skills (Python, Spark; dbt preferred).
• Experience with CDC technologies (e.g., Debezium, Informatica CDC, or equivalent).
• Familiarity with API development (FastAPI, Node.js) and OpenAPI standards.
• Experience working with AI-assisted development tools (Claude Code preferred).

Responsibilities

• Build governed virtual data models in Denodo, including cross-system joins, canonical schemas, ABAC policies, and column-level masking.
• Configure and optimize semantic layer connectivity (JDBC/ODBC, connection pooling, failover).
• Implement data quality rules (completeness, validity, uniqueness, anomaly detection, scoring).
• Configure data catalog, lineage, and marketplace publishing.
• Build physical data products: source extraction (mainframe, CDC, COBOL copybooks); Data Lakehouse storage using Apache Iceberg on AWS S3.
• Define and implement machine-readable data contracts (YAML/JSON), including schema guarantees, SLAs, and dependency tracking.
• Develop APIs, SLA monitoring dashboards, reconciliation processes, and production-grade delivery patterns.
• Partner with client teams through demos, rotations, and structured knowledge transfer.
• Leverage AI-assisted development tools (Claude Code) to accelerate delivery and improve productivity.

Skills

Denodo

Iceberg on AWS

Informatica

Data quality

SQL

Python

Spark

dbt

CDC tech

API development

OpenAPI

Claude Code

Tools

FastAPI

Node.js

OpenAPI

AWS

Job description

BDIPlus, a data and AI consulting firm, seeks a Senior Data Engineer to support a Fortune 100 financial services client’s real-time intelligence initiatives.

In a virtualization-first architecture using Denodo, you’ll build governed virtual data models, ensure high quality data, and establish trusted data assets with formal contracts and full lineage.

This US-based role in New York emphasizes data lakehouse storage on AWS, API development, and AI-assisted delivery to accelerate outcomes.