Data Engineer - Real-Time Ab Initio Pipelines

Smart IT Frame LLC; Berkeley Heights (NJ); On-site; USD 120,000 - 160,000; Full time

Job summary

Smart IT Frame LLC is seeking a highly skilled data engineer to design and implement high-performance, event-driven data pipelines for a large credit card processing system in Berkeley Heights, NJ. The role focuses on building stateful services that ingest, filter, and transform data from sources like Kafka, pushing updates to dashboards and downstream databases with near-real-time latency.

You will work with Ab Initio (GDE, EME, ConductIt) to optimize continuous flows, ensure exactly-once

Qualifications

• Bachelor's degree in computer science, software engineering, or a related field.
• Experience with data pipelines and ETL design.
• Familiarity with Ab Initio components and big data platforms.

Responsibilities

• Create Ab Initio continuous flow graphs with stateful joins, windows, and aggregations.
• Implement event-driven pipelines using Kafka, MQ, and file streams.
• Ensure exactly-once processing with checkpointing and recovery.
• Monitor production streams for 24/7 reliability and troubleshoot data issues.
• Automate workflows and improve operational efficiency.

Skills

Data pipelines

Ab Initio

Kafka

Unix shell

SQL

Cloud

Education

Bachelor's degree in CS/CE

Tools

Ab Initio GDE

Kafka

Docker

Kubernetes

Teradata

Oracle

Job description

Smart IT Frame LLC is seeking a highly skilled data engineer to design and implement high-performance, event-driven data pipelines for a large credit card processing system in Berkeley Heights, NJ. The role focuses on building stateful services that ingest, filter, and transform data from sources like Kafka, pushing updates to dashboards and downstream databases with near-real-time latency.

You will work with Ab Initio (GDE, EME, ConductIt) to optimize continuous flows, ensure exactly-once