Ab Initio Data Engineer – Real-Time, Low-Latency Pipelines

Envision Technology Solutions; Berkeley Heights (NJ); On-site; USD 120,000 - 170,000; Full time

Job summary

Envision Technology Solutions is seeking a high-performance data engineer to design and implement event-driven data pipelines for a large credit card processing system. You will work with Ab Initio components to ingest, filter, and transform data from Kafka and other sources in near-real-time.

The role emphasizes low-latency, exactly-once processing, checkpointing, and resilience of continuous flows, plus collaboration with data scientists and application teams.

Qualifications

• Deep knowledge of credit card processing systems.
• Strong expertise with GDE, EME, Conduct It and continuous flows.
• Experience with Kafka, messaging queues and real-time stateful services.
• Proficiency in Unix/Linux shell, SQL and database tech (Oracle/Teradata).
• Experience with cloud-based big data platforms and containerization.

Responsibilities

• Create complex Ab Initio continuous flow graphs with stateful joins and windows.
• Implement event-driven data pipelines using Kafka, MQ, and file streams.
• Ensure resilience of flows with checkpointing and exactly-once processing.
• Apply Ab Initio components (Reformat, Rollup, Join, Partition) for low latency.
• Monitor live production streams 24/7 and troubleshoot data issues.
• Develop ETL pipelines for batch and real-time ingestion and transformation.
• Ensure data validation, security, integrity, and compliance across platforms.
• Monitor and troubleshoot performance in large-scale clusters.
• Collaborate with data scientists, analysts, and application teams to deliver data solutions.
• Automate workflows and improve operational efficiency with scripting and orchestration.

Skills

Problem solving

Data wrangling

Real-time processing

Tools

GDE

EME

Conduct It

Kafka

Docker

Kubernetes

Unix shell scripting

SQL

Java/Scala/Python

Oracle/Teradata

Job description

Envision Technology Solutions is seeking a high-performance data engineer to design and implement event-driven data pipelines for a large credit card processing system. You will work with Ab Initio components to ingest, filter, and transform data from Kafka and other sources in near-real-time.

The role emphasizes low-latency, exactly-once processing, checkpointing, and resilience of continuous flows, plus collaboration with data scientists and application teams.