Ab Initio Real-Time Data Pipeline Architect - Onsite NJ

Mphasis; Berkeley Heights (NJ); On-site; USD 140,000 - 190,000; Full time

Job summary

Mphasis in New Jersey seeks a data engineer to design and implement high-performance, event-driven data pipelines for a large credit card processing system. You will work with Ab Initio (GDE, EME, ConductIt) to build stateful services ingesting data from Kafka and message queues for near-real-time dashboards.

Responsibilities include ensuring exactly-once processing, monitoring 24/7 production streams, and collaborating with data scientists and developers to deliver reliable data solutions.

Qualifications

• Experience designing Ab Initio continuous flow graphs with stateful joins and sliding windows.
• Proficient with Kafka, MQs, and real-time streaming.
• Knowledge of data security, governance, and compliance.
• Experience with large-scale data processing and dashboards.
• Background in Unix/Linux and SQL databases.
• Experience with cloud-based big data platforms.
• Exposure to containerization for big data workloads.

Responsibilities

• Design and implement high-performance, event-driven data pipelines using Ab Initio.
• Build stateful services ingesting data from Kafka or message queues.
• Ensure exactly-once processing with checkpointing and recovery.
• Apply Ab Initio components like Reformat, Rollup, Join, Partition for low latency.
• Monitor production streams for 24/7 reliability and troubleshoot data issues.
• Develop ETL pipelines for batch and real-time ingestion.
• Ensure data validation, security, integrity, and compliance across platforms.
• Monitor and troubleshoot performance in large-scale clusters.
• Collaborate with data scientists and developers to deliver data solutions.
• Automate workflows using scripting and orchestration tools.

Skills

Ab Initio GDE/EME/ConductIt

Kafka

Stateful data pipelines

Unix/Linux shell scripting

SQL / Oracle / Teradata

Java / Scala / Python

CI/CD pipelines

Docker

Kubernetes

Education

Bachelor’s degree in Computer Science, Software Engineering, or related field

Tools

Ab Initio software suite

Docker

Kubernetes

Oracle

Teradata

Job description

Mphasis in New Jersey seeks a data engineer to design and implement high-performance, event-driven data pipelines for a large credit card processing system. You will work with Ab Initio (GDE, EME, ConductIt) to build stateful services ingesting data from Kafka and message queues for near-real-time dashboards.

Responsibilities include ensuring exactly-once processing, monitoring 24/7 production streams, and collaborating with data scientists and developers to deliver reliable data solutions.