Data Engineer (Ab Initio, OzoneCH, Flink)

Compunnel Inc.; Berkeley Heights (NJ); On-site; USD 120,000 - 160,000; Full time

Job summary

Compunnel, Inc. is seeking a highly skilled engineer to design and implement high-performance, event-driven data pipelines for a large credit card processing system.

You will work with the Ab Initio ecosystem to build stateful services ingesting data from Kafka and other sources, pushing updates to dashboards and downstream databases in near-real-time. You will create complex Ab Initio graphs, implement checkpointing for exactly-once processing, monitor production streams, and optimize

Qualifications

• Deep understanding of credit card process system.
• Familiarity with Ab Initio components (GDE/EME/ConductIt) and continuous flows.
• Knowledge of Kafka, message queues, and real-time stateful services.
• Proficiency in Unix/Linux shell scripting, SQL, and database technologies (Oracle/ Teradata).

Responsibilities

• Create complex Ab Initio continuous flow graphs with stateful joins, sliding windows, and aggregations.
• Implement event-driven data pipelines using Kafka, MQ, and file streams.
• Ensure resilience of continuous flows with checkpointing and recovery for exactly-once processing.
• Apply Ab Initio components (Reformat, Rollup, Join, Partition) for low-latency performance.
• Monitor live production streams for 24/7 reliability and troubleshoot data issues.
• Develop ETL pipelines for batch and real-time ingestion and transformation.
• Implement data validation, data security, integrity, and compliance across big data platforms.
• Monitor and troubleshoot performance issues in large-scale clusters.
• Collaborate with data scientists, analysts, and application teams to deliver data solutions.
• Automate workflows and improve operational efficiency using scripting/orchestration tools.

Skills

Unix/Linux shell scripting

SQL

Tools

Ab Initio GDE/EME/ConductIt

Kafka

Docker/Kubernetes

Unix/Linux

Job description

Job Description

We are seeking a highly skilled resource to design and implement high-performance, event-driven data pipelines, ensuring low-latency data processing and high availability system for the large credit card processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE, EME, Conduct>It) to build stateful services that ingest, filter, and transform data from sources like Kafka or message queues, pushing updates to dashboards or downstream databases in near-real-time.

Key Responsibilities

• Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and aggregations.
• Implement event-driven data pipelines using Kafka, MQ, and file streams.
• Ensure the resilience of continuous flows, including checkpointing and recovery, to guarantee "exactly-once" processing.
• Apply advanced Ab Initio components (e.g., Reformat, Rollup, Join, Partition) to ensure low-latency performance.
• Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting data issues.
• Develop ETL pipelines for batch and real-time data ingestion and transformation.
• Implement and ensure data validation, data security, integrity, and compliance across big data platforms.
• Monitor and troubleshoot performance issues in large-scale clusters.
• Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions.
• Automate workflows and improve operational efficiency using scripting and orchestration tools.

Required Qualifications

• Deep understanding of credit card process system.
• Deep knowledge of GDE (Graphical Development Environment), EME (Enterprise Meta>Environment), Conduct>It, and Continuous Flows.
• Understanding of Kafka, message queues, and real-time stateful services.
• Proficiency in Unix/Linux shell scripting, SQL, and database technologies (e.g., Oracle, Teradata).
• Familiarity with Linux/Unix environments and shell scripting.
• Understanding of data security, governance, and compliance standards.

Preferred Qualifications

• Experience in Java , Scala, Python or Kafka.
• Experience with cloud-based big data platforms.
• Exposure to containerization (Docker, Kubernetes) for big data workloads.
• Knowledge of CI/CD pipelines for data engineering projects.