The Data Engineer Ab initio role in Houston, TX focuses on designing and implementing high-performance, event-driven Ab Initio data pipelines. The position emphasizes low-latency processing, high availability, and production reliability through continuous-flow resilience, checkpointing, and active monitoring.
Role Overview
Build and maintain Ab Initio continuous flow graphs for both batch and real-time ingestion and transformation. Deliver exactly-once processing behavior by combining continuous flow orchestration with checkpointing and recovery, while ensuring data quality, security, integrity, and compliance across big data platforms.
Key Responsibilities
- Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and aggregations.
- Implement event-driven pipelines using Kafka, MQ, and file streams.
- Ensure resilience of continuous flows through checkpointing and recovery to support exactly-once processing.
- Apply advanced Ab Initio components such as Reformat, Rollup, Join, and Partition to achieve low-latency performance.
- Monitor live production streams for 24/7 reliability, and troubleshoot data issues proactively.
- Develop ETL pipelines for batch and real-time ingestion and transformation.
- Implement and enforce data validation, data security, integrity, and compliance across big data platforms.
- Monitor and troubleshoot performance issues in large-scale clusters.
- Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions.
- Automate workflows and improve operational efficiency using scripting and orchestration tools.
Required Qualifications
- Deep understanding of credit card process system.
- Deep knowledge of GDE (Graphical Development Environment), EME (Enterprise Meta Environment), Conduct-It, and Continuous Flows.
- Understanding of Kafka, message queues, and real-time stateful services.
- Proficiency in Unix/Linux shell scripting, SQL, and database technologies such as Oracle and Teradata.
- Experience with Java, Scala, Python, or Kafka is a plus.
- Familiarity with Linux/Unix environments and shell scripting.
- Understanding of data security, governance, and compliance standards.
- Experience with cloud-based big data platforms.
- Exposure to containerization tools such as Docker and Kubernetes for big data workloads.
- Knowledge of CI/CD pipelines for data engineering projects.
Technology Stack
- Ab Initio; GDE (Graphical Development Environment); EME (Enterprise Meta Environment); Conduct-It; Continuous Flows
- Kafka; MQ; Reformat; Rollup; Join; Partition; ETL
- Unix/Linux shell scripting; SQL; Oracle; Teradata
- Java; Scala; Python; Linux/Unix
- Docker; Kubernetes
- CI/CD; checkpointing; recovery
Location, Duration, and Employment Details
- Job ID: J54188
- Location: Houston, TX (onsite)
- Duration: 8 Months + Extension
- Salary: USD 78 - 78 per hourly
- Employment Type: W-2, 1099, C2C
- Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other
- Client: To Be Discussed Later
- Domain Exposure: Material