Data Engineer Ab initio

Ethereum Technologies LLC

Houston (TX)

On-site

USD 89,000 - 126,000

Full time

12 days ago

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Ethereum Technologies is seeking a Data Engineer specialized in Ab Initio to design and run high-performance, event-driven data pipelines in Houston, TX. You will work with the Ab Initio ecosystem to ingest, filter, and transform data from sources like Kafka, pushing updates to dashboards and downstream databases in near real-time.

The role emphasizes building stateful services, ensuring exactly-once processing, and maintaining 24/7 reliability across large clusters.

Qualifications

  • Experience with Ab Initio ecosystem (GDE, EME, ConductIt) and continuous flows.
  • Familiarity with Kafka, MQ, and real-time stateful services.
  • Proficiency in Unix/Linux shell scripting, SQL, and database tech (Oracle, Teradata).
  • Knowledge of containerization (Docker, Kubernetes) for big data workloads.
  • Understanding of CI/CD pipelines for data engineering projects.

Responsibilities

  • Create Ab Initio continuous flow graphs with stateful joins, windows, and aggregations.
  • Build event-driven data pipelines using Kafka, MQ, and file streams.
  • Ensure exactly-once processing with checkpointing and recovery.
  • Apply Ab Initio components to optimize low-latency performance.
  • Monitor production streams for 24/7 reliability and troubleshoot data issues.
  • Develop ETL pipelines for batch and real-time ingestion and transformation.
  • Ensure data security, governance, and compliance across big data platforms.
  • Monitor large-scale clusters and troubleshoot performance issues.
  • Collaborate with data scientists and analysts to deliver data solutions.
  • Automate workflows using scripting and orchestration tools.

Skills

Credit card processing
Ab Initio
GDE
EME
ConductIt
Kafka
Unix/Linux scripting
SQL
Java/Scala/Python
CI/CD for data pipelines
Docker/Kubernetes

Tools

Docker
Kubernetes
Oracle
Teradata

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Job Title:Data Engineer Ab initio
Location: Houston, TX
Duration: 8 Months
Bill Rate: $78/hour (C2C)
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other
Job Description:

We are seeking a highly skilled resource to design and implement high-performance, event-driven data pipelines, ensuring low-latency data processing and high availability system for the large credit card processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE, EME, Conduct>It) to build stateful services that ingest, filter, and transform data from sources like Kafka or message queues, pushing updates to dashboards or downstream databases in near-real-time.

Key Responsibilities
  • Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and aggregations.
  • Implement event-driven data pipelines using Kafka, MQ, and file streams.
  • Ensure the resilience of continuous flows, including checkpointing and recovery, to guarantee "exactly-once" processing.
  • Apply advanced Ab Initio components (e.g., Reformat, Rollup, Join, Partition) to ensure low-latency performance.
  • Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting data issues
  • Develop ETL pipelines for batch and real-time data ingestion and transformation.
  • Implement and ensure data validation, data security, integrity, and compliance across big data platforms.
  • Monitor and troubleshoot performance issues in large-scale clusters.
  • Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions.
  • Automate workflows and improve operational efficiency using scripting and orchestration tools.
Required Skills & Qualifications
  • Deep understanding of credit card process system
  • Deep knowledge of GDE (Graphical Development Environment), EME (Enterprise Meta>Environment), Conduct>It, and Continuous Flows.
  • Understanding of Kafka, message queues, and real-time stateful services.
  • Proficiency in Unix/Linux shell scripting, SQL, and database technologies (e.g., Oracle, Teradata).
  • Experience in Java , Scala, Python or Kafka is plus.
  • Familiarity with Linux/Unix environments and shell scripting.
  • Understanding of data security, governance, and compliance standards.
  • Experience with cloud-based big data platforms
  • Exposure to containerization (Docker, Kubernetes) for big data workloads.
  • Knowledge of CI/CD pipelines for data engineering projects.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer Ab initio
Data Engineer Ab initio

Quantum Technologies LLC • Houston (TX)

On-site
USD 89,000 - 126,000
Data Engineer Ab initio
Data Engineer Ab initio

Valiant Technologies LLC • Houston (TX)

On-site
USD 89,000 - 126,000
Data Engineer Ab initio
Data Engineer Ab initio

Robotics Technologies • Houston (TX)

On-site
USD 89,000 - 126,000
Abintio Data Engineer
Abintio Data Engineer

Envision Technology Solutions • Berkeley Heights (NJ)

On-site
USD 120,000 - 170,000
Real-Time Data Engineer: Ab Initio & Kafka Pipelines
Real-Time Data Engineer: Ab Initio & Kafka Pipelines

Compunnel Inc. • Berkeley Heights (NJ)

On-site
USD 120,000 - 160,000
Ab Initio Consultant
Ab Initio Consultant

Mphasis • Berkeley Heights (NJ)

On-site
USD 140,000 - 190,000
Data Engineer (Ab Initio, OzoneCH, Flink)
Data Engineer (Ab Initio, OzoneCH, Flink)

Compunnel Inc. • Berkeley Heights (NJ)

On-site
USD 120,000 - 160,000
Ab Initio Data Engineer — Real-Time Pipelines
Ab Initio Data Engineer — Real-Time Pipelines

Quantum Technologies. LLC • Houston (TX)

On-site
USD 89,000 - 126,000
Ab Initio Developer
Ab Initio Developer

7Th Sky Techologies llc • Charlotte (NC)

Hybrid
USD 100,000 - 140,000
Real-Time Data Engineer (Ab Initio) - Kafka & Streams
Real-Time Data Engineer (Ab Initio) - Kafka & Streams

Valiant Technologies LLC • Houston (TX)

On-site
USD 89,000 - 126,000