Data Integration Engineer :: Remote

Feuji; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

Feuji is seeking an integration-focused engineer to design, build, and maintain end-to-end integration pipelines. The role emphasizes containerized Python services on Azure, Event Hub and Service Bus-based flows, and robust error handling to ensure reliable data movement between source systems and targets.

Responsibilities include implementing idempotent processing, API design with OAuth 2.0, and transformations that enforce business rules.

Responsibilities

• Integration Development: Design, build, and maintain integration pipelines connecting source systems to downstream targets, using containerized Python services on Azure Container Apps with Service Bus and Event Hub.
• Event-Driven and Streaming Integration: Build and operate flows against Salesforce's Pub/Sub API over gRPC, including Platform Event publish and subscribe, Avro decode, durable replay checkpointing, and reconnection handling for long-lived subscriptions.
• Reliability and Failure Handling: Implement idempotent processing, deduplication, retry policies, and dead-letter consumers so at-least-once delivery does not produce duplicate or lost records in target systems.
• API Design and Authentication: Design and consume REST APIs, including OAuth 2.0 with token caching, rate limit handling, and error semantics that distinguish retryable from terminal failures.
• Transformation and Business Logic: Build and maintain Python-based transformation and validation services that enforce business rules as data moves between systems.
• Data Platform Integration: Land request and response event streams into Snowflake via Event Hub as a durable parallel stream, so integration activity is available for reporting, reconciliation, and audit.
• Observability and Handoff to Operations: Instrument integrations for distributed tracing, structured logging, and correlation across asynchronous hops, and deliver monitoring hooks, alert thresholds, runbooks, and escalation paths to Operations before go-live.
• Testing, CI/CD, and Requirements: Write unit and integration tests covering failure and replay paths, maintain CI/CD pipelines (e.g., GitHub Actions) across DEV, SIT, UAT, and PROD, and translate stakeholder requirements into technical designs, including which pattern applies.

Job description

• Integration Development: Design, build, and maintain integration pipelines connecting source systems to downstream targets, using containerized Python services on Azure Container Apps with Service Bus and Event Hub.
• Event-Driven and Streaming Integration: Build and operate flows against Salesforce's Pub/Sub API over gRPC, including Platform Event publish and subscribe, Avro decode, durable replay checkpointing, and reconnection handling for long-lived subscriptions.
• Reliability and Failure Handling: Implement idempotent processing, deduplication, retry policies, and dead-letter consumers so at-least-once delivery does not produce duplicate or lost records in target systems.
• API Design and Authentication: Design and consume REST APIs, including OAuth 2.0 with token caching, rate limit handling, and error semantics that distinguish retryable from terminal failures.
• Transformation and Business Logic: Build and maintain Python-based transformation and validation services that enforce business rules as data moves between systems.
• Data Platform Integration: Land request and response event streams into Snowflake via Event Hub as a durable parallel stream, so integration activity is available for reporting, reconciliation, and audit.
• Observability and Handoff to Operations: Instrument integrations for distributed tracing, structured logging, and correlation across asynchronous hops, and deliver monitoring hooks, alert thresholds, runbooks, and escalation paths to Operations before go-live.
• Testing, CI/CD, and Requirements: Write unit and integration tests covering failure and replay paths, maintain CI/CD pipelines (e.g., GitHub Actions) across DEV, SIT, UAT, and PROD, and translate stakeholder requirements into technical designs, including which pattern applies.