Senior Python Kafka Engineer DRS Platform Modernization || Boston MA/Remote

Envision Technology Solutions; Boston (MA); On-site; USD 140,000 - 190,000; Full time

Job summary

Envision Technology Solutions seeks an experienced Senior Software Engineer to enhance an event-driven digital repository platform. The role focuses on observability, event lineage, and Kafka delivery reliability across a distributed microservices ecosystem built primarily on Python and Kafka.

You will collaborate with the client’s internal engineering team to implement solutions in a live production environment.

Qualifications

• 8 years of software engineering experience preferred.
• 3 years working with Kafka-based distributed systems.
• Experience in Kubernetes environments.
• Experience with observability and monitoring implementations.

Responsibilities

• Implement structured logging standards across Python-based services.
• Enhance Kafka consumer and producer reliability.
• Improve retry, recovery and dead-letter handling mechanisms.
• Design end-to-end event lineage and tracing capabilities.
• Develop reconciliation and completion tracking for workflows.
• Troubleshoot production issues related to event processing.
• Collaborate to improve observability and operational excellence.
• Contribute to distributed systems architecture decisions.

Skills

Python

Kafka

Concurrency & multithreading

API integrations

Testing (PyTest)

PostgreSQL

JSON/JSONB

Observability & logging

Distributed systems

Event-driven design

Tools

Kafka

PostgreSQL

Kubernetes

OpenTelemetry

Grafana

Job description

Duration :: long terms

Responsibilities

We are seeking a hands-on Senior Software Engineer to support the enhancement of an event driven digital repository platform. The role will focus on improving observability event lineage workflow reconciliation and Kafka delivery reliability across a distributed microservices ecosystem built primarily on Python and Kafka The engineer will work closely with the clients internal engineering team to implement solutions in a live production environment

Primary Skills Mandatory

• Python Development
• Strong experience developing and maintaining production grade Python applications
• Understanding of concurrency multithreading and asynchronous processing
• Experience with API integrations and distributed systems
• Familiarity with Python testing frameworks PyTest preferred
• Handson experience with Kafka in production environments
• Strong understanding of
• Offset management
• Retry mechanisms
• Consumer rebalancing and troubleshooting
• Experience with PostgreSQL performance tuning
• Experience using JSONJSONB capabilities
• Data modelling and query optimization
• Observability Logging
• Design and implementation of structured JSON logging
• Monitoring ing and troubleshooting best practices
• Understanding of distributed systems and event driven design patterns
• Experience with idempotency transactional processing and messaging reliability

Secondary Skills Preferred

• OpenTelemetry
• Distributed tracing implementation
• Instrumentation of microservices
• End-to-end transaction visibility
• Open Lineage Data Lineage Solutions
• Workflow traceability
• Data movement and process visibility
• Working knowledge of containerized workloads
• Deployments ConfigMaps Secrets scaling and troubleshooting
• GitOps Deployment Tools
• ArgoCD
• Monitoring Platforms
• Grafana
• Loki
• Metrics and dashboard creation

Key Responsibilities

• Implement structured logging standards across Python based services
• Enhance Kafka consumer and producer reliability
• Improve retry recovery and dead letter handling mechanisms
• Design and implement end-to-end event lineage and tracing capabilities
• Develop reconciliation and completion tracking mechanisms to identify incomplete workflows
• Analyse and resolve production issues related to event processing and workflow execution
• Collaborate with platform engineers to improve observability and operational excellence
• Contribute to architecture discussions and implementation decisions related to distributed systems

Experience

• 8 years of software engineering experience
• 3 years working with Kafka based distributed systems
• Experience working in Kubernetes based environments
• Experience with observability and monitoring implementations