Senior Data Engineer (Python, AWS)

EPAM Systems; Polska; Sur place; PLN 180 000 - 240 000; Plein temps

Résumé du poste

EPAM Systems Inc is seeking a Senior Data Engineer in Poland to join an established ETF analytics platform. You will design, build, and optimize Python-based data pipelines handling real-time and historical market data, leveraging AWS S3, ECS, EC2, Lambda, and workflow services.

You will work in a small agile team, participate in client coordination, and produce comprehensive documentation while providing L3 production support and exploring new technologies to improve performance and reliability.

Qualifications

• 3+ years of experience in Data Software Engineering.
• Proficient in Python development for data engineering environments.
• Experience designing data lakehouse architectures and real-time streaming pipelines.
• Experience integrating AWS services including S3, ECS, EC2.

Responsabilités

• Design, develop, and optimize Python-based data pipelines to process streaming market data with minimal latency.
• Integrate AWS cloud services to support ingestion, processing, storage, and delivery of real-time and historical data.
• Collaborate within a small agile team of 3 to 5 engineers, enabling rapid decision-making.
• Participate in coordination meetings with client managers and internal teams to align deliverables.
• Produce and maintain technical documentation for system understanding and knowledge transfer.
• Provide L3 production support, root cause analysis, and corrective actions.
• Evaluate and adopt emerging technologies to enhance pipeline performance and reliability.

Connaissances

Python

Data pipelines

AWS

Lakehouse

Streaming data

S3

ECS

EC2

Lambda

Step Functions

EventBridge

CloudWatch

DynamoDB

NoSQL

Production support

Finance domain

Description du poste

We are seeking a Senior Data Engineer to join an established secondary market ETF analytics platform with over 7 years of production history. The team ingests daily trading reports from multiple sources, including file providers, market data providers, spread data, and bookings, to produce analytics outputs. Built entirely on Python and AWS with a Data Lakehouse and data pipeline architecture, the project is transitioning toward a "run-org" model, requiring someone who can confidently pick up and evolve an existing codebase.

Responsibilities

• Design, develop, and optimize Python-based data pipelines to process streaming market data with minimal latency
• Integrate AWS cloud services to support ingestion, processing, storage, and delivery of real-time and historical data
• Collaborate within a small agile team of 3 to 5 engineers, ensuring rapid decision-making and iterative development
• Participate in regular coordination meetings with client managers and internal development teams to align deliverables
• Produce and maintain comprehensive technical documentation to support system understanding and knowledge transfer
• Provide L3 support for production incidents, performing root cause analysis and implementing corrective actions
• Evaluate and adopt emerging technologies and methodologies to enhance pipeline performance and reliability

Requirements

• 3+ years of experience in Data Software Engineering
• Proficiency in Python development within data engineering environments, emphasizing scalable and maintainable code
• Expertise in designing and implementing data lakehouse architectures and real-time streaming data pipelines
• Skills in integrating and managing AWS services, including S3, ECS, and EC2
• Competency in working with Lambda, Step Functions, and EventBridge
• Familiarity with CloudWatch and DynamoDB for monitoring and NoSQL data management
• Capability to provide Level 3 production support for critical data engineering systems with strong troubleshooting abilities
• Understanding of financial or banking domain concepts, enabling effective collaboration with business stakeholders
• English proficiency at an Upper-Intermediate level (B2) or higher