We are seeking a highly skilled Senior Data Engineer to design, build, and optimize scalable, highperformance data pipelines and platforms on AWS. This role will be instrumental in developing a modern data Lakehouse architecture using AWS EMR, Python, PySpark, and Apache Iceberg, while orchestrating serverless and containerized workflows through Step Functions, Lambda, and ECS. The ideal candidate will integrate diverse data sources across PostgreSQL and DynamoDB, ensuring data is reliable, well-governed, and readily available to power advanced analytics and data-driven decisionmaking across the organization.
Job Description / Duties and Responsibilities
- Design, develop, and maintain analytical data models and metrics using modern, scalable data platform technologies.
- Architect and implement data Lakehouse solutions on AWS using AWS EMR and Apache Iceberg.
- Build, optimize, and maintain robust batch and incremental data pipelines using Python and PySpark.
- Orchestrate and automate data workflows using AWS Step Functions, Lambda, and ECS.
- Design and implement event-driven data processing solutions using AWS Lambda, Step Functions to enable near real-time data ingestion and transformation.
- Build event-triggered pipelines that respond to data arrivals, system events, and business transactions (e.g., S3 events, DynamoDB Streams).
- Design and manage data storage and access patterns across PostgreSQL and DynamoDB.
- Ensure high performance, reliability, and scalability of data systems.
- Collaborate closely with business stakeholders to translate requirements into data solutions.
- Partner with architecture and operations teams to enforce best practices, standards, and governance.
- Adhere to the Information Security Management policies and procedures.
Job Specification / Skills and Competencies
- Graduate degree in Computer Science, Engineering, or a related field.
- 7+ years of overall professional experience in software development, with a focus on data and analytical platform development.
- 4+ years of relevant hands-on experience in data engineering and big data ecosystems.
- Strong expertise in the following technologies:
- AWS EMR
- PySpark
- DynamoDB
- Python
- Hands-on experience with AWS Step Functions, Lambda, and ECS for workflow orchestration and serverless/containerized processing.
- Hands-on experience designing and implementing event-driven architectures on AWS, including event triggers, asynchronous processing, and decoupled workflows.
- Experience working with DynamoDB.
- Proven experience designing and implementing data Lakehouse architectures on AWS.
- Experience building highly scalable, low-latency data systems.
Preferred Qualifications
- Experience with Amazon Redshift for data warehousing.
- Experience with AWS Glue for ETL and data cataloging.
- Strong ability to learn and adapt to new technologies quickly.
- Excellent problem-solving and analytical thinking skills.
- Ability to work independently and collaboratively in a team environment.
- Strong multitasking skills in a fast-paced, dynamic setting.
- Excellent verbal and written communication skills