AWS Data Engineer

Compunnel, Inc.; Pennsylvania; On-site; USD 120,000 - 150,000; Full time

Job summary

Compunnel, Inc. is seeking a data engineering lead to design and implement scalable ETL pipelines and data products for critical trading applications. You will work with autonomy on end-to-end data solutions, from ingestion to API delivery, and archive data to data lakes for analytics.

The role demands expertise in Python, SQL, cloud services, and resilient API architectures, plus experience with monitoring stacks and modern data tooling. U.S.

Qualifications

• Expert-level Python and SQL programming required.
• Experience with AWS services and data lake/warehouse tooling.
• Strong API design and microservices experience.
• Proven ability to build scalable, resilient data pipelines.

Responsibilities

• Implement diverse data solutions with autonomy, building and maintaining ETL pipelines.
• Apply data quality, validation, anomaly detection, and AI/LLM approaches to data analysis.

Skills

Python

SQL

AWS Cloud Services

API design principles

Distributed systems

Data engineering

Tools

Docker

Kubernetes

Helm charts

Swagger/OpenAPI

Postman

GraphQL

RESTful API design

Job description

Responsibilities

• Data Pipeline Expertise: Capable of implementing a diverse range of data solutions with significant autonomy, demonstrating the ability to independently build and maintain complex ETL (Extract, Transform, Load) pipelines. Possesses advanced skills in pipeline complexity management and orchestration, with a proven track record of developing high-performance data products accessible via APIs for critical tier 1 trading applications. Demonstrates expertise in archiving data products to data lakes for comprehensive analytics purposes.
• Data Quality and Advanced Analytics: Strong proficiency in Data Quality, Data Validation, and Anomaly Detection techniques. Experience with AI/ML and Large Language Model (LLM) approaches to data analysis and validation. Ability to implement sophisticated data integrity and verification processes.

Required Qualifications

• Programming and Cloud Technologies: Expert-level skills in Python, SQL, AWS Cloud Services (Cloud Storage, Lambda, ECS, Glue, Data Quality, HDMI, Redshift, Machine Learning, OpenSearch, RDS).
• API and Resilience Technologies: Advanced skills in building resilient API architectures, including microservices design patterns, API gateway configuration, circuit breaker implementations, fault tolerance and error handling mechanisms, and expertise in API development and management tools (Swagger/OpenAPI, Postman, GraphQL, RESTful API design principles).
• Resilience Engineering Techniques: Distributed system design, load balancing, failover strategies, horizontal and vertical scaling approaches, containerization and orchestration (Docker, Kubernetes, Helm charts).
• Monitoring and Observability: Prometheus, Grafana, ELK Stack (Elasticsearch, Logstash, Kibana), distributed tracing with Jaeger or Zipkin.
• Additional Technical Competencies: Proficient in Apache Flink, advanced Java programming skills, proven ability to create and optimize complex data products, and demonstrated subject‑matter expertise in data engineering principles.

Preferred Qualifications

• Professional Attributes: Capability to work independently through complex data challenges, with a strong preference for candidates with prior experience in the investment data domain and related technologies.

Key Differentiators

• Deep understanding of data engineering best practices.
• Ability to design scalable, efficient, and resilient data solutions.
• Proven track record of solving intricate data infrastructure problems with robust API architectures.