AWS Data Engineer - Real-Time Data Pipelines

TechDigital Group; Fort Worth (TX); Sur place; USD 90 000 - 120 000; Plein temps

Résumé du poste

A technology solutions provider in Fort Worth, Texas is looking for a skilled professional to lead a team in developing scalable data pipelines. The role requires expertise in AWS technologies and Python, with responsibilities including architecture design, managing customer delivery, and optimizing data workflows. If you are passionate about cloud-native solutions and data integration, this opportunity is for you.

Responsabilités

• Lead the team to complete milestones on time.
• Understand requirements, create architecture, and update stakeholders.
• Manage delivery/release to customer.

Connaissances

AWS

Python

Py‑Spark

SQL

NoSQL databases

Description du poste

Skills	Years of Experience	Last Worked in Which Year	Rating 1-10
AWS
Glue
SNS/SQS
Python
Py-Spark
Data Lake
Cloud Watch
Cloud Trail
DB Design
SQL

•Lead the team technically to complete milestones on time.
•Understand complete requirement, create Architecture and update all stakeholders.
•Create POCs
•Manage delivery/release to customer
•Develop Services to enable data ingestion from and synchronization with system which exposes required data access mechanisms ensuring near-real-time updates
•Ingest data from multiple sources using the python and any other ETL tools
•Design and implement an event-driven architecture using AWS EventBridge, Kafka, or SNS/SQS for real-time data streaming
•Design, implement, and maintain scalable data pipelines that integrate both on-prem and AWS cloud environments.
•Develop efficient Python scripts and applications using libraries like pandas, NumPy, etc., to handle and process large datasets.
•Work with various NoSQL databases (e.g., MongoDB, Cassandra, DynamoDB) to support high-performance data storage and retrieval.
•Develop and deploy applications in a cloud-native architecture, leveraging modern cloud technologies for scalability and resilience.
•Continuously monitor data workflows and systems, troubleshoot issues, and optimize performance for reliability and scalability, Transition existing pipeline to MSSQL server
•Collaborate with the business application owner on the existing data architecture, including data ingestion, data pipelines, business logic, data consumption patterns, and analytics requirements
•Design and document the target data architecture, pipelines, processing and analytics architecture
•Identify opportunities for optimization and consolidation
•Collaboration with data team on decomposition of business logic and data transformation patterns