Data Engineer

Compunnel, Inc.; New York (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Compunnel, Inc. is seeking a software engineer to design, develop, and maintain data pipelines for efficient ingestion, transformation, and processing. You will implement Node.js Express backend APIs with unit tests and utilize AWS Lambda for serverless tasks.

The role involves deploying containerized apps with AWS EKS, provisioning AWS resources (EC2, S3, API Gateway, SQS, SES, Pinpoint), and collaborating with DevOps and Data Science teams.

Qualifications

• Proficiency in Python programming.
• Experience with AWS cloud services.
• Knowledge of PySpark for big data processing.
• Strong SQL querying and data manipulation.
• Familiarity with healthcare data domains.

Responsibilities

• Design and deploy containerized applications using AWS EKS for scalability.
• Develop data pipelines and backend APIs, with unit tests to ensure reliability.
• Collaborate with DevOps and Data Science to understand data needs and deliver solutions.

Skills

Python

AWS

Pyspark

SQL

Healthcare domain

Job description

JOB SUMMARY

Design develop and maintain data pipelines for efficient data ingestion, transformation, and processing. Develop Node.js Express backend APIs with comprehensive unit testing to ensure functionality and reliability. Utilize AWS Lambda serverless functions to handle event-driven tasks and optimize resource utilization. Implement and manage NoSQL databases including DynamoDB, MongoDB, and Elasticsearch for efficient data storage and retrieval.

Key Responsibilities

• Design and deploy containerized applications using AWS EKS for scalability and manageability.
• Provision and manage AWS resources including EC2 instances, S3 buckets, API Gateway endpoints, SQS queues, SES email services, and Pinpoint for user engagement.
• Collaborate with cross-functional teams (DevOps, Data Science, etc.) to understand data needs and deliver effective solutions.

Required Qualifications

• Python
• AWS
• Pyspark
• SQL
• Healthcare domain

Preferred Qualifications

None mentioned

Certifications

None mentioned