Data Engineer

Compunnel, Inc.; Boston (MA); On-site; USD 110,000 - 140,000; Full time

Job summary

A tech solutions provider is looking for an experienced AWS Data Engineer to design and maintain scalable data pipelines on the AWS cloud platform. This role involves developing ETL/ELT processes, ensuring data quality and security, and collaborating with stakeholders. The ideal candidate has over 6 years of experience and strong skills in AWS services, SQL, Scala, and Python. This position focuses on delivering optimized data solutions for analytics and business intelligence.

Qualifications

• Minimum 6+ years of experience in designing and developing enterprise-wide big data solutions.
• Strong SQL experience, preferably with Redshift.
• Hands-on experience with ETL/ELT processes and frameworks.

Responsibilities

• Design and build data pipelines and ETL/ELT processes using AWS services.
• Integrate data from various sources ensuring quality and compliance.
• Monitor and optimize data processing jobs and databases.

Skills

AWS services

Scala

Python

SQL

ETL/ELT processes

Big Data technologies

Tools

AWS Glue

Redshift

Hadoop

Spark

Job description

We are seeking an experienced AWS Data Engineer to design, build, and maintain scalable data pipelines and architectures on the AWS cloud platform.

This role involves developing ETL/ELT processes, integrating data from multiple sources, ensuring data quality and security, and collaborating with stakeholders to deliver optimized data solutions for analytics and business intelligence.

Key Responsibilities

• Design and build data pipelines and ETL/ELT processes using AWS services such as Glue, EMR, and Redshift.
• Develop and maintain efficient data models for storage and analysis.
• Integrate data from various sources and ensure data quality, security, and compliance.
• Implement validation checks and security best practices across data workflows.
• Monitor and optimize data processing jobs and databases for performance.
• Collaborate with stakeholders to understand data requirements and deliver solutions.
• Maintain and operationalize existing data solutions and support modernization initiatives.

Required Qualifications

• Minimum 6+ years of experience in designing and developing enterprise-wide big data solutions.
• Strong experience with AWS services: Glue, Lambda, S3, EMR, SNS/SQS, CloudWatch, Redshift.
• Proficiency in Scala and Python for application development and automation.
• Strong SQL experience, preferably with Redshift; experience with relational databases (Oracle, MySQL, PostgreSQL).
• Hands‑on experience with ETL/ELT processes and frameworks.
• Experience with Big Data technologies such as Hadoop and Spark.
• Familiarity with shell scripting in Linux/Unix environments.

Preferred Qualifications

• Experience with Snowflake.
• Knowledge of CI/CD pipelines and automation tools.
• Exposure to machine learning models, regression, and validation techniques.