Data Engineer

TechDigital Group; Princeton (NJ); On-site; USD 90,000 - 120,000; Full time

Job summary

A leading tech company located in Princeton, NJ, is looking for a skilled Data Engineer. The candidate will design and maintain data pipelines, integrate data from various sources, and leverage AWS services to develop cloud-native solutions. Strong experience with Python and AWS is essential. Responsibilities include monitoring data systems for quality and performance while collaborating with data scientists and business teams to meet their needs. This role is critical to ensuring data quality and availability across platforms.

Responsibilities

• Design, build, and maintain scalable data pipelines.
• Integrate structured and unstructured data from various sources.
• Ensure data quality, consistency, and availability.
• Leverage AWS services to build cloud-native data solutions.
• Optimize cloud resources for performance and cost.
• Use Python for data manipulation and automation.
• Collaborate with data scientists and analysts.
• Monitor data pipelines and troubleshoot issues.

Skills

Python

SQL

AWS

Apache Spark

Pandas

PySpark

Airflow

PostgreSQL

MySQL

Git

Terraform

CloudFormation

Job description

Key Skills & Technologies

• Programming Languages: Python (primary), SQL
• Cloud Platforms: AWS (S3, Glue, Lambda, Redshift, EC2, EMR)
• Data Tools: Apache Spark, Pandas, PySpark, Airflow
• Databases: PostgreSQL, MySQL, NoSQL (e.g., DynamoDB)
• ETL & Workflow Orchestration: AWS Glue, Apache Airflow
• Version Control: Git
• DevOps & CI/CD: Basic understanding of CI/CD pipelines and infrastructure as code (e.g., Terraform, CloudFormation)

Job Summary

• Data Pipeline Development - Design, build, and maintain scalable and reliable data pipelines to ingest, process, and transform data from various sources.
• Data Integration & Management - Integrate structured and unstructured data from internal and external systems.
• Ensure data quality, consistency, and availability across platforms.
• Cloud-Based Data Engineering - Leverage AWS services (e.g., S3, Lambda, Glue, Redshift, EMR) to build cloud-native data solutions.
• Optimize cloud resources for performance and cost-efficiency.
• Programming & Automation - Use Python for data manipulation, ETL workflows, and automation of data tasks.
• Develop reusable scripts and modules for data processing.
• Collaboration & Stakeholder Engagement - Work closely with data scientists, analysts, and business teams to understand data needs.
• Translate business requirements into technical solutions.
• Monitoring & Optimization - Monitor data pipelines and troubleshoot issues proactively.
• Continuously improve performance, scalability, and reliability of data systems.