AWS Data Engineer with PostgreSQL

E-Solutions; Basking Ridge (NJ); In loco; USD 95.000 - 150.000; Tempo pieno

Descrizione del lavoro

E-Solutions in Basking Ridge, NJ seeks a Data Reliability Engineer II to build, optimize, and manage data lakes and warehouses from multiple sources using PostgreSQL and AWS. You will monitor RDS performance, tune queries, and support data pipelines with Debezium, Kafka Connect, and Airflow, while implementing security and backup strategies.

The role emphasizes automation through Terraform/Crossplane and on-call incident response, with collaboration across teams to improve reliability and

Competenze

• Experience with PostgreSQL and AWS data pipelines (ETL/ELT).
• Monitor PostgreSQL RDS using CloudWatch/Prometheus.
• Identify and tune performance with query plans and indexes.
• Monitor Debezium and Kafka Connect data capture.
• Oversee ETL workflows and data pipelines for reliability.
• Support data-related issues and root cause analysis.
• Monitor Airflow DAGs and troubleshoot tasks.
• Develop IaC scripts (Terraform, Crossplane) for automation.
• Participate in on-call rotations for DB incidents.
• Implement cloud DB security controls and encryption best practices.
• Audit DB security configurations.
• Configure backup/recovery strategies for data integrity.

Mansioni

• Develop, optimize, and manage large data lakes and data warehouses from multiple sources.
• Proactively monitor and optimize PostgreSQL RDS performance and resources.
• Troubleshoot data capture issues with Debezium and Kafka Connect.
• Oversee ETL/ELT pipelines and airflow DAG executions.
• Create automation scripts and IaC templates for deployments.
• Participate in on-call rotations for incident response.
• Ensure database security and compliance with regulations.
• Collaborate with developers to optimize SQL queries and schemas.
• Design and optimize cloud database schemas for scalability.

Conoscenze

PostgreSQL

AWS

ETL/ELT pipelines

SQL scripting

Python

Bash

Airflow

Debezium

Kafka Connect

Security best practices

Formazione

Bachelor's degree in Computer Science or IT

Strumenti

Terraform

Crossplane

CloudWatch

Prometheus

Kafka Connect

Debezium

Apache Airflow

Apache Spark

Amazon Redshift

S3

Descrizione del lavoro

Role : AWS Data Engineer with PostgreSQL - In-person interview must

Location : Basking ridge, NJ (onsite/5 days)

Job Description

As a Data Reliability Engineer II, you will play a crucial role in developing, optimizing, and managing several large data lakes and data warehouses, comprising data from multiple disparate sources.

Responsibilities:

• Need a data engineer with extensive experience in PostgreSQL and AWS, who has previous experience creating data pipelines (ETL/ELT). He / She who have previous experience in creating data pipelines ( ETL / ELT)
• Proactively monitor PostgreSQL RDS instances for performance, availability, and resource utilization (CPU, memory, storage, connections) using established monitoring tools (e.g., CloudWatch, Prometheus).
• Assist in identifying performance bottlenecks in PostgreSQL RDS. Apply basic performance tuning techniques like reviewing query execution plans, adding missing indexes, and recommending parameter adjustments.
• Monitor the health and performance of Debezium and Kafka Connect connectors, identifying and troubleshooting basic issues related to data capture and delivery.
• Monitor ETL workflows and data pipelines for errors, performance bottlenecks, and processing delays. Troubleshoot and resolve issues to ensure reliable and timely data movement.
• Provide support for data-related issues and participate in root cause analysis.
• Monitor the execution of Apache Airflow DAGs, identify failed tasks, and troubleshoot and re-run them.
• Develop and maintain automation scripts and infrastructure as code (IAC) templates (e.g., using Crossplane, Terraform) to automate routine database tasks, deployments, and updates.
• Participate in on-call rotations to respond to database-related incidents and perform troubleshooting and root cause analysis.
• Assist in implementing and maintaining security best practices for cloud databases, including access controls, encryption, and compliance with regulatory requirements.
• Regularly audit and assess database security configurations.
• Configure and manage database backup and recovery strategies to ensure data integrity and availability in case of failures or data loss.
• Analyse database query performance and collaborate with developers to optimize SQL queries and schemas.
• Participate in continuous improvement initiatives to enhance the reliability, scalability, and performance of cloud databases.
• Assist in the design and optimization of database schemas for cloud environments.

Skills:

• Familiarity with data pipeline concepts and technologies such as Debezium, Kafka Connect, and ETL frameworks.
• Basic understanding of Amazon Redshift and S3.
• Exposure to Apache Spark for data processing.
• Basic understanding of Apache Airflow for workflow orchestration.
• Strong SQL scripting skills for querying and basic data manipulation.
• Familiarity with scripting languages (e.g., Python, Bash) is a plus.
• Knowledge of database security best practices, including access controls, encryption, and compliance with regulatory requirements (e.g., GDPR, HIPAA).
• Having an ‘AWS Certified Database - Specialty' certification is a plus.

Experience and Qualifications:

• Bachelor's degree in computer science, Information Technology, or a related field.
• 1-2 years of experience in data Engineering, with a focus on PostgreSQL.