Data Engineer: Lakehouse Automation & AWS Data Platform

TechDigital Group; Massachusetts; Hybrid; USD 100,000 - 130,000; Full time

Job summary

A leading technology firm in the United States is seeking a data engineer to automate data model creation and implement ETL pipelines. The successful candidate will design integrations across AWS Data Platform and collaborate on optimizing data architecture. Strong proficiency in Terraform and knowledge of AWS services is essential. This role offers a dynamic work environment focused on cutting-edge data solutions within a collaborative team.

Qualifications

• Experience with automating data integration processes.
• Proficiency in Terraform for managing AWS resources.
• Familiarity with AWS Data Services like RDS.

Responsibilities

• Automate creation and transformation of data models.
• Implement automated ETL pipelines.
• Design IAM solutions including EKS Service Accounts.
• Collaborate on data architecture and analytics requirements.

Skills

Automated ETL pipelines

Terraform

AWS Data Services

HashiCorp Vault

Data architecture

Job description

Job Description:
Automate creation, transformation, and integration of raw and enriched data models in the data Lakehouse.
Implement automated ETL pipelines for data ingestion, transformation, and consumption
Design and implementation for integrating Lakeformation across AWS Data Platform Accounts.
Design and implementation for integrating Lakeformation with Keycloak.
Design and implementation for an integrated IAM solution including EKS Service Accounts.
Design and implementation of HashiCorp Vault with integration to AWS Services such as Secrets Manager, Certificate Manager and Keycloak.
Development of integration of HashiCorp Vault with AWS Data Services such as RDS Aurora PostgreSQL, RDS MSSQL and other RDS solutions as required.
Provision MSSQL server on AWS using Terraform
Deploy read replica for MSSQL server using Terraform
Implement monitoring & logging for MSSQL server using Terraform
Transition existing pipeline to MSSQL server
Collaborate with the business application owner on the existing data architecture, including data ingestion, data pipelines, business logic, data consumption patterns, and analytics requirements
Design and document the target data architecture, pipelines, processing and analytics architecture
Identify opportunities for optimization and consolidation
Collaboration with data team on decomposition of business logic and data transformation patterns