Senior AWS Data Engineer - ETL & Streaming Pipelines

Metis Technology Solutions Inc; California; On-site; USD 140,000 - 200,000; Full time

Job summary

Metis Technology Solutions is seeking a Principal Data Engineer – AWS at Moffett Field to design, build, deploy, and operate scalable data pipelines in AWS. You will work with researchers, analysts, and developers to deliver robust data infrastructure and a Sherlock data warehouse integration.

The role requires extensive AWS data services experience, ETL/ELT expertise, and strong SQL skills, including PostgreSQL/PostGIS.

Qualifications

• Bachelor's degree or higher in a related technical field.
• 10+ years of progressive software/data engineering experience.
• 5+ years hands-on AWS experience designing and operating data pipelines.
• Experience with ETL/ELT pipelines at scale.
• Strong SQL skills; PostgreSQL/PostGIS preferred.
• Experience with Airflow, Dagster, or Step Functions.
• Experience with Terraform or IaC and security best practices.

Responsibilities

• Architect, design, deploy and maintain scalable ETL/ELT pipelines in AWS.
• Design automated ingestion and processing for varied data types.
• Develop production data-processing software with Python/SQL/shell.
• Build and maintain workflow orchestration using Airflow, Dagster, or Step Functions.
• Create real-time streaming pipelines with Kinesis, Kafka, or similar.
• Maintain cloud data solutions using S3, Lambda, Redshift, RDS, DynamoDB.
• Administer PostgreSQL/PostGIS and Redshift databases.
• Develop IaC with Terraform and automate deployment pipelines.
• Ensure reliability, observability, security and efficient AWS usage.
• Implement AWS security practices and least-privilege access.
• Develop monitoring, logging, metrics, and alerts for data pipelines.
• Document architectures, data flows, and deployment procedures.
• Collaborate with researchers, analysts, and developers to translate requirements.

Skills

AWS

SQL

ETL/ELT

Data pipelines

PostgreSQL

Terraform

CI/CD

Data warehousing

Airflow

Dagster

Kinesis

Redshift

Cloud security

Education

Bachelor's degree in CS/CE/IS or related

Tools

Airflow

Dagster

Terraform

Kinesis

Redshift

PostgreSQL

Job description

Metis Technology Solutions is seeking a Principal Data Engineer – AWS at Moffett Field to design, build, deploy, and operate scalable data pipelines in AWS. You will work with researchers, analysts, and developers to deliver robust data infrastructure and a Sherlock data warehouse integration.

The role requires extensive AWS data services experience, ETL/ELT expertise, and strong SQL skills, including PostgreSQL/PostGIS.