Senior AWS Data Engineer & Pipelines Architect

Metis Technology Solutions Inc; California (MO); On-site; USD 140,000 - 200,000; Full time

Job summary

Metis Technology Solutions is seeking a Principal Data Engineer - AWS to lead the design, development, and operation of the data infrastructure supporting research platforms and the Sherlock data warehouse. The role collaborates with software engineers, researchers, analysts, and system administrators to deliver scalable ETL/ELT pipelines in AWS.

You will architect streaming pipelines, manage cloud data stores, and implement IaC and CI/CD for data applications, while ensuring security and

Qualifications

• Bachelor’s degree or higher in a technical discipline.
• 10+ years of progressive software/data/DB engineering experience.
• 5+ years AWS production data pipelines design and operation.
• Strong SQL skills and PostgreSQL/PostGIS experience preferred.
• Experience with ETL/ELT pipelines across large, diverse datasets.
• Hands-on with AWS data/compute services and security practices.
• Proven ability to design scalable, reliable data architectures.
• Experience with Airflow or Dagster and CI/CD for data apps.

Responsibilities

• Architect, design, build, deploy, and operate scalable data pipelines in AWS.
• Develop automated data ingestion and processing for various data types.
• Implement real-time and near-real-time streaming pipelines (Kinesis, Kafka, etc.).
• Maintain data stores including Redshift and PostgreSQL/PostGIS.
• Define IaC with Terraform and manage CI/CD for data apps.
• Ensure security, observability, and reliability of data platforms.
• Collaborate with researchers, analysts, and developers to translate needs.

Skills

AWS data services

SQL (PostgreSQL)

ETL/ELT pipelines

Python

Workflow orchestration

Streaming data

Terraform

NoSQL databases

Cloud architecture

Monitoring/observability

Education

Bachelor's degree in CS/CE/IS

Tools

Apache Airflow

Dagster

Kinesis

Kafka

RabbitMQ

S3

Redshift

PostgreSQL

Terraform

Job description

Metis Technology Solutions is seeking a Principal Data Engineer - AWS to lead the design, development, and operation of the data infrastructure supporting research platforms and the Sherlock data warehouse. The role collaborates with software engineers, researchers, analysts, and system administrators to deliver scalable ETL/ELT pipelines in AWS.

You will architect streaming pipelines, manage cloud data stores, and implement IaC and CI/CD for data applications, while ensuring security and