Principal Data Engineer - AWS Pipelines Architect

Metis Technology Solutions, Inc.

Mountain View (CA)

On-site

USD 150,000 - 200,000

Full time

3 days ago

Be an early applicant

Application generator

Get a reply from this employer — a resume and cover letter tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Metis Technology Solutions, Inc. is seeking an experienced Principal Data Engineer – AWS to join our Software Operations team.

This role collaborates with researchers, analysts, developers, and DBAs to design, build, deploy, and operate data infrastructure and pipelines that connect research platforms, external data sources, cloud services, and the Sherlock data warehouse. You will architect scalable ETL/ELT pipelines in AWS, implement real-time streaming, manage cloud data warehouses, and

Qualifications

10+ years of progressively responsible software engineering, data engineering, database engineering, or closely related technical experience.
5+ years of substantial hands-on AWS experience, including the design, implementation, deployment, and operation of production data-processing or data-pipeline solutions.
Demonstrated experience architecting and developing ETL/ELT pipelines involving large, heterogeneous, or rapidly changing datasets.
Strong hands-on experience with AWS data and compute services (S3, Lambda, Redshift, RDS, DynamoDB, Kinesis, Firehose, EC2, API Gateway, IAM, CloudWatch).

Responsibilities

Architects, designs, develops, deploys, and maintains scalable and reliable ETL/ELT data pipelines in Amazon Web Services (AWS).
Designs automated ingestion and processing solutions for structured, semi-structured, unstructured, batch, and streaming data.
Develops production-quality data-processing and pipeline software using Python, SQL, and shell scripting.
Designs and maintain workflow orchestration solutions using technologies such as Apache Airflow, Dagster, AWS Step Functions, or equivalent platforms.
Designs and implements real-time and near-real-time streaming and event-driven data pipelines using technologies such as AWS Kinesis, Amazon Data Firehose, Kafka, RabbitMQ, SQS/SNS, or equivalent technologies.
Develops and maintains cloud data solutions using AWS services such as Amazon S3, AWS Lambda, Amazon Redshift, Amazon RDS, Amazon DynamoDB, Amazon EC2, API Gateway, IAM, and CloudWatch, as appropriate to project requirements.
Administers and optimizes relational databases and cloud data warehouses, including PostgreSQL/PostGIS and Amazon Redshift or comparable technologies.
Develops and maintains infrastructure as code (IaC) using Terraform or equivalent technologies.
Develops and maintains automated deployment and CI/CD processes for data applications and supporting infrastructure.
Designs pipelines and supporting infrastructure for reliability, scalability, maintainability, observability, security, and efficient use of AWS resources.
Implements appropriate AWS security practices, including IAM roles and policies, encryption, secrets management, network controls, logging, and least-privilege access.
Develops monitoring, logging, metrics, and alerting that provide operational visibility into data-pipeline health and performance.
Documents data architectures, data flows, interfaces, infrastructure, deployment processes, operational procedures, and troubleshooting practices.
Collaborates with researchers, analysts, and application developers to translate research and application requirements into reliable and maintainable data-processing solutions.