Remote Senior Backend Data Engineer: Metadata & Pipelines

Atlas Data Science; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health benefits

401(k) matching

Unlimited PTO

Remote-friendly

Job summary

Atlas Data Science is seeking a Software Engineer to design and deliver a central metadata hub for scalable data management. You will build production backend services in Python, integrate REST APIs, and work across relational databases, microservices, and cloud resources.

You will implement infrastructure as code with Terraform, collaborate with DevOps on CI/CD, and contribute to data pipelines for AI/ML workloads. A strong emphasis on reliability and scalable design pervades the role.

Qualifications

• Production backend services experience with Python.
• REST APIs and backend service development.
• Knowledge of containerized environments (Docker).
• Relational databases and data modeling (PostgreSQL/Oracle/MySQL).
• Secure API design and secret handling.
• Troubleshooting distributed services via logs/metrics.
• Strong communication and teamwork in Agile settings.
• Ability to decompose features and drive to completion.
• Experience with Airflow or similar orchestration tools.
• Git-based workflows and CI/CD (GitLab preferred).
• Experience with OpenSearch or search engines is a plus.

Responsibilities

• Design, build, and maintain data service pipelines for metadata.
• Integrate structured data from relational databases into pipelines.
• Optimize pipelines for performance and scale.
• Develop backend services using Java, Node.js, and Python.
• Implement IaC with Terraform for platform resources.
• Collaborate with DevOps to integrate CI/CD workflows.
• Write unit/integration/system tests to ensure quality.
• Support AI/ML initiatives with metadata pipelines.
• Keep up with evolving data lake/lakehouse tech and data brokerage trends.

Skills

Backend services development

Python

REST APIs

Docker

SQL databases

PostgreSQL

Oracle

MySQL

Authentication & secrets

Observability & metrics

Agile

Distributed systems

Asynchronous workflows

Airflow

CI/CD (GitLab)

Kubernetes

OpenSearch

Git versioning

Tools

Docker

PostgreSQL

Oracle

MySQL

OpenSearch

Airflow

Kubernetes

GitLab

Terraform

Job description

Atlas Data Science is seeking a Software Engineer to design and deliver a central metadata hub for scalable data management. You will build production backend services in Python, integrate REST APIs, and work across relational databases, microservices, and cloud resources.

You will implement infrastructure as code with Terraform, collaborate with DevOps on CI/CD, and contribute to data pipelines for AI/ML workloads. A strong emphasis on reliability and scalable design pervades the role.