Backend Data Engineer: Metadata & AI Pipelines (Remote)

Atlas Data Science

United States

Remote

USD 140,000 - 190,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health coverage
Remote work
401k matching
Unlimited PTO

Job summary

Atlas Data Science is building a central metadata hub to streamline discovery, access, and sharing of data across an expanding ecosystem of suppliers and consumers. You will help design and deliver data management and governance solutions at scale.

As an L2/L3 Software Engineer, you’ll work with a modern stack (TypeScript, Python, Java, Postgres, OpenSearch, Kubernetes, Airflow, Terraform, GitLab) to ship reliable, scalable backend services and metadata pipelines.

Qualifications

  • Proficiency developing production backend services using Python.
  • Experience developing and integrating REST APIs and backend services.
  • Understanding of Docker and deploying services in containerized environments.
  • Experience with SQL, relational databases, and data modeling using platforms such as PostgreSQL, Oracle, or MySQL.
  • Knowledge of authentication, authorization, secure API design, and secrets handling.
  • Experience troubleshooting distributed services using logs and operational metrics.
  • Strong communication, problem-solving, and collaborative development skills.
  • Ability to work collaboratively in a team-oriented Agile environment.
  • Experience diagnosing and improving backend performance, reliability, and scalability.
  • Experience designing reliable asynchronous workflows, including retries, idempotency, timeouts, and failure handling.
  • Experience with Airflow, Step Functions, Argo, or another orchestration platform.
  • Experience with Git-based source control, code review, and CI/CD workflows; GitLab experience is preferred.
  • Ability to decompose complex features into smaller deliverables and drive them from design through completion.
  • Clear written and verbal communication skills.

Responsibilities

  • Design, develop, and maintain data service metadata capture, transformation, and delivery pipelines.
  • Integrate structured data from relational databases (Oracle, PostgreSQL, MySQL, etc.) into pipelines.
  • Optimize and troubleshoot pipelines for performance, reliability, and scalability.
  • Contribute to backend service development using Java, Node.js, and Python.
  • Implement infrastructure as code using Terraform to provision and manage platform resources.
  • Collaborate with DevOps to integrate pipelines into CI/CD workflows using GitLab.
  • Write unit, integration, and system tests to ensure code quality and functionality.
  • Support AI/ML initiatives by building and optimizing metadata pipelines for LLM training and data processing.
  • Stay current with emerging technologies, development best practices, modern data lake/lakehouse frameworks, and evolving approaches in data brokerage services.],
  • CoT_job_summary_short":"COMPANY NAME: Atlas Data Science\nKEY POINTS: Metadata hub, scalable data governance, backend services","job_summary_short":"<p>Atlas Data Science is building a central metadata hub to streamline discovery, access, and sharing of data across an expanding ecosystem of suppliers and consumers. You will help design and deliver data management and governance solutions at scale.</p><p>As an L2/L3 Software Engineer, you’ll work with a modern stack (TypeScript, Python, Java, Postgres, OpenSearch, Kubernetes, Airflow, Terraform, GitLab) to ship reliable, scalable backend services and metadata pipelines.</p>"

Skills

Python backend
REST APIs
Java
Node.js
SQL design
Distributed systems
Agile
Team collaboration
Problem solving
Communication

Tools

Docker
PostgreSQL
OpenSearch
Kubernetes
Airflow
Terraform
GitLab
CI/CD

Job description

Atlas Data Science is building a central metadata hub to streamline discovery, access, and sharing of data across an expanding ecosystem of suppliers and consumers. You will help design and deliver data management and governance solutions at scale.

As an L2/L3 Software Engineer, you’ll work with a modern stack (TypeScript, Python, Java, Postgres, OpenSearch, Kubernetes, Airflow, Terraform, GitLab) to ship reliable, scalable backend services and metadata pipelines.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Mid-Level Backend Software Engineer (L2/L3)
Mid-Level Backend Software Engineer (L2/L3)

Atlas Data Science • United States

Remote
USD 140,000 - 190,000
Health coverage
Remote work
401k matching
+1
Staff Backend Engineer — AI Metadata Platform (Remote)
Staff Backend Engineer — AI Metadata Platform (Remote)

DataHub • Palo Alto (CA)

Hybrid
USD 225,000 - 300,000
Flexible savings accounts
Unlimited PTO and sick leave
Monthly coworking stipend
+1
Senior Software Engineer - Metadata
Senior Software Engineer - Metadata

Fivetran, Inc. • United States

Remote
USD 120,000 - 180,000
Health coverage
Stock grants
Phone stipend
+2
Data Platform Engineer - Databricks & Pipelines
Data Platform Engineer - Databricks & Pipelines

Atlas Air • Miami (FL)

On-site
USD 74,000 - 100,000
Medical, dental and vision insurance
Employee assistance program
Generous paid time off
+1
Senior Data Engineer - Remote, Scalable Pipelines
Senior Data Engineer - Remote, Scalable Pipelines

Astrodata • United States

Remote
USD 140,000 - 190,000
Remote work with annual team offsite
US health coverage
401k with employer matching
Data Engineer
Data Engineer

Virtusa Corporation • United States

Remote
USD 120,000 - 160,000
Remote Data Engineer: Build Secure, Scalable Pipelines
Remote Data Engineer: Build Secure, Scalable Pipelines

Atgardas • Snowflake (AZ)

Remote
USD 110,000 - 150,000
Remote work flexibility
Professional development budget
Generous paid time off
+3
Senior Analytics Engineer - Remote Data Pipelines
Senior Analytics Engineer - Remote Data Pipelines

Astrodata • United States

Remote
USD 120,000 - 190,000
Remote work with annual team offsite
Great health coverage (US)
401k with employer matching
Senior Data Engineer: Data Mesh & AI Pipelines (Remote)
Senior Data Engineer: Data Mesh & AI Pipelines (Remote)

Safeguardglobal • United States

Remote
USD 140,000 - 210,000
Remote-first
Global teams
generous leave
+4
Data Engineer - Databricks Pipelines & Data Platforms
Data Engineer - Databricks Pipelines & Data Platforms

Atlas Air • Erlanger (KY)

On-site
USD 74,000 - 100,000