Systems & Data Infrastructure Engineer

Astrome; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Benefits offered by this job

Health Insurance

ESOPs

Gratuity

PF

Job summary

Astrome Bengaluru is seeking a data engineering specialist to design crash-safe binary data formats and real-time streaming services for HPC workloads.

You\'ll build async backends with FastAPI, implement RBAC, token-based auth, and integrate with external AI/LLM services while coordinating with compute jobs.

This role emphasizes scalability, streaming durability, and cross-platform orchestration in a Linux cluster environment.

Qualifications

• Experience with Python 3, async/await and FastAPI.
• Proficient in relational DB modeling and async ORM (SQLAlchemy, Tortoise).
• Experience with Server-Sent Events or WebSockets and RBAC/JWT/OAuth.
• Familiarity with subprocess management and batch job lifecycles.
• Design crash-safe binary/streaming data formats for high-speed writes.
• REST/SDK integration with external AI/LLM services.

Responsibilities

• Design crash-safe binary data formats and fast reads for query-time calculations.
• Model relational data via async ORM across engines.
• Manage lifecycle of batch and interactive compute jobs and ensure cluster filesystem coherency.
• Build async backend services and streaming with server-pushed events and RBAC.
• Coordinate with AI/analytics services for integration.

Skills

Python

FastAPI

SQL

Async ORM

RBAC

JWT/OAuth

Subprocess management

Streaming data

REST APIs

AI/LLM integration

Tools

SQLAlchemy

Tortoise ORM

PostgreSQL

Redis

RabbitMQ

Kafka

Parquet

Job description

Functional Focus: Data Engineering, Real-Time Streaming, and HPC Job Orchestration

• HPC schedulers (SLURM, PBS), Linux cluster filesystems

• Scientific Data Architecture: Design crash-safe, binary data formats capable of handling high-speed incremental writes from compute jobs and concurrent reads for query-time calculations
• Operational Data Modeling: Design and maintain a relational database schema via an async ORM, backend-agnostic across database engines
• HPC Workload Management: Manage the lifecycle of batch and interactivecompute jobs, handling subprocess monitoring, status polling, and cluster filesystem coherency
• Real-Time API & Streaming: Build async backend services and long-lived, server-pushed event channels with backpressure handling, and enforce role-based access control on all APIs
• AI Service Integration: Integrate and orchestrate calls to an AI/analytics service from the backend, coordinating with application state

Requirements

Mandatory:

• Python 3, async/await
• FastAPI (or similar async Python web framework)
• Relational DB modeling, async ORM (e.g. Tortoise, SQLAlchemy), SQL
• Server-Sent Events or WebSockets, backpressure handling
• Role-based access control (RBAC), token-based auth (JWT/OAuth)
• Subprocess management, batch job lifecycle/status polling
• Binary/streaming file format design, crash-safe writes
• REST/SDK integration with an external AI/LLM service

Optional:

• Go or another async-capable backend language
• GraphQL
• PostgreSQL administration, Alembic/migrations tooling
• Message queues/brokers (Redis, RabbitMQ, Kafka)
• Air-gapped/offline deployment experience
• HPC schedulers (SLURM, PBS), Linux cluster filesystems
• NumPy/columnar formats (Parquet, HDF5)
• Tool-calling / function-calling orchestration patterns

Benefits

• We offer great career growth, ESOPs, Gratuity, PF and Health Insurance.