Systems & Data Infrastructure Engineer

Astrome Technologies Pvt. Ltd.; Bengaluru; On-site; INR 1,800,000 - 2,600,000; Full time

Benefits offered by this job

Career growth

ESOPs

Gratuity

PF

Health Insurance

Job summary

Astrome Technologies Pvt. Ltd. in Bengaluru invites a skilled Data Engineer to design crash-safe binary data formats, scalable data models, and efficient HPC job orchestration. You will build asynchronous backends and streaming services, enforce RBAC, and integrate AI/analytics services.

The role emphasizes Python 3, FastAPI, async ORMs, and server-sent events or WebSockets for real-time communication within a robust, secure backend.

Qualifications

• Proficient in Python 3 with asynchronous programming
• Experience with FastAPI or similar async frameworks
• Relational DB modeling using async ORM and SQL
• Experience with Server-Sent Events or WebSockets and backpressure
• RBAC and token-based authentication (JWT/OAuth)
• Subprocess management and batch job lifecycle polling
• Binary/streaming data format design for crash-safe writes
• REST/SDK integration with external AI/LLM services

Responsibilities

• Design crash-safe binary data formats for high-speed writes and concurrent reads
• Model relational schemas using async ORM across engines
• Manage lifecycle of HPC batch and interactive compute jobs
• Build async backends and real-time streaming via SSE/WebSockets with RBAC
• Integrate AI/analytics service calls from backend coordination

Skills

Async Python

Backend design

Batch job lifecycle

Subprocess management

RBAC

JWT/OAuth

Backpressure handling

Tools

FastAPI

SQLAlchemy

Tortoise ORM

SQL

SSE/WebSockets

Job description

Functional Focus: Data Engineering, Real-Time Streaming, and HPC Job Orchestration

• Scientific Data Architecture: Design crash-safe, binary data formats capable of handling high-speed incremental writes from compute jobs and concurrent reads for query-time calculations.
• Operational Data Modeling: Design and maintain a relational database schema via an async ORM, backend-agnostic across database engines.
• HPC Workload Management: Manage the lifecycle of batch and interactivecompute jobs, handling subprocess monitoring, status polling, and cluster filesystem coherency.
• Real-Time API & Streaming: Build async backend services and long-lived, server-pushed event channels with backpressure handling, and enforce role-based access control on all APIs.
• AI Service Integration: Integrate and orchestrate calls to an AI/analytics service from the backend, coordinating with application state.

Mandatory:

• Python 3, async/await
• FastAPI (or similar async Python web framework)
• Relational DB modeling, async ORM (e.g. Tortoise, SQLAlchemy), SQL
• Server-Sent Events or WebSockets, backpressure handling
• Role-based access control (RBAC), token-based auth (JWT/OAuth)
• Subprocess management, batch job lifecycle/status polling
• Binary/streaming file format design, crash-safe writes
• REST/SDK integration with an external AI/LLM service

Optional:

• Go or another async-capable backend language
• GraphQL
• PostgreSQL administration, Alembic/migrations tooling
• Message queues/brokers (Redis, RabbitMQ, Kafka)
• Air-gapped/offline deployment experience
• HPC schedulers (SLURM, PBS), Linux cluster filesystems
• NumPy/columnar formats (Parquet, HDF5)
• Tool-calling / function-calling orchestration patterns

We offer great career growth, ESOPs, Gratuity, PF and Health Insurance.