Staff Software Engineer - Infinia Data Engine

DataDirect Networks; United States; Remote; USD 120,000 - 150,000; Full time

Job summary

A leading data management company is seeking a Staff Software Engineer for the Infinia Data Engine team. You will design and optimize data execution engines, collaborate with cross-functional teams, and contribute to open-source projects. The ideal candidate has at least 8 years of software development experience, with a strong background in SQL, Python, and distributed systems. This position offers the opportunity to shape innovative data workflows in a remote work environment.

Benefits

Remote work flexibility

Collaborative work environment

Qualifications

• 8+ years of experience in software development, with 5+ years focused on distributed systems.
• Expert-level knowledge of SQL, Python, and Java or Scala.
• Experience with distributed query engines or databases.

Responsibilities

• Design autonomous logic for optimizing SQL and non-SQL queries.
• Build and tune execution plans for large-scale AI workloads.
• Contribute to relevant open-source ecosystems.

Skills

SQL

Python

Java

Distributed systems

Performance optimization

Education

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Tools

Apache Spark

Kafka

HDFS

Hive Metastore

Job description

Staff Software Engineer – Infinia Data Engine

Job ID: 2025-5422

Job Locations: US-Remote

Overview

This is an incredible opportunity to be part of a company that has been at the forefront of AI and high-performance data storage innovation for over two decades. DataDirect Networks (DDN) is a global market leader renowned for powering many of the world's most demanding AI data centers, in industries ranging from life sciences and healthcare to financial services, autonomous cars, Government, academia, research and manufacturing.

‘DDN's A3I solutions are transforming the landscape of AI infrastructure.’ – IDC

‘The real differentiator is DDN. I never hesitate to recommend DDN. DDN is the de facto name for AI Storage in high performance environments’ – Marc Hamilton, VP, Solutions Architecture & Engineering | NVIDIA

DDN is the global leader in AI and multi‑cloud data management at scale. Our cutting‑edge data intelligence platform is designed to accelerate AI workloads, enabling organizations to extract maximum value from their data. With a proven track record of performance, reliability, and scalability, DDN empowers businesses to tackle the most challenging AI and data‑intensive workloads with confidence.

Our success is driven by our unwavering commitment to innovation, customer‑centricity, and a team of passionate professionals who bring their expertise and dedication to every project. This is a chance to make a significant impact at a company that is shaping the future of AI and data management.

Our commitment to innovation, customer success, and market leadership makes this an exciting and rewarding role for a driven professional looking to make a lasting impact in the world of AI and data storage.

Job Description

We are seeking a Staff Software Engineer to join the Infinia Data Engine team – the group responsible for powering high-performance, AI-native data workflows on DDN's next-generation distributed data platform.

In this role, you will lead the design and optimization of data execution engines, data format handling, and query-layer integration with industry-standard open-source frameworks including Apache Iceberg, Delta Lake, Apache Spark, Trino, and others. You'll play a key role in bridging proprietary high-performance systems with open ecosystems - enabling large-scale, real-time data access, transformation, and analytics.

This is a hands‑on, high‑impact position ideal for an engineer who thrives at the intersection of distributed systems, open-source data technologies, and performance optimization.

Key Responsibilities

Core System Design & Development

• Design autonomous logic for optimizing SQL and non-SQL analytic queries to leverage Infinia's distributed infrastructure.
• Implement high-performance indexing for structured and non-structured data using B-epsilon trees, full-text indexing, and vectorization.
• Develop internal systems for high-throughput data access and transformation using Parquet, ORC, and Avro.
• Engineer integration layers that support open interfaces like Trino, Apache Spark, Apache Iceberg, Delta Lake, HDFS, and Hive Metastore, enabling seamless compatibility with open-source clients.

Performance Optimization & Scaling

• Build and tune execution plans that leverage Infinia's high-throughput I/O and compute capabilities for large-scale AI and analytics workloads.
• Analyze and optimize performance of distributed query execution, data storage, caching, and memory usage.
• Write automated tests to validate correctness and performance of analytic queries among varied cluster topologies.

Open-Source Collaboration & Innovation

• Contribute to relevant open-source ecosystems, where appropriate, through collaboration, feature integration, or direct code contributions.
• Stay up to date with the evolving open data lake and query engine landscape to guide architectural decisions.

Cross-Functional Collaboration & Leadership

• Partner with Data Scientists, Platform Engineers, and Product Managers to deliver integrated, end-to-end solutions.
• Provide technical leadership, mentorship, and design direction to other engineers on the team.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in software development, with 5+ years in distributed systems, data platforms, or big data technologies.
• Expert-level knowledge of SQL, Python, and Java or Scala.
• Experience working with Apache Spark, distributed query engines, or distributed databases.
• Strong familiarity with HDFS, Hive Metastore, and data partitioning strategies.

Preferred Qualifications

• Hands‑on experience with Apache Iceberg and/or Delta Lake.
• Deep understanding of file formats including Parquet, ORC, Avro, and their performance characteristics.
• Background in real-time data streaming using tools such as Apache Kafka.
• Prior experience with C++.
• Prior contributions to open-source projects; committer status is a plus.
• Proven ability to lead complex technical initiatives and mentor junior engineers.

This position requires participation in an on-call rotation to provide after-hours support as needed.

Success Metrics – First 30 Days

Technical Integration

• Ramp up on Infinia's architecture, codebase, and core data processing capabilities.
• Shadow key design and development efforts across integration points and open-source connectors.

Early Impact

• Deliver a performance benchmark or prototype showcasing data access or query layer improvement.
• Identify 2–3 areas in the codebase or architecture where optimization or architectural refactoring would drive meaningful performance gains.

Team Engagement

• Begin providing technical guidance and mentorship within the Data Engine team.
• Partner with product and architecture teams to scope out upcoming integration initiatives.

Success Metrics – Beyond 30 Days

• Delivery of performant, production-ready connectors and execution engines integrated into Infinia.
• Measurable improvements in query throughput, latency, and data ingestion time across large-scale workloads.
• Positive feedback from peers and partners on technical leadership, code quality, and collaboration.
• Contributions to open-source ecosystems that reflect DDN's thought leadership and technical depth.

Join us to help shape the data engine behind the most advanced AI and analytics infrastructure – where open standards meet high performance, and distributed scale meets elegant execution.

Apply now to engineer the core of intelligent data access with DDN Infinia.

Interview Process

• Coding assessment: Often in a language of your choice.
• Systems design: Translate high-level requirements into a scalable, fault‑tolerant service (depending on role).
• Real-time problem-solving: Demonstrate practical skills in a live problem-solving session.
• Meet and greet with the wider team.
• Our goal is to finish the main process in 2–3 weeks at most.

DataDirect Networks (DDN) is an Equal Opportunity/Affirmative Action employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity, gender expression, transgender, sex stereotyping, sexual orientation, national origin, disability, protected Veteran Status, or any other characteristic protected by applicable federal, state, or local law.

#LI-Remote