Lead Software Engineer -Python or Java, big data

United States Digital Space LLC; Karnataka; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

United States Digital Space LLC is seeking an Experienced Software Engineer to join its Corporate Technology team in India. You will design and operate scalable data platforms, build ingestion, processing, and storage layers, and leverage Python and Java with Spark/Databricks to accelerate data-driven products and ML workloads.

The role requires 10+ years in software engineering, strong backend expertise, and collaboration across global teams in distributed locations.

Qualifications

• 10+ years of hands-on software engineering experience delivering data platforms and backend systems.
• Strong proficiency in Python and Java for APIs, services, and distributed systems.
• Extensive experience with Spark, Databricks, Data Lake technologies and large-scale data pipelines.
• Experience designing and operating data platform components (ingestion, processing, storage, access) in AWS cloud ecosystems.

Responsibilities

• Provide technical guidance to business and tech teams, contractors, and vendors.
• Design, develop, and operate scalable data platform services and pipelines with production-grade code.
• Drive architecture decisions for data platform and application functionality.
• Serve as subject matter expert in data processing, storage, and platform reliability.
• Advocate for SDLC automation and AI-assisted engineering practices across teams.
• Ensure secure handling of data, governance, and responsible AI usage in workflows.
• Collaborate with global teams across time zones to deliver solutions.

Skills

Python

Java

Big data

Distributed systems

Tools

Spark

Databricks

Airflow

Kafka

AWS (ECS, EKS, EMR, Lambda)

Job description

JOB DESCRIPTION

As an Experienced Software Engineer at the company within the Corporate Technology team, you will be a key member of an agile team, responsible for designing and delivering the trusted, market-leading data platforms and infrastructure that our products and teams depend on, in a secure, stable, and scalable manner. In this role, you will build and operate the foundational data platform components like ingestion, processing, storage, and access layers, that enable analytics, machine learning, and application workloads across the firm. You will leverage a mix of Python and Java, alongside big data technologies such as Spark and Databricks, to accelerate and harden our data ecosystem.

Job responsibilities

• Regularly provides technical guidance and direction to support the business and its technical teams, contractors, and vendors
• Designs, develops, and operates scalable data platform services and pipelines, and produces secure and high-quality production code, while reviewing and debugging code written by others
• Drives decisions that influence data platform architecture, application functionality, and technical operations and processes
• Serves as a function-wide subject matter expert in one or more areas of focus (e.g., distributed data processing, data storage, platform reliability)
• Actively contributes to the engineering community as an advocate of firmwide frameworks, tools, and practices of the Software Development Life Cycle
• Leads team adoption of enterprise-authorized AI-assisted engineering practices and SDLC/TLM automation to improve delivery speed, quality, and operational outcomes, while setting expectations for human validation, secure handling of inputs/outputs, and consistent use of reusable patterns across teams.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation and support capacity unlock initiatives.
• Influences peers and project decision-makers to consider the use and application of leading-edge data and backend technologies

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 10+ years applied experience
• Hands‑on practical experience delivering system design, application development, testing, and operational stability for data platforms and backend systems
• Advanced proficiency across both Python and Java, with strong general backend engineering experience (e.g., building APIs, services, and distributed systems)
• Strong big data and database skills, including hands‑on experience with Spark, Databricks, and/or Data Lake, and building large-scale data pipelines
• Experience designing and operating data platform components like ingestion, processing, storage, and access/serving layers and experience with AWS cloud computing using ECS, EKS, EMR, Lambda, etc.
• Experience leading responsible adoption of enterprise-authorized AI-assisted development and delivery tools across engineering teams, including defining ways of working (review/validation expectations), measuring outcomes, and ensuring secure handling of data.
• Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, resiliency/security implications, and governance expectations; ability to coach engineers on compliant and effective usage.
• Advanced knowledge of software applications and technical processes with considerable in-depth knowledge in one or more technical disciplines (e.g., cloud, distributed data processing, artificial intelligence, machine learning, etc.)
• Ability to tackle design and functionality problems independently with little to no oversight
• Experience in developing, debugging, and maintaining code in a large corporate environment and ability to collaborate well with global teams in geographically distributed locations across time zones
• Self‑starter, able to reach out to various team members, users, and partner teams to get solutions delivered.

Preferred qualifications, capabilities, and skills

• AI, ML, Claude, MCP
• Familiar with agile development methodologies (e.g., Scrum) and CI/CD, Applicant Resiliency, and Security
• Experience with data orchestration and workflow tools (e.g., Airflow) and streaming technologies (e.g., Kafka)