Senior Lead Software Engineer - Java & Databricks

Fairygodboss; Bengaluru; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

JPMorganChase in Bengaluru is seeking a Senior Lead Software Engineer to drive design and delivery of scalable, secure, and compliant data-intensive applications within the Digital Communications Compliance team.

You will own end-to-end development of microservices and data pipelines using Databricks, Spark, Delta Lake, and cloud-native technologies, mentor engineers, and champion best practices across SDLC, testing, observability, and AI-assisted development.

Qualifications

• Formal training or certification in software engineering with 5+ years of applied experience.
• Hands-on practical experience delivering system design, application development, testing, and operational stability.
• Expertise in building resilient, scalable, enterprise-grade cloud-native products, with strong exposure in compliance for the financial industry.
• Databricks production workloads with Spark, Delta Lake, Databricks Workflows; strong Java/Kotlin and Python API development; microservices for streaming/batch with Spark/Flink.
• AWS services (EC2, S3, EMR, EKS); Kafka, PostgreSQL; ML/LLM pipelines in production; observability (Prometheus, Grafana, OpenTelemetry).
• Experience building CI/CD pipelines using ArgoCD, Helm, Terraform, Jenkins, and GitHub Actions.
• Prior experience in Test-Driven Development, delivering products with well-defined SLI/SLO/SLAs.
• Strong ownership mentality with excellent communication skills and a collaborative mindset; mentoring engineers and providing technical leadership.

Responsibilities

• Design and develop scalable microservices and APIs for rule-based and ML-based detection pipelines.
• Model and implement reviewer workflows with state machines.
• Build streaming and batch data pipelines that ingest, index, and enrich communications content and alerts.
• Design data models on Databricks for surveillance, search, retention, and lifecycle management at scale.
• Develop and optimize large-scale batch and streaming pipelines using Databricks, Spark, and Delta Lake.
• Build tests with TDD; automate data processing and deployments using Databricks Workflows and CI/CD.
• Drive adoption of Databricks best practices across data eng and software eng, observability, and platform operations.
• Promote AI-assisted engineering practices with measurable validation standards.
• Apply SDLC tooling and AI-assisted development to improve automation.
• Partner with product management and compliance SMEs to monitor alert accuracy and reliability.
• Identify hidden patterns in communications data to improve detection quality.

Skills

Java/Kotlin

Python

Databricks

Spark

APIs

AWS

CI/CD

Test-Driven Development

Mentoring

Observability

Education

Software engineering certification

Tools

Databricks

Spark

Delta Lake

Prometheus

Grafana

OpenTelemetry

ArgoCD

Helm

Terraform

Jenkins

GitHub Actions

Job description

Be an integral part of an agile team that's constantly pushing the envelope to enhance, build, and deliver top-notch technology products.

As a Senior Lead Software Engineer at JPMorganChase within the Digital Communications Compliance team, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. Drive significant business impact through your capabilities and contributions, and apply deep technical expertise and problem-solving methodologies to tackle a diverse array of challenges that span multiple technologies and applications.

Job responsibilities

• Design and develop scalable, fault-tolerant microservices and APIs that support rule-based and ML-based detection pipelines
  Model and implement supervision and reviewer workflows using state machines
• Build streaming and batch data pipelines that ingest, index, and enrich communications content and alerts
• Design data models on Databricks for surveillance, search, retention, and life cycle management at scale
• Develop and optimize large-scale batch and streaming data pipelines using Databricks, Apache Spark, and Delta Lake
• Build robust unit, integration, and performance tests following Test-Driven Development. Leverage Databricks Workflows, notebooks, and CI/CD pipelines to automate data processing and deployments. Troubleshoot and tune Databricks jobs, clusters, and data pipelines for efficiency and performance
  Drive adoption of Databricks best practices for data engineering, software engineering, observability, and platform operations
• Drives adoption and governance of approved AI-assisted engineering practices across teams to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test acceleration, release readiness, incident/root-cause analysis), while establishing measurable validation standards (secure coding, peer review, automated testing) and promoting reuse of proven patterns and automation within the SDLC/TLM toolchain.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including approved AI-assisted development and automation capabilities, to improve the value realized by automation at scale.
• Partner with product management and compliance SMEs to monitor and improve alert accuracy and reliability. Proactively identify hidden problems and patterns in communications data to improve detection quality

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years applied experience
• Hands-on practical experience delivering system design, application development, testing, and operational stability
• Expertise in building resilient, scalable, enterprise-grade cloud-native products, with strong exposure in compliance for the financial industry
• Hands-on Databricks experience including development of production workloads using Spark, Delta Lake, and Databricks Workflows. Expert Java/Kotlin and Python programmer with experience building headless, externally consumable APIs. Experience building cloud-native microservices for streaming and batch architectures using Spark and/or Flink
• Proficient with AWS services including EC2, ECS, EKS, EMR, S3, and Glacier. Hands-on with Semantic search, Kafka, and PostgreSQL. Experience integrating and operationalizing ML/LLM models and pipelines in production. Experience with observability and monitoring tools such as Prometheus, Grafana, and OpenTelemetry
• Experience building CI/CD pipelines using ArgoCD, Helm, Terraform, Jenkins, and GitHub Actions
• Prior experience in Test-Driven Development, delivering products with well-defined SLI/SLO/SLAs
• Strong ownership mentality with excellent communication skills and a collaborative mindset. Experience mentoring engineers and providing technical leadership at a senior level

Preferred qualifications, capabilities, and skills

• Exposure to Unity Catalog, and Databricks Workflows
• Exposure to building and optimizing large-scale batch and streaming data pipelines in cloud-native environments
• Understanding of distributed systems, data engineering best practices, and operationalizing data and ML workloads in production