Senior Data Engineer Lead

Optum; Hyderabad; Hybrid; INR 4,000,000 - 7,000,000; Full time

Job summary

Optum in Hyderabad seeks an experienced data engineer to design and optimize large-scale data pipelines using Azure Databricks, Apache Spark, and Scala. You will work with Spark SQL, manage distributed data processing, and integrate AI tooling for code and data tasks.

The role requires deep knowledge of cloud platforms (Azure) and big data technologies, plus experience with SQL, Data Warehousing, and ETL processes. Collaboration with DevOps and API teams is expected.

Qualifications

• 10+ years proven experience with Azure Databricks, Apache Spark, and Scala.
• Strong understanding of distributed computing principles and big data processing.
• Proficiency in writing complex SQL queries and working with relational databases.
• 5-7 years' experience with Microsoft Azure.
• Experience with data warehousing solutions and ETL tools.
• Ability to adopt AI tools in applications like Github Copilot, StoryCraft, GenAI etc.
• Development experience with REST or GraphQL APIs.
• Familiarity with Hadoop, Hive, HDFS and data lakes.

Skills

Scala

Apache Spark

Spark SQL

Databricks

Distributed computing

Big data concepts

REST/GraphQL

DevOps basics

Hadoop/Hive/HDFS

Kubernetes

Azure cloud

AI tools (Copilot/GenAI)

Java/.NET basics

Education

Undergraduate degree or equivalent experience

Tools

Azure Databricks

Azure SQL Server

Kubernetes

GitHub Copilot

M365 Copilot

StoryCraft

GenAI

Hadoop

Hive

ETL tools

Job description

Strong experience in working with Databricks, including its architecture, features, and capabilities.
In-depth knowledge and hands‑on experience with Scala programming language, including its functional programming concepts, collections, and advanced language features.
Expertise in Apache Spark, including Spark RDDs, DataFrames, and Spark SQL. Understanding of Spark's distributed computing model and ability to optimize and tune Spark jobs for performance.
Solid experience in Cloud-based deployment platforms like Azure, Kubernetes, etc.
Ability to adopt AI tools in application like Github Copilot, M365 Copilot, StoryCraft, GenAI etc.
Experience with RDBMS(Azure SQL Server).
Developer should have DevOps hands on experience like build and deployment pipeline creations.
Developer should API development experience like REST or GraphQL
Good to have knowledge on Java / J2EE or .Net
Familiarity with big data technologies and concepts, such as Hadoop, Hive, HDFS, and data lakes. Experience in working with large-scale datasets and processing them efficiently.

Qualifications - External

• Undergraduate degree or equivalent experience.
• 10+ years Proven experience with Azure Databricks, Apache Spark, and Scala.
• Strong understanding of distributed computing principles and big data processing.
• Ability to adopt AI tools in application like Github Copilot, StoryCraft, GenAI etc.
• Proficiency in writing complex SQL queries and working with relational databases.
• Experience with data warehousing solutions and ETL tools.
• 5-7 years' experience with Microsoft Azure.
• Excellent problem-solving skills and attention to detail.