Databricks Developer

UST; Bengaluru; On-site; INR 2,400,000 - 4,000,000; Full time

Job summary

UST Bengaluru seeks a Lead I - Data Engineering to design and optimize data pipelines using Databricks and PySpark. You will drive scalable data solutions and contribute to platform modernization.

You will lead engineering initiatives, implement CI/CD with GitHub Actions or Jenkins, ensure data quality and governance, and collaborate with cross-functional teams to translate business needs into technical solutions.

Qualifications

• 5+ years of experience in Data Engineering or related fields.
• Strong hands-on expertise in Databricks (notebooks, Delta Lake, job orchestration).
• Deep knowledge of Apache Spark (PySpark, Spark SQL, optimization techniques).
• Strong proficiency in Python for data processing, automation, and framework development.
• Strong proficiency in SQL, including complex queries, performance tuning, and analytical functions.
• Strong knowledge of Databricks Genie and leveraging it for engineering workflows.
• Strong experience with CI/CD and Git-based development workflows.
• Proficiency in data modeling and ETL/ELT pipeline design.
• Experience with automation frameworks and scheduling tools.
• Solid understanding of distributed systems and big data concepts.

Responsibilities

• Design, build, and optimize data pipelines and ETL/ELT workflows using Databricks and Apache Spark (PySpark).
• Develop scalable, high performance data solutions using Spark distributed processing.
• Lead engineering initiatives focused on automation, performance tuning, and platform modernization.
• Implement and manage CI/CD pipelines using Git-based workflows and tools such as GitHub Actions or Jenkins.
• Collaborate with cross-functional teams to translate business needs into technical solutions.
• Ensure data quality, governance, and security across all processes.
• Troubleshoot and optimize Spark jobs, Databricks clusters, and workflows.
• Participate in code reviews and develop reusable engineering frameworks.
• Should have knowledge of utilizing AI tools to improve productivity and support daily engineering activities.

Skills

Databricks

Apache Spark (PySpark)

Python

SQL

Data modeling

ETL/ELT design

CI/CD

Git-based workflows

Databricks Genie

Automation

Tools

GitHub Actions

Jenkins

Delta Lake

Job description

Who We Are

At UST, we help the world’s best organizations grow and succeed through transformation. Bringing together the right talent, tools, and ideas, we work with our client to co-create lasting change. Together, with over 30,000 employees in 30+ countries, we build for boundless impact—touching billions of lives in the process. Visit us at .

Role Description

Lead I - Data Engineering

Who We Are

At UST, we help the world’s best organizations grow and succeed through transformation. Bringing together the right talent, tools, and ideas, we work with our client to co-create lasting change. Together, with over 30,000 employees in 30+ countries, we build for boundless impact—touching billions of lives in the process. Visit us at .

Job Description

• Design, build, and optimize data pipelines and ETL/ELT workflows using Databricks and Apache Spark (PySpark).
• Develop scalable, high performance data solutions using Spark distributed processing.
• Lead engineering initiatives focused on automation, performance tuning, and platform modernization.
• Implement and manage CI/CD pipelines using Git-based workflows and tools such as GitHub Actions or Jenkins.
• Collaborate with cross-functional teams to translate business needs into technical solutions.
• Ensure data quality, governance, and security across all processes.
• Troubleshoot and optimize Spark jobs, Databricks clusters, and workflows.
• Participate in code reviews and develop reusable engineering frameworks.
• Should have knowledge of utilizing AI tools to improve productivity and support daily engineering activities.
• Strong knowledge and hands‑on experience in Databricks Genie, including prompt engineering, workspace usage, and automation. Required Skills & Experience:
• 5+ years of experience in Data Engineering or related fields.
• Strong hands-on expertise in Databricks (notebooks, Delta Lake, job orchestration).
• Deep knowledge of Apache Spark (PySpark, Spark SQL, optimization techniques).
• Strong proficiency in Python for data processing, automation, and framework development.
• Strong proficiency in SQL, including complex queries, performance tuning, and analytical functions.
• Strong knowledge of Databricks Genie and leveraging it for engineering workflows.
• Strong experience with CI/CD and Git-based development workflows.
• Proficiency in data modeling and ETL/ELT pipeline design.
• Experience with automation frameworks and scheduling tools.
• Solid understanding of distributed systems and big data concepts.

What We Believe

We’re proud to embrace the same values that have shaped UST since the beginning. Since day one, we’ve been building enduring relationships and a culture of integrity. And today, it's those same values that are inspiring us to encourage innovation from everyone, to champion diversity and inclusion and to place people at the centre of everything we do.

Humility

We will listen, learn, be empathetic and help selflessly in our interactions with everyone.

Humanity

Through business, we will better the lives of those less fortunate than ourselves.

Integrity

We honour our commitments and act with responsibility in all our relationships.

Equal Employment Opportunity Statement

UST is an Equal Opportunity Employer. We believe that no one should be discriminated against because of their differences, such as age, disability, ethnicity, gender, gender identity and expression, religion, or sexual orientation.

All employment decisions shall be made without regard to age, race, creed, colour, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

UST reserves the right to periodically redefine your roles and responsibilities based on the requirements of the organization and/or your performance.

• To support and promote the values of UST.
• Comply with all Company policies and procedures

Skills

Apache Spark, Big Data, CI/CD, Data Governance