Specialist Software Engineer - Data Engineer Databricks pySpark

Societe Generale SA; Bengaluru; On-site; INR 800,000 - 1,600,000; Full time

Job summary

Societe Generale SA in Bengaluru is seeking a Specialist Software Engineer – Data Engineer with expertise in Databricks-pySpark to design and build scalable data pipelines. You will own data flows within the Hadoop ecosystem and optimize performance on Databricks while collaborating with global teams in an Agile environment.

Experience with data governance, CI/CD practices, and production support is valued as you contribute to architecture decisions and runbooks.

Qualifications

• Bachelor's degree in engineering or equivalent.
• Strong hands-on experience in PySpark for distributed processing and optimization.
• Solid understanding of Hadoop ecosystem tools: Hive, HDFS, YARN, Oozie, Starburst.
• Proven experience designing and maintaining data pipelines in Big Data environments.
• Good command of Python for pipeline utilities.
• Strong understanding of data modeling, partitioning, and governance.

Responsibilities

• Own design and development of robust, high-performance data pipelines using Databricks(PySpark).
• Build and optimize data flows within Hadoop ecosystem (Hive, HDFS, Starburst, Oozie).
• Ensure performance tuning, resource optimization on the Databricks platform.
• Contribute to data architecture decisions around storage, processing, security, exposure.
• Apply CI/CD, unit/integration testing, and pipeline validation practices.
• Model data based on business needs and maintain data dictionaries and governance rules.
• Participate in Agile ceremonies and collaborate with Product Owners, architecture, business, and Ops teams.
• Support production processes, troubleshoot issues, update runbooks, and maintain system stability.

Skills

Databricks

pySpark

Python

Data modeling

Git

Education

Bachelor of Engineering in Computer Science/IT

Tools

Hive

HDFS

YARN

Oozie

Starburst

NiFi

Kafka

Cloudera Manager

Spark SQL

Databricks

Job description

ResponsibilitiesSpecialist Software Engineer – Data Engineer (Databricks-pySpark)ResponsibilitiesUnderstand business and technical requirements and translate them into scalable, maintainable data engineering solutions.Own the design and development of robust, high‑performance data pipelines using Databricks(pySpark) .Build and optimize data flows within the Hadoop ecosystem (Hive, HDFS, Starburst, Oozie).Ensure performance tuning, resource optimization, and efficient cluster usage on Databricks platform.Contribute to data architecture decisions related to storage, processing, security, and exposure.Apply industrialization best practices including CI/CD, unit testing, integration testing, and pipeline validation.Model data based on business needs and maintain shared data components, dictionaries, and governance rules.Perform functional and technical analysis, document specifications, and validate implemented solutions.Participate in Agile ceremonies and collaborate with Product Owners, architecture, business, and Ops teams.Support production processes, troubleshoot issues, update runbooks, and maintain system stability.Ensure compliance with data quality, traceability, and security standards across the data lifecycle.Engage in continuous improvement activities, proposing upgrades, refactoring, and automation opportunities.Work closely with global teams and support cross‑location coordination when required.Demonstrate ownership during key delivery phases, including planned on‑call or deployment activities.Profile requiredProfile RequiredExperienced Data Engineer with strong expertise in Databricks , Hadoop ecosystem components, and large‑scale data processing environments.Minimum QualificationsBachelor of Engineering in Computer Science, Information Technology and equivalent streams.Strong hands‑on experience in pySaprk for distributed processing and performance optimization.Solid understanding of Hadoop ecosystem tools: Hive, HDFS, YARN, Oozie, Starburst.Proven experience designing and maintaining data pipelines in complex Big Data environments.Good command of Python for supplemental processing and pipeline utilities.Strong understanding of data modeling, partitioning strategies, and governance principles.Proficiency in Git and modern SDLC practices (branching, PR reviews, tagging).Experience with CI/CD pipelines (GitHub Actions, Ansible, AWX).Strong documentation skills and ability to produce clear technical specifications.Preferred Qualifications4+ years of experience in data engineering or large‑scale distributed systems.Experience with NiFi/Kafka for batch and real‑time ingestion.Working knowledge of JFrog (artifact deployment, vulnerability remediation).Strong experience in Apache Hive performance tuning and Spark SQL integration.Exposure to Cloudera Manager for monitoring clusters and troubleshooting.Familiarity with Data Governance, security policies, and traceability frameworks.Experience in Agile @ Scale (SAFe) and backlog contribution (user stories, acceptance criteria).Ability to work independently and collaboratively across multicultural teams.Strong analytical, communication, and problem‑solving skills.cloud technologies (Azure/AWS) and modern platforms like Databricks .Why join us“We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status”Business insightAt Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious.Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating and taking action are part of our DNA.If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a daily basis and develop or strengthen your expertise, you will feel right at home with us!Still hesitating?You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices and sharing their skills with charities. There are many ways to get involved.We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.Diversity and InclusionWe are an equal opportunities employer and we are proud to make diversity a strength for our company. Societe Generale is committed to recognizing and promoting all talents , regardless of their beliefs, age, disability, parental status, ethnic origin, nationality, gender identity, sexual orientation, membership of a political, religious, trade union or minority organisation, or any other characteristic that could be subject to discrimination.