Lead Software Engineer - PySpark

Societe Generale SA; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Societe Generale SA in Bengaluru, India is seeking a senior data engineering lead to drive the design and impact analysis of new features and data pipelines. You will own the SDLC in an Agile environment, mentor teams, and oversee code quality, automation, and production health.

You will work with PySpark, SQL, ETL concepts on Azure Databricks, Delta Lake and Data Factory to build scalable data solutions, implement audit recommendations, and continually modernize applications.

Qualifications

• 8+ years of experience on Microsoft Azure environment.
• Design, Develop code on Pyspark, Design Patterns, Design Principles and Functional Patterns.
• Strong ETL and data warehousing concepts.
• Hands-on experience with Pyspark, Databricks, DataFactory and ADLS.
• Experience with Azure Devops and build/release pipelines is a plus.
• Knowledge of MS SQL and Power BI is advantageous.
• Experience developing scalable data ingest/processing frameworks.

Responsibilities

• Lead the design and impact analysis of all new features and enhancements.
• Lead SDLC in-line with Agile and IT craftsmanship principles.
• Mentor teams and ensure code quality through reviews.
• Automate maintenance tasks to improve efficiency.
• Progress application modernization initiatives and DevOps practices.
• Ensure data quality through testing and validation.
• Collaborate with cross-functional teams and stakeholders.

Skills

PySpark

SQL

ETL concepts

Azure Databricks

Delta Lake

Azure Data Factory

Azure Data Lake Store

Python

Power BI

Azure DevOps

MS SQL

Tools

Azure Databricks

Delta Lake

Data Factory

Power BI

ADLS

Job description

ResponsibilitiesLead the design and impact analysis of all new features and enhancements.Lead all aspects of Software Development Lifecyle (SDLC) in-line with Agile and IT craftsmanship principles.Identify methods to sustain and improve various craftsmanship areas.Lead the timely delivery of assigned artifacts with defined quality parameters.Perform code review and ensure the quality of code, constantly.Automate manual tasks of application maintenance and support to improve efficiency.Lead initiatives of application modernization.Ensure constant review and update of Agile and DevOps practices.Constantly learn new/emerging technologies and mentor teams.Monitor the overall production processes such as daily checks, open tickets and aging of issues.Collaborate with customers, partners, development teams, chapter and feature teams.Co-ordinate with development team and implement audit recommendations.Lead DevOps chapters and Guilds.Complete the assigned learning path and contribute to daily meetings.Guide the team on data processing solutions and building data pipelines.Identify new areas of technology and use cases for data validation and implementation.Profile requiredPrimary skills - PySpark, Sql, ETL conceptsSecondary skills - Azure cloud experience(Azure databricks, Delta lake, Azure data factory, Azure data lake store) PythonGood to have - Data modelling and visualization (Power BI, Azure Analysis Services) Microsoft Azure with good knowledge on Data Bricks, Data factory and Delta Lake* 8+ years of experience on Microsoft Azure environment.* Design, Develop code on Pyspark, Design Patterns, Design Principles and Functional Patterns .*Good understanding of ETL Processes and data warehousing concepts.*Good hands-on Pyspark, Databricks, DataFactory and ADLS.*Good hands on Azure Devops, understanding of build and release pipeline will be added advantage.*Good hands-on MS SQL, experience of Power BI will be added advantage.* Knowledge on Developing scalable and reusable frameworks for ingesting and enhancing datasets.*Good analytical and troubleshooting skills to resolve the issues in timebound manner.*Ensure data quality and accuracy through testing and validation.*Effective team player with ability to work autonomously and in team with cross-cultural environment*Effective verbal and written communication to work closely with all the stakeholdersWhy join usWe are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status.Business insightAt Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious. Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating, and taking action are part of our DNA. If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a daily basis and develop or strengthen your expertise, you will feel right at home with us!Still hesitating?You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices, and sharing their skills with charities. There are many ways to get involved.We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.Diversity and InclusionWe are an equal opportunities employer and we are proud to make diversity a strength for our company. Societe Generale is committed to recognizing and promoting all talents , regardless of their beliefs, age, disability, parental status, ethnic origin, nationality, gender identity, sexual orientation, membership of a political, religious, trade union or minority organisation, or any other characteristic that could be subject to discrimination.