Senior Software Engineer_Data

Giant Eagle GCC; Bengaluru; On-site; INR 1,200,000 - 1,900,000; Full time

Job summary

Giant Eagle Bangalore GCC seeks a Software Engineer – Data Engineering to design, build, and optimize enterprise data pipelines for marketing analytics. You will develop scalable batch and streaming workflows on Databricks using PySpark, Spark SQL, and Delta Lake, and implement data governance with Unity Catalog.

You will migrate Oracle-based ETL to Databricks, collaborate with data analysts and application teams, and contribute to CI/CD and data quality initiatives in a fast-paced environment.

Qualifications

• 5+ years’ experience in software engineering.
• 3+ years’ hands-on experience with Databricks, including PySpark, Spark SQL, Delta Lake, notebooks, and Databricks Workflows/Jobs.
• Strong programming skills in Python and advanced SQL.
• Experience with Unity Catalog, including storage credentials, external locations, and access management.
• Experience with cloud data platforms (Azure preferred), including ADLS Gen2, Azure Data Factory, and related services.
• Strong knowledge of data modeling, data warehousing, and Lakehouse/medallion architecture concepts.
• Experience building batch and streaming pipelines (e.g., Structured Streaming, Auto Loader, Kafka or Event Hubs).
• Experience with data quality, testing, and monitoring practices for data pipelines.
• Working knowledge of relational databases (Oracle / SQL Server), including basic PL/SQL.
• Experience with Unix/Linux and shell scripting.
• Understanding of APIs, RESTful services, and integration with applications.
• Experience working with large-scale enterprise data.
• Proven experience leading technical design, conducting code reviews, and mentoring team members.
• Strong problem-solving and analytical skills.
• Experience with .NET development (C#, ASP.NET Core, Web API).
• Experience with infrastructure as code (e.g., Terraform) for Databricks and Azure resources.
• Experience with integration tools such as SSIS or MuleSoft.
• Experience with BI tools such as Power BI.
• Experience with customer / loyalty / marketing data.

Responsibilities

• Own the technical design and architecture of data engineering solutions, ensuring scalability, reliability, and cost efficiency.
• Translate business requirements into technical designs, data models, and pipeline specifications.
• Interact with analysts, users, and business stakeholders to gather, understand, and document technical requirements.
• Establish and enforce data engineering standards, coding best practices, and reusable frameworks.
• Implement CI/CD for data pipelines and manage code using Git and Azure DevOps.
• Provide technical leadership and mentoring to junior engineers.
• Conduct formal code reviews and quality assurance activities for newly developed pipelines and programs.
• Prepare and maintain documentation, including data lineage, data dictionaries, and operational runbooks.
• Maintain and support existing Oracle PL/SQL processes and SQL queries as needed during the transition to Databricks.
• Recommend and implement process improvements to enhance efficiency and system performance.
• Participate in an off-hour support rotation to provide critical system support.
• Provide training and knowledge transfer to the Application Service Desk Team for 24x7 business support.
• Plan and monitor testing and deployment of data pipelines to ensure smooth releases with minimal downtime.
• Ensure data security, privacy, and compliance with company security policies and best practices.
• Prevent unauthorized access to sensitive information and assets while reporting any security attempts.
• Identify and report system issues or challenges to appropriate management in a timely manner.
• Continuous Learning & Industry Awareness: Stay updated with industry trends, recent technologies, and data engineering and cloud platform advancements to maximize IT investment benefits.
• Participate in educational and on-the-job training programs as recommended by IS Management.
• Assist other team members as needed to ensure customer requirements are met.
• Effectively communicate with non-IT personnel to explain technical concepts in an understandable manner.
• Support special projects and undertake additional responsibilities as assigned.

Skills

Databricks

Python

SQL

Data Modeling

Education

Bachelor’s Degree in Information Systems

Tools

Azure Data Factory

Unity Catalog

Power BI

Terraform

Kafka

Job description

Grow. Learn. Thrive. Giant Eagle is where your career can soar high. At Giant Eagle, you are so much more than an employee. You are part of the Family.

About the Company

Since our founding in 1931, Giant Eagle, Inc. has evolved into one of the top 40 largest private corporations in the U.S. and one of the country’s largest food retailers and distributors. With more than 37,000 Team Members and $9.7billion in revenue, we are committed to investing in people, technology, and data to elevate our customer’s experience across multiple touchpoints. It helps us follow on our commitment to serving others and improving our communities.

About Giant Eagle Bangalore

The Giant Eagle GCC in Bangalore is our global capability center. Our team of more than 350 members at the GCC enables us to expand internal capabilities in the areas such as data analytics, merchandising and eCommerce, quality engineering, and automation to generate insights for faster decision-making and help us accelerate our business strategy. Our team in India plays a pivotal role in helping the company transition to new ways of working by redefining the food and grocery shopping experience for over 4.6 million customers.

Position Summary:

The Software Engineer Team Member is part of the Marketing Technology Team; under general direction, they will design, develop, and implement systems to ensure customer requirements are met in a timely and cost-effective manner. The role is responsible for enterprise Marketing Technology. Key development responsibilities include the development of internal systems, management & integration of third-party software systems, and systems support.

Job Description:

• Design, build, and maintain scalable batch and streaming data pipelines on Databricks using PySpark, Spark SQL, and Delta Lake.
• Develop ingestion frameworks to bring data from databases, APIs, files, and third-party marketing platforms into the Lakehouse.
• Build and maintain curated data models using the medallion (bronze/silver/gold) architecture for analytics, reporting, and marketing use cases.
• Implement data governance, security, and access control using Unity Catalog.
• Build data quality checks, validation, and monitoring to ensure the accuracy, completeness, and reliability of data.
• Orchestrate and schedule pipelines using Databricks Workflows/Jobs and Azure Data Factory where applicable.
• Optimize Spark jobs, cluster configurations, and Delta tables (partitioning, Z-ordering, file compaction) for performance and cost.
• Support the migration and modernization of legacy Oracle-based ETL and PL/SQL processes to Databricks.
• Collaborate with data analysts, data scientists, business stakeholders, and application teams to deliver data solutions.
• Troubleshoot and resolve pipeline failures, data issues, and performance bottlenecks.

Job Responsibilities:

• Own the technical design and architecture of data engineering solutions, ensuring scalability, reliability, and cost efficiency.
• Translate business requirements into technical designs, data models, and pipeline specifications.
• Interact with analysts, users, and business stakeholders to gather, understand, and document technical requirements.
• Establish and enforce data engineering standards, coding best practices, and reusable frameworks.
• Implement CI/CD for data pipelines and manage code using Git and Azure DevOps.
• Provide technical leadership and mentoring to junior engineers.
• Conduct formal code reviews and quality assurance activities for newly developed pipelines and programs.
• Prepare and maintain documentation, including data lineage, data dictionaries, and operational runbooks.
• Maintain and support existing Oracle PL/SQL processes and SQL queries as needed during the transition to Databricks.
• Recommend and implement process improvements to enhance efficiency and system performance.
• Participate in an off-hour support rotation to provide critical system support.
• Provide training and knowledge transfer to the Application Service Desk Team for 24x7 business support.
• Plan and monitor testing and deployment of data pipelines to ensure smooth releases with minimal downtime.
• Ensure data security, privacy, and compliance with company security policies and best practices.
• Prevent unauthorized access to sensitive information and assets while reporting any security attempts.
• Identify and report system issues or challenges to appropriate management in a timely manner.
• Continuous Learning & Industry Awareness:
• Stay updated with industry trends, recent technologies, and data engineering and cloud platform advancements to maximize IT investment benefits.
• Participate in educational and on-the-job training programs as recommended by IS Management.
• Assist other team members as needed to ensure customer requirements are met.
• Effectively communicate with non-IT personnel to explain technical concepts in an understandable manner.
• Support special projects and undertake additional responsibilities as assigned.

Experience:

Required:

• 5+ years’ experience in software engineering.
• 3+ years’ hands-on experience with Databricks, including PySpark, Spark SQL, Delta Lake, notebooks, and Databricks Workflows/Jobs.
• Strong programming skills in Python and advanced SQL.
• Experience with Unity Catalog, including storage credentials, external locations, and access management.
• Experience with cloud data platforms (Azure preferred), including ADLS Gen2, Azure Data Factory, and related services.
• Strong knowledge of data modeling, data warehousing, and Lakehouse/medallion architecture concepts.
• Experience building batch and streaming pipelines (e.g., Structured Streaming, Auto Loader, Kafka or Event Hubs).
• Experience with data quality, testing, and monitoring practices for data pipelines.
• Working knowledge of relational databases (Oracle / SQL Server), including basic PL/SQL.
• Experience with Unix/Linux and shell scripting.
• Understanding of APIs, RESTful services, and integration with applications.
• Experience working with large-scale enterprise data.
• Proven experience leading technical design, conducting code reviews, and mentoring team members.
• Strong problem-solving and analytical skills.
• Experience with .NET development (C#, ASP.NET Core, Web API).
• Experience with infrastructure as code (e.g., Terraform) for Databricks and Azure resources.
• Experience with integration tools such as SSIS or MuleSoft.
• Experience with BI tools such as Power BI.
• Experience with customer / loyalty / marketing data.

Required Training and Education:

• Bachelor’s Degree Information Systems or equivalent job experience