Senior Data Engineer

IN01 Illumina India

Bengaluru

On-site

INR 1,800,000 - 3,200,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Illumina, Inc. in Bengaluru is seeking a Senior Data Engineer to design, build, and support enterprise data pipelines that integrate SAP S/4HANA data into the Databricks Lakehouse via SAP Datasphere and Amazon S3.

You will own SAP datasets end-to-end, from source analysis to production support, collaborating with SAP, platform, governance, and business teams. The role requires hands-on Databricks delivery, PySpark and SQL expertise, CI/CD practices, and strong documentation.

Qualifications

  • Minimum 6 years of enterprise data engineering or data integration experience including SAP data integration.
  • Hands-on Databricks delivery with PySpark, Spark SQL, Delta Lake, Auto Loader, Lakeflow DL Pipelines and Unity Catalog.
  • Strong SQL and Python skills for reconciliation, validation, and automation.

Responsibilities

  • Own end-to-end onboarding of SAP objects with source analysis, CDS wrapper view creation, Datasphere replication, Bronze ingestion, validation, and sign-off.
  • Design and maintain Bronze ingestion pipelines in Databricks for full, delta, CDC, and watermark loads.
  • Monitor Datasphere replication to S3, resolve connectivity and capacity issues with platform team.
  • Run source-to-Bronze reconciliation, investigate discrepancies, and document corrective actions.
  • Plan SIT/UAT, reconcile SAP validation with Bronze data, and compile acceptance evidence.
  • Promote changes via Git and CI/CD; manage Unity Catalog access controls.

Skills

Databricks
PySpark
SQL
Python
Git
CI/CD
Unity Catalog
SAP S/4HANA
Data integration

Education

Bachelor’s degree

Tools

Databricks
Delta Lake
Auto Loader
Lakeflow DL Pipelines
Unity Catalog

Job description

What if the work you did every day could impact the lives of people you know? Or all of humanity? At Illumina, we are expanding access to genomic technology to realize health equity for billions of people around the world. Our efforts enable life-changing discoveries that are transforming human health through the early detection and diagnosis of diseases and new treatment options for patients. Working at Illumina means being part of something bigger than yourself. Every person, in every role, has the opportunity to make a difference. Surrounded by extraordinary people, inspiring leaders, and world changing projects, you will do more and become more than you ever thought possible.

Position Summary: The Senior Data Engineer will be part of the Data Engineering and Analytics team, designing, building, validating, and supporting enterprise data pipelines that integrate SAP S/4HANA data into the Databricks Lakehouse through SAP Datasphere and Amazon S3. The engineer owns assigned SAP datasets end to end, from source analysis and ingestion through reconciliation, testing, release, and production support, within approved architecture, security, and change-control standards. The role partners with SAP, platform, architecture, governance, and business teams, leads complex troubleshooting and continuous improvement, and provides technical guidance to other engineers.

Responsibilities
  • Own end-to-end onboarding of assigned SAP objects following the standard onboarding runbook, including source analysis, CDS wrapper view creation, Datasphere replication flow setup, Bronze ingestion configuration, validation, and sign-off.
  • Design, build, and maintain Bronze ingestion pipelines in Databricks (Auto Loader, Lakeflow Declarative Pipelines, PySpark, Delta Lake) for full, delta, CDC, and watermark loads, handling restartability, deletes, and schema changes within approved design patterns.
  • Operate and monitor Datasphere replication flows into Amazon S3, recover failed runs, and work with the platform administration team to resolve connectivity and capacity issues.
  • Run source-to-Bronze reconciliation as a continuous control; review PASS/FAIL results, investigate aggregate and record-level discrepancies, and document corrective actions.
  • Plan and execute SIT/UAT scenarios covering full loads, deltas, deletes, and field additions or removals; reconcile SAP validation output against Bronze data and compile acceptance evidence.
  • Monitor pipeline observability dashboards and alerts; resolve ingestion failures and schema drift, lead root cause analysis (RCA) for complex issues, and implement preventive actions.
  • Promote all changes through Git and CI/CD, review peer code and configurations, and apply approved Unity Catalog access controls for assigned datasets.
  • Develop automation and reusable components that improve onboarding, reconciliation, monitoring, and recovery.
  • Maintain the SAP object inventory (extraction method, load pattern, Bronze location, delivery wave, status, and exceptions) along with per-object operational documentation, recovery procedures, and knowledge-transfer material.
  • Resolve Jira and ServiceNow tickets for SAP data pipelines, record hypercare incidents and resolutions, mentor junior engineers, and provide after-hours on-call support on a rotating basis.
  • undefined Listed responsibilities are an essential, but not exhaustive list, of the usual duties associated with the position. Changes to individual responsibilities may occur due to business needs.
Requirements
  • Required 6+ years of enterprise data engineering or data integration experience, including hands-on Databricks delivery and production SAP data integration.
  • Hands-on Databricks experience with PySpark, Spark SQL, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines (DLT), Workflows, Databricks SQL, and Unity Catalog.
  • Advanced SQL and Python, including reconciliation queries, aggregate comparisons, record-level validation, and automation scripting.
  • SAP S/4HANA knowledge, including core table structures, CDS view development, ODP delta extraction, and record-level verification (SE16).
  • Strong understanding of full, delta, pseudo-delta, and CDC loads, including watermarks, delete handling, restartability, error handling, and schema drift.
  • Practical experience with Git, code review, testing, and CI/CD deployment across environments.
  • Experience with pipeline monitoring and incident management using Databricks dashboards, Grafana, Jira, and ServiceNow.
  • Proven ability to troubleshoot complex production issues, lead root cause analysis, and maintain clear technical documentation.
  • Strong communication and collaboration skills across SAP, platform, and business teams, with experience mentoring junior engineers.
  • Flexibility to work occasional non-standard hours and participate in a rotating on-call schedule; international travel may be needed.
  • Preferred Experience with SAP Datasphere / Business Data Cloud replication flows, or replication tools such as SLT, HVR, or Fivetran.
  • SIT/UAT, cutover, and hypercare experience on an ERP program.
  • Exposure to dbt.
  • Experience using AI-enabled engineering tools, such as Cursor, Claude, ChatGPT, or GitHub Copilot.
Experience/Education
  • Typically requires a minimum of 6 years of related experience with a Bachelor’s degree; or 3 years and a Master’s degree; or a PhD without experience; or equivalent work experience.

We are a company deeply rooted in belonging, promoting an inclusive environment where employees feel valued and empowered to contribute to our mission. Built on a strong foundation, Illumina has always prioritized openness, collaboration, and seeking alternative perspectives to propel innovation in genomics.

We are proud to confirm a zero-net gap in pay, regardless of gender, ethnicity, or race. We also have several Employee Resource Groups (ERG) that deliver career development experiences, increase cultural awareness, and offer opportunities to engage in social responsibility.

We are proud to be an equal opportunity employer committed to providing employment opportunity regardless of sex, race, creed, color, gender, religion, marital status, domestic partner status, age, national origin or ancestry, physical or mental disability, medical condition, sexual orientation, pregnancy, military or veteran status, citizenship status, and genetic information.

Illumina conducts background checks on applicants for whom a conditional offer of employment has been made. Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable local, state, and federal laws. Background check results may potentially result in the withdrawal of a conditional offer of employment. The background check process and any decisions made as a result shall be made in accordance with all applicable local, state, and federal laws.

Illumina prohibits the use of generative artificial intelligence (AI) in the application and interview process.

If you require accommodation to complete the application or interview process, please contact accommodations@illumina.com. To learn more, visit: https://www.dol.gov/ofccp/regs/compliance/posters/pdf/eeopost.pdf.

The position will be posted until a final candidate is selected or the requisition has a sufficient number of qualified applicants. This role is not eligible for visa sponsorship.

The potential of genomics to change the world is unparalleled. At Illumina, we are leading the way. Our technology empowers researchers and clinicians globally to improve cancer treatments, change the course of diseases, protect the environment, and fight food insecurity. We are made up of dedicated professionals, all-in on creating a better world - and we need every skill set on our path to making genomics useful for all. Learn more about life at Illumina.

Illumina has been made aware of fraudulent schemes in which individuals falsely claim to represent Illumina, our Talent Acquisition team, or our Human Resources organization. Please note: All legitimate Illumina career opportunities are posted on the Illumina Careers website. Illumina will never ask candidates to pay fees, transfer money, purchase equipment, or provide banking information during the recruitment process. Illumina does not extend employment offers without a formal interview process. Official recruiting communications will come from authorized Illumina email addresses (xxxxxx@illumina.com). If you are unsure whether a communication is legitimate, please contact us at candidatehelp@illumina.com before sharing any personal information. Thank you for helping us protect the integrity of the hiring process and your experience.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr Data Engineer
Sr Data Engineer

Illumina • Bengaluru

On-site
INR 4,000,000 - 7,000,000
Associate SAP Developer
Associate SAP Developer

Illumina, Inc. • Bengaluru

Hybrid
INR 2,400,000 - 3,600,000
Data Engineer 2
Data Engineer 2

Illumina, Inc. • Bengaluru

On-site
INR 1,200,000 - 1,800,000
Data Engineer
Data Engineer

Illumina, Inc. • Bengaluru

Hybrid
INR 600,000 - 1,200,000
Director, IT, SAP & Enterprise Applications - Bengaluru
Director, IT, SAP & Enterprise Applications - Bengaluru

Illumina • Bengaluru

On-site
INR 4,500,000 - 9,000,000
Associate Business System Analyst
Associate Business System Analyst

Illumina, Inc. • Bengaluru

Hybrid
INR 600,000 - 1,200,000
Assoc Principal Engineer - Platform Engineering
Assoc Principal Engineer - Platform Engineering

Illumina • Bengaluru

On-site
INR 4,000,000 - 7,000,000
Staff Integration Engineer
Staff Integration Engineer

IN01 Illumina India • Bengaluru

On-site
INR 4,000,000 - 6,000,000
Associate Principal/Senior Manager / Associate Director – Platform Engineering, Cloud & DevOps
Associate Principal/Senior Manager / Associate Director – Platform Engineering, Cloud & DevOps

Illumina, Inc. • India

On-site
INR 3,500,000 - 6,000,000
ERGs programs
Inclusive work environment
Staff Integration Engineer
Staff Integration Engineer

Illumina, Inc. • Bengaluru

On-site
INR 3,000,000 - 5,000,000