Senior Data Engineer- Databricks

Omnicom Global Solutions; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Omnicom Global Solutions in Hyderabad, India seeks a Senior Data Engineer to own substantial Databricks-based data pipelines from ingestion to Gold-layer modeling. You will work with ERP data sources and ensure secure, well-documented end-to-end delivery within an agile pod.

The role requires 5+ years in data engineering with Databricks experience, strong SQL/Python skills, and excellent English communication. Collaboration with onshore/offshore teams is expected.

Qualifications

• 5+ years of data engineering experience with strong Databricks hands-on work.
• 0-3 years of experience in Databricks is insufficient for this senior role.
• Proficient in SQL and Python and end-to-end pipeline ownership.
• Experience in agile sprints and daily standups.

Responsibilities

• Design and build Databricks pipelines across Bronze, Silver, Gold layers.
• Own a defined workstream from intake through delivery with testing and docs.
• Participate in sprint planning and standups, surfacing risks clearly.
• Review code and provide constructive feedback to peers.
• Implement row/column security (RLS/CLS) on owned datasets.
• Support Boomi-based integrations and data validation tasks.
• Collaborate with Lead Engineer to decompose backlog into tasks.
• Proactively identify data quality issues and technical debt.

Skills

Data engineering

Databricks

SQL

Python

Agile

English communication

Tools

Databricks

Delta Lake

Unity Catalog

Boomi

JDE / ERP data sources

Job description

Overview

The Senior Data Engineer owns substantial components of the Databricks platform, including full pipelines from ingestion through Gold-layer modeling. This role is expected to take on tasks with unresolved ambiguity, resolve open questions efficiently, and deliver without close supervision. The Senior Data Engineer identifies data and design issues before they affect downstream work and supports less experienced Data Engineers on the pod in developing the same judgment.

Responsibilities

• Design and build Databricks pipelines (Spark/PySpark, Delta Lake, Unity Catalog) across Bronze, Silver, and Gold layers for financial and operational data sets.
• Own a defined workstream (e.g., a source system or a domain such as project financials) from intake through delivery, including testing and documentation.
• Participate actively in sprint planning and daily standups, surfacing risks and clarifying requirements rather than waiting for direction.
• Review code and output from Data Engineers on the pod and provide direct, constructive feedback.
• Implement and validate row-level and column-level security controls on owned data sets.
• Support Boomi-based integration build and validation work (e.g., JDE-Concur, JDE-UKG) as the pod assumes ownership from the onshore Integration & Middleware team.
• Partner with the Lead Engineer to break down larger backlog items into task-ready work.
• Identify and raise technical debt, data quality issues, or design gaps proactively, rather than deferring to QA.

Qualifications

Required Qualifications

• 5+ years of data engineering experience, including at least 2–3 years of hands‑on Databricks experience (Spark/PySpark, Delta Lake, Unity Catalog).
• Experience building pipelines from financial or ERP source systems — JD Edwards (JDE) directly preferred, with EcoSys or Microsoft Dynamics also relevant — including general ledger, cost, or project financial data.
• Strong SQL and Python skills, with the ability to own a pipeline end-to-end.
• Experience working in agile sprints with daily standups and sprint-based delivery.
• Strong written and verbal English communication skills.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.
• Experience in a construction, EPC, energy, or other asset-heavy industry, particularly with cost-to-complete or earned value data.
• Experience implementing RLS/CLS security models in Unity Catalog.
• Familiarity with Databricks Genie (AI/BI) or Power BI reporting on governed Gold-layer data.
• Prior experience in a hybrid onshore/offshore delivery model.
• Hands‑on experience with Boomi or a comparable iPaaS platform (pipeline build, deployment, monitoring).
• Familiarity with REST/webhook API integration patterns and file‑based integration (EDI, flat file, CSV).

Professional Attributes:

• Defaults to asking questions when requirements are unclear rather than proceeding on assumptions.
• Prioritizes efficient resolution of ambiguity over continuing without clarity.
• Takes end‑to‑end ownership of assigned work and maintains a high quality bar without close supervision.
• Communicates clearly and directly with the Lead Engineer and, when required, the onshore team.