Data Engineer- Databricks

Annalect, Omnicom Media Group India Private Limited.; Hyderabad; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Annalect in Hyderabad seeks a Data Engineer to build and test Databricks data pipelines, covering ingestion, transformation, and gold-layer modeling within sprints. Early-to-mid career engineers will grow rapidly by tackling clear requirements and raising blockers promptly.

You will work with PySpark/SQL, Delta Lake, and adapt data quality checks, participate in daily standups, and adhere to data security standards. Prior exposure to Boomi and ERP data is a plus.

Qualifications

• 2-4 years of data engineering experience
• Hands-on Spark/PySpark and SQL
• Databricks experience with Delta Lake, notebooks, jobs/workflows
• Understanding of data warehousing/lakehouse concepts
• Experience in an agile team with daily standups and sprint deliveries
• Strong written and verbal English communication

Responsibilities

• Build and test Databricks pipelines (PySpark/SQL, Delta Lake) for ingestion, transformation, and modeling tasks during sprint planning.
• Write and maintain data quality checks and tests for pipelines.
• Participate in daily standups and report progress, raising blockers promptly.
• Follow security models and coding standards set by senior engineers.
• Support integration testing for Boomi-based data flows and reconciliations.
• Apply feedback from code reviews to future work.
• Document pipelines clearly for ownership transfer.

Skills

Spark/PySpark

SQL

Databricks

Agile

English communication

Tools

Delta Lake

notebooks / jobs

Job description

Overview

The Data Engineer builds and tests data pipelines in Databricks, covering ingestion, transformation, and Gold-layer modeling, against tasks assigned by the Lead Engineer at the start of each sprint and refined daily. This role suits an early-to-mid career engineer seeking rapid growth. The Data Engineer is expected to ask questions when requirements are unclear and to raise blockers promptly rather than delay progress.

Responsibilities

• Build and test Databricks pipelines (PySpark/SQL, Delta Lake) for ingestion, transformation, and modeling tasks assigned during sprint planning.
• Write and maintain data quality checks and tests for pipelines built.
• Participate in daily standups, reporting progress accurately, asking questions when requirements are unclear, and flagging blockers promptly.
• Follow the platform's data security model (RLS/CLS) and coding/documentation standards set by the Lead and Senior Engineers.
• Support integration testing and reconciliation checks for Boomi-based data flows (e.g., JDE-Concur, JDE-UKG) as assigned.
• Apply feedback from code reviews to subsequent work.
• Document pipelines clearly enough for another engineer to assume ownership.

Qualifications

Required Qualifications

• 2-4 years of data engineering experience, including hands-on work with Spark/PySpark and SQL.
• Hands-on Databricks experience (Delta Lake, notebooks, jobs/workflows).
• Working understanding of data warehousing/lakehouse concepts (Bronze/Silver/Gold or comparable layered modeling).
• Experience working in an agile team with daily standups and sprint-based delivery.
• Strong written and verbal English communication skills.

Preferred Qualifications

• Databricks Certified Data Engineer Associate.
• Exposure to financial or ERP source data - JD Edwards (JDE), EcoSys, or Microsoft Dynamics - or general ledger/cost data.
• Exposure to an EPC, construction, energy, or other asset-heavy industry.
• Familiarity with Unity Catalog-based security (RLS/CLS) or Power BI/Genie reporting on governed data.
• Exposure to Boomi or another iPaaS/integration middleware platform.

Professional Attributes:

• Asks questions early when a task is unclear rather than proceeding on assumptions.
• Demonstrates visible progress daily rather than only at sprint boundaries.
• Applies feedback from the Lead and Senior Engineers quickly and consistently.
• Seeks increasing ownership and more complex problems over time.