Data Engineer

Calidad Infotech

Ahmedabad

On-site

INR 1,600,000 - 2,400,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Calidad Infotech in Ahmedabad is seeking a hands-on Data Engineer to design and run batch and real-time data pipelines on Databricks, using PySpark and Structured Streaming on the Azure stack. The role is build‑oriented, requiring production PySpark code and end‑to‑end ownership of streaming pipelines.

You will optimize Delta Lake tables, tune Spark jobs, model data for analytics, orchestrate workflows, and collaborate with data architects and business stakeholders to translate requirements into

Qualifications

  • 5-7 years of hands-on data engineering experience.
  • Databricks notebooks, jobs and workflows, clusters, Delta Lake.
  • PySpark DataFrame API, Spark SQL, UDFs, performance tuning.
  • Structured Streaming pipelines with stateful processing, triggers, checkpoints, watermarks.
  • Core data engineering: ETL/ELT design, data modelling, data warehousing and lakehouse concepts.
  • Strong SQL and Python.
  • Git-based development.

Responsibilities

  • Design, build and maintain batch and streaming pipelines on Databricks using PySpark.
  • Develop real-time ingestion and processing with Spark Structured Streaming, including checkpointing, watermarking and late-data handling.
  • Build and manage Delta Lake tables across bronze, silver and gold layers.
  • Tune Spark jobs for performance and cost: partitioning, joins, caching, cluster sizing.
  • Model data for analytics and downstream consumers.
  • Orchestrate and schedule workflows, and monitor pipelines for failures and data quality issues.
  • Put pipelines through version control, code review and CI/CD.
  • Work with data architects, analysts and business stakeholders to turn requirements into working pipelines.

Skills

Databricks
PySpark
Structured Streaming
Delta Lake
SQL
Python
Git

Tools

Azure Databricks
Azure Data Factory
Databricks Workflows
Azure Synapse
Delta Live Tables
Auto Loader
Unity Catalog
CI/CD (GitHub Actions/DevOps)

Job description

About the role

We need a hands-on Data Engineer with 5-7 years of experience to build and run batch and real-time data pipelines on Databricks. The work centres on PySpark and Structured Streaming, preferably on the Azure data stack. This is a build role, not a support or reporting role. Candidates must have written production PySpark code and owned streaming pipelines end to end.

Key responsibilities

Design, build and maintain batch and streaming pipelines on Databricks using PySpark.

Develop real-time ingestion and processing with Spark Structured Streaming, including checkpointing, watermarking and late-data handling.

Build and manage Delta Lake tables across bronze, silver and gold layers.

Tune Spark jobs for performance and cost: partitioning, joins, caching, cluster sizing.

Model data for analytics and downstream consumers.

Orchestrate and schedule workflows, and monitor pipelines for failures and data quality issues.

Put pipelines through version control, code review and CI/CD.

Work with data architects, analysts and business stakeholders to turn requirements into working pipelines.

Must-have skills

5-7 years of relevant, hands-on data engineering experience. Databricks: notebooks, jobs and workflows, clusters, Delta Lake. PySpark: DataFrame API, Spark SQL, UDFs, performance tuning. Structured Streaming: production streaming pipelines, with stateful processing, triggers, checkpoints and watermarks.

Core data engineering: ETL/ELT design, data modelling, data warehousing and lakehouse concepts, incremental loads and change data capture, data quality checks.

Strong SQL and Python.

Git-based development.

Preferred skills (Azure stack)

Azure Databricks and Azure Data Lake Storage Gen2.

Azure Data Factory or Databricks Workflows for orchestration.

Azure Event Hubs or Kafka as streaming sources.

Unity Catalog, Delta Live Tables and Auto Loader.

Azure Synapse Analytics or Microsoft Fabric.

Azure DevOps or GitHub Actions for CI/CD.

Databricks or Azure data engineering certification.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks data engineer
Databricks data engineer

NYX TECH SOLUTION • Pune District

On-site
INR 1,200,000 - 2,100,000
Azure Data Engineer
Azure Data Engineer

InfoBeans • Indore District, Pune District

On-site
INR 3,000,000 - 5,400,000
Databricks Data Engineer
Databricks Data Engineer

Infosys • Pune District, Bengaluru

On-site
INR 900,000 - 1,400,000
Data Engineer (Azure & Databricks)
Data Engineer (Azure & Databricks)

Lufthansa Technik Services India • Bengaluru

On-site
INR 1,200,000 - 1,500,000
Data Architect with Databricks
Data Architect with Databricks

Unique Erp Technologies • Hyderabad

On-site
INR 1,800,000 - 4,000,000
Senior Data Engineer
Senior Data Engineer

CloudBoson • Dadri

On-site
INR 1,800,000 - 3,000,000
Data Engineer
Data Engineer

Intact India • Gurugram District

On-site
INR 1,200,000 - 2,400,000
Azure Databricks Data Engineer
Azure Databricks Data Engineer

WTW Global Delivery And Solutions • Kolkata District

Hybrid
INR 1,200,000 - 2,600,000
Data Engineer (Azure & Databricks)
Data Engineer (Azure & Databricks)

Lufthansa Technik Services India Pvt Ltd • Bengaluru

On-site
INR 1,000,000 - 1,500,000
Databricks Data Engineer
Databricks Data Engineer

Emergys • Pune District

On-site
INR 1,400,000 - 2,100,000