A complete application in a minute
Tailored resume and cover letter, ready to send.
Calidad Infotech in Ahmedabad is seeking a hands-on Data Engineer to design and run batch and real-time data pipelines on Databricks, using PySpark and Structured Streaming on the Azure stack. The role is build‑oriented, requiring production PySpark code and end‑to‑end ownership of streaming pipelines.
You will optimize Delta Lake tables, tune Spark jobs, model data for analytics, orchestrate workflows, and collaborate with data architects and business stakeholders to translate requirements into
We need a hands-on Data Engineer with 5-7 years of experience to build and run batch and real-time data pipelines on Databricks. The work centres on PySpark and Structured Streaming, preferably on the Azure data stack. This is a build role, not a support or reporting role. Candidates must have written production PySpark code and owned streaming pipelines end to end.
Design, build and maintain batch and streaming pipelines on Databricks using PySpark.
Develop real-time ingestion and processing with Spark Structured Streaming, including checkpointing, watermarking and late-data handling.
Build and manage Delta Lake tables across bronze, silver and gold layers.
Tune Spark jobs for performance and cost: partitioning, joins, caching, cluster sizing.
Model data for analytics and downstream consumers.
Orchestrate and schedule workflows, and monitor pipelines for failures and data quality issues.
Put pipelines through version control, code review and CI/CD.
Work with data architects, analysts and business stakeholders to turn requirements into working pipelines.
5-7 years of relevant, hands-on data engineering experience. Databricks: notebooks, jobs and workflows, clusters, Delta Lake. PySpark: DataFrame API, Spark SQL, UDFs, performance tuning. Structured Streaming: production streaming pipelines, with stateful processing, triggers, checkpoints and watermarks.
Core data engineering: ETL/ELT design, data modelling, data warehousing and lakehouse concepts, incremental loads and change data capture, data quality checks.
Strong SQL and Python.
Git-based development.
Azure Databricks and Azure Data Lake Storage Gen2.
Azure Data Factory or Databricks Workflows for orchestration.
Azure Event Hubs or Kafka as streaming sources.
Unity Catalog, Delta Live Tables and Auto Loader.
Azure Synapse Analytics or Microsoft Fabric.
Azure DevOps or GitHub Actions for CI/CD.
Databricks or Azure data engineering certification.