Data Engineer

EZSpace Ventures; Kerala; On-site; INR 900,000 - 1,500,000; Full time

Job summary

EZSpace Ventures is seeking an experienced Data Engineer in Kerala, India to design, develop, and maintain scalable data solutions. You will build robust data pipelines, manage ingestion into ADLS Gen2, and implement data quality and performance targets across bronze, silver, and gold data layers.

You will craft SQL and PySpark transformations, ensure security via Unity Catalog and secret management, and collaborate with U.S.-based stakeholders.

Qualifications

• .3-5 years in data engineering, with 2+ years delivering Azure Databricks solutions.
• .

Responsibilities

• Ingest data into ADLS Gen2 and Azure Databricks with database, file, and API ingestion.
• Develop SQL and Python/PySpark transformations and Delta Lake tables for bronze/silver/gold layers.
• Schedule and manage jobs via Databricks Jobs or Azure Data Factory; implement validation, monitoring, retries, and data freshness.
• Apply Unity Catalog access controls and secret management; deliver tests and CI/CD releases via Git-based review.
• Tune Spark jobs/queries and document data mappings, quality rules, and runbooks to meet performance goals.
• Leverage AI tools to assist SQL/PySpark development while validating outputs and protecting client data.

Skills

SQL

Python/PySpark

ETL/ELT

Data Modeling

Spark performance tuning

CI/CD

Power BI integration

Education

Databricks Data Engineering Certification

Tools

Azure Databricks

ADLS Gen2

Azure Data Factory

Git

Delta Lake

Terraform

Microsoft Fabric

Azure Synapse Analytics

Job description

Our client, is looking for an experienced Data Engineer to design, develop, and maintain scalable data solutions, build efficient data pipelines, manage data integration processes, and ensure data quality and performance.

Responsibilities

Job Description

• Pipelines & Ingestion: Independently build database, file, and API ingestion into ADLS Gen2 and Azure Databricks; implement incremental loads, change data capture where appropriate, and safe reprocessing.
• Transformations & Modeling: Develop SQL and Python/PySpark transformations and Delta Lake tables; build bronze, silver, and gold layers and analytics-ready models with clear data contracts.
• Reliability & Orchestration: Schedule dependencies through Databricks Jobs or Azure Data Factory; implement validation, monitoring, alerts, retries, and recovery procedures to meet agreed data freshness targets.
• Security & Delivery: Apply Unity Catalog access controls and lineage, use Azure Key Vault or approved secret management, and deliver automated tests and CI/CD releases through Git-based review.
• Performance & Partnership: Tune Spark jobs, queries, and table layout for performance and cost; gather requirements with U.S.-based stakeholders and document mappings, quality rules, and operational runbooks.
• AI-Assisted Engineering: Use approved AI tools to assist with SQL/Python development, testing, troubleshooting, and documentation; validate generated code and outputs and protect client data.

Requirements

• Required - Experience: Typical range: 3-5 years in data engineering, including 2+ years delivering Azure Databricks solutions. Demonstrate independent pipeline delivery from requirements through deployment and production support.
• Required - Core Skills: Strong SQL, Python/PySpark, Delta Lake, ADLS Gen2, ETL/ELT, and data modeling; hands‑on incremental ingestion, orchestration, automated testing, and Spark performance tuning.
• Required - Delivery: Practical Unity Catalog, secure Azure access, Git/CI/CD, monitoring, and recovery experience; ability to manage commitments and explain data quality issues to U.S.-based stakeholders.
• Preferred - Additional Platforms: Experience with one or more: Azure Data Factory, Event Hubs, Structured Streaming, Lakeflow pipelines, Terraform, Microsoft Fabric, or Azure Synapse Analytics. Depth across all tools is not expected.
• Preferred - Integration & Certification: Power BI integration or analytics data modeling is a plus. Relevant Databricks data engineering certification is preferred; equivalent practical experience is welcomed.

Skills: python/pyspark,ci/cd,databricks data engineering certification,git,sql,automated testing,power bi integration,data modeling,etl/elt,azure data factory,spark performance tuning,azure databricks