Data Engineer

Hire22.ai; Kerala; In loco; INR 900.000 - 1.500.000; Tempo pieno

Descrizione del lavoro

Hire22.ai is seeking an experienced Data Engineer to design, develop, and maintain scalable data solutions. You will build data pipelines, manage integrations, and ensure data quality and performance across Azure Databricks environments.

The role requires 3–7 years of experience, with strong SQL, Python/PySpark, Delta Lake, and ADLS Gen2 expertise. You will collaborate with US-based stakeholders and participate in CI/CD and automated testing practices.

Competenze

• 3-5 years in data engineering, including 2+ years delivering Azure Databricks solutions.
• Strong SQL, Python/PySpark, Delta Lake, ADLS Gen2.
• Experience with Git/CI/CD, monitoring, and recovery.
• Ability to explain data quality issues to stakeholders.
• Familiarity with one or more Azure data services (ADF, Synapse, Lakehouse).
• Databricks data engineering certification is a plus.

Mansioni

• Pipelines & Ingestion: build database, file, and API ingestion into ADLS Gen2 and Databricks; implement incremental loads and safe reprocessing.
• Transformations & Modeling: develop SQL and PySpark transformations and Delta Lake tables; create bronze/silver/gold layers.
• Reliability & Orchestration: schedule dependencies via Databricks Jobs or ADF; implement validation, monitoring, alerts, retries.
• Security & Delivery: apply Unity Catalog controls and secret management; deliver tests and CI/CD releases via Git-based review.
• Performance & Partnership: tune Spark jobs and queries; document mappings and data quality rules for US stakeholders.
• AI-Assisted Engineering: leverage AI tools for SQL/PySpark development, testing, and documentation while protecting client data.

Conoscenze

SQL

Python

PySpark

Delta Lake

ADLS Gen2

ETL/ELT

Data modeling

Git/CI/CD

Spark tuning

Strumenti

Azure Databricks

Azure Data Factory

Unity Catalog

Azure Key Vault

Terraform

Microsoft Fabric

Azure Synapse Analytics

Descrizione del lavoro

Our client, is looking for an experienced Data Engineer to design, develop, and maintain scalable data solutions, build efficient data pipelines, manage data integration processes, and ensure data quality and performance.

Experience 3-7 years

Job Description

RESPONSIBILITIES

• Pipelines & Ingestion: Independently build database, file, and API ingestion into ADLS Gen2 and Azure Databricks; implement incremental loads, change data capture where appropriate, and safe reprocessing.
• Transformations & Modeling: Develop SQL and Python/PySpark transformations and Delta Lake tables; build bronze, silver, and gold layers and analytics-ready models with clear data contracts.
• Reliability & Orchestration: Schedule dependencies through Databricks Jobs or Azure Data Factory; implement validation, monitoring, alerts, retries, and recovery procedures to meet agreed data freshness targets.
• Security & Delivery: Apply Unity Catalog access controls and lineage, use Azure Key Vault or approved secret management, and deliver automated tests and CI/CD releases through Git-based review.
• Performance & Partnership: Tune Spark jobs, queries, and table layout for performance and cost; gather requirements with U.S.-based stakeholders and document mappings, quality rules, and operational runbooks.
• AI-Assisted Engineering: Use approved AI tools to assist with SQL/Python development, testing, troubleshooting, and documentation; validate generated code and outputs and protect client data.

REQUIREMENTS

Experience ranges are indicative; equivalent demonstrated capability will be considered.

• Required - Experience: Typical range: 3-5 years in data engineering, including 2+ years delivering Azure Databricks solutions. Demonstrate independent pipeline delivery from requirements through deployment and production support.
• Required - Core Skills: Strong SQL, Python/PySpark, Delta Lake, ADLS Gen2, ETL/ELT, and data modeling; hands-on incremental ingestion, orchestration, automated testing, and Spark performance tuning.
• Required - Delivery: Practical Unity Catalog, secure Azure access, Git/CI/CD, monitoring, and recovery experience; ability to manage commitments and explain data quality issues to U.S.-based stakeholders.
• Preferred - Additional Platforms: Experience with one or more: Azure Data Factory, Event Hubs, Structured Streaming, Lakeflow pipelines, Terraform, Microsoft Fabric, or Azure Synapse Analytics. Depth across all tools is not expected.
• Preferred - Integration & Certification: Power BI integration or analytics data modeling is a plus. Relevant Databricks data engineering certification is preferred; equivalent practical experience is welcomed.