Data Engineer

Techcarrot Fz-llc; Chennai District; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Techcarrot Fz-llc is seeking an experienced Data Engineer to translate business and technical requirements into robust data solutions on the Azure Data Platform. The role covers end-to-end ownership of data pipelines, from design through deployment, using ADF and Databricks, with PySpark, SQL and Delta Lake to handle diverse data sources.

Responsibilities include building scalable data ingestion frameworks, optimizing Spark workloads, implementing data quality checks, and enabling CI/CD with

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering or related discipline.
• 4+ years of hands-on experience with Microsoft Azure Data Platform technologies.
• Strong hands-on experience with Azure Data Factory, Databricks, SQL and Python/PySpark.
• Experience delivering at least one end-to-end Azure data engineering project.
• Good understanding of data warehousing, data lake and modern data platform concepts.

Responsibilities

• Independently understand business and technical requirements and translate them into scalable data engineering solutions.
• Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
• Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
• Build scalable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
• Develop data transformation logic using Python, PySpark and SQL.
• Work with structured, semi-structured, and unstructured data sources.
• Design batch and near-real-time data processing solutions where required.
• Develop optimized Databricks workloads using Apache Spark and Delta Lake.
• Perform data analysis, profiling and validation for quality and consistency.
• Implement data quality checks, error handling, logging and monitoring.
• Troubleshoot pipeline failures, performance issues and data discrepancies.
• Optimize SQL queries, Spark jobs and pipelines for performance.
• Create reusable components to improve development efficiency.
• Participate in solution design discussions and provide implementation recommendations.
• Implement CI/CD for data components using Azure DevOps.
• Deploy and manage ADF, Databricks across environments.

Skills

Azure Data Factory

Azure Databricks

SQL

Python/PySpark

Delta Lake

Azure Synapse

Azure Data Lake Storage Gen2

CI/CD (Azure DevOps)

Databricks Spark

Data Pipelines

Education

Bachelor's degree in CS/IT/Engineering

Tools

Apache Spark

Power BI

Job description

Responsibilities

• Independently understand business and technical requirements and translate them into scalable data engineering solutions.
• Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
• Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
• Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
• Develop data transformation and processing logic using Python, PySpark and SQL.
• Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
• Design and implement batch and, where required, near-real-time data processing solutions.
• Develop optimized Databricks workloads using Apache Spark and Delta Lake.
• Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
• Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
• Troubleshoot data pipeline failures, performance issues and data discrepancies independently.
• Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
• Create reusable components and frameworks to improve development efficiency across projects.
• Participate in solution design and technical discussions and provide recommendations on implementation approaches.
• Implement CI/CD processes for data engineering components using Azure DevOps.
• Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
• Prepare appropriate technical design, data mapping and operational documentation.
• Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
• Support production deployments and troubleshoot post-production issues when required.

Requirements

• Strong hands-on experience with:
  • Azure Data Platform
    • Azure Data Factory (ADF) pipelines, datasets, linked services, triggers, parameterization and reusable frameworks
    • Azure Databricks
    • Azure Data Lake Storage Gen2 (ADLS Gen2)
    • Azure Synapse Analytics / Azure SQL
    • Azure Key Vault
    • Azure Monitor / Log Analytics or equivalent monitoring capabilities
    • Azure Event Hubs exposure/experience is preferred
  • Databricks & Data Processing
    • Strong hands-on experience with Databricks and Apache Spark
    • Strong PySpark development skills
    • Experience with Delta Lake / Delta tables
    • Understanding of Medallion or similar layered data architectures
    • Experience implementing incremental and full-load processing patterns
    • Ability to troubleshoot and optimize Spark workloads
    • Experience with Databricks Workflows/Jobs
    • Knowledge of Unity Catalog is preferred
  • SQL & Python
    • Strong SQL development skills
    • Ability to write and optimize complex SQL queries
    • Experience with joins, CTEs, window functions and analytical queries
    • Strong working knowledge of Python
    • Ability to develop reusable Python/PySpark modules and utilities
  • DevOps / CI-CD
    • Hands-on experience with Azure DevOps
    • Git-based source control
    • Branching and code-management practices
    • CI/CD implementation for Azure Data Factory and Databricks
    • Experience deploying solutions across multiple environments

  Qualifications & Experience

  • 46 years of overall experience in Data Engineering / Data Platform development.
  • Preferably 4+ years of hands-on experience with Microsoft Azure Data Platform technologies.
  • Strong hands-on experience with Azure Data Factory, Databricks, SQL and Python/PySpark.
  • Bachelors degree in Computer Science, Information Technology, Engineering or a related discipline.
  • Experience delivering at least one or more end-to-end Azure data engineering projects.
  • Good understanding of data warehousing, data lake and modern data platform concepts.
  • Strong analytical and problem-solving skills.
  • Ability to work independently and manage multiple technical activities.
  • Good written and verbal communication skills.

  Preferred Skills

  • Microsoft Azure / Databricks certifications.
  • Experience with Unity Catalog and Databricks governance.
  • Experience with REST API-based data ingestion.
  • Knowledge of dimensional modelling and data warehouse concepts.
  • Experience working with Power BI or downstream analytics platforms.
  • Exposure to streaming/event-driven data processing.
  • Experience working in enterprise-scale Azure environments.

  Job Details

  • Job Type: Full time
  • Job Opening Name: Data Engineer
  • State: Telangana
  • Country: India
  • Location: Hyderabad, Noida/ Chennai