Data Engineer

techcarrot; Chennai; In loco; INR 1.500.000 - 2.100.000; Tempo pieno

Descrizione del lavoro

techcarrot is seeking a Data Engineer to build scalable data solutions on Azure, integrating SAP data, and delivering trusted datasets for reporting and BI. You will develop data pipelines with ADF, transform data in Databricks, and support Bronze/Silver/Gold data layers to enable Power BI reporting.

You will collaborate with BIA, Data Architects, and Analysts to meet data requirements, ensure governance, and optimize performance and cost.

Competenze

• 5–8 years overall experience, 3–6 years in Data Engineering.
• 2+ years of experience with Azure Databricks.
• 2+ years of experience with Azure Data Factory.
• Experience building enterprise-scale cloud data pipelines.

Mansioni

• Ingest data from SAP, databases, APIs, flat files, and cloud apps using Azure Data Factory.
• Develop PySpark and SQL transformations in Databricks; maintain Bronze/Silver/Gold datasets.
• Support enterprise data models and CDM; ensure data consistency across domains.

Conoscenze

SAP integration

REST API integrations

Event-driven architectures

Unity Catalog

Microsoft Fabric

Retail or Automotive domain experience

Formazione

Bachelor's degree in Computer Science, IT, Engineering

Strumenti

Azure Data Factory

Azure Databricks

Python

PySpark

SQL

Azure Data Lake Storage

Delta Lake

Git

Azure DevOps

Power BI

Descrizione del lavoro

We are looking for a Data Engineer to build, enhance, and support scalable data solutions on Azure. The role involves developing data pipelines, integrating SAP and non-SAP source systems, implementing data transformations in Databricks, and delivering trusted datasets for reporting, analytics, and business intelligence.

Key responsibilities

Data ingestion & integration

• Develop and maintain data ingestion pipelines using Azure Data Factory (ADF).
• Integrate data from SAP, databases, APIs, flat files, and cloud applications.
• Implement incremental and CDC-based data loading patterns.
• Troubleshoot data integration and pipeline issues.

Data transformation & processing

• Develop PySpark and SQL-based transformations in Azure Databricks.
• Build and maintain Bronze, Silver, and Gold layer datasets.
• Cleanse, validate, and enrich data to meet business requirements.
• Optimize jobs for performance, scalability, and cost efficiency.

Data warehousing & modeling

• Support implementation of enterprise data models and Common Data Models (CDM).
• Develop curated datasets for reporting and analytics.
• Ensure data consistency across multiple business domains.

Analytics enablement

• Deliver trusted datasets for Power BI and self-service analytics.
• Work closely with Business Analysts, BI Developers, and Data Architects to understand data requirements.
• Support report and dashboard performance improvements.

Governance & quality

• Implement data quality checks and validation frameworks.
• Adhere to data governance standards and security policies.
• Document data flows, business rules, and technical processes.

Support & operations

• Monitor scheduled jobs and data pipelines.
• Investigate and resolve production incidents.
• Perform root cause analysis and implement corrective actions.
• Participate in deployment and release activities.

Requirements

Preferred skills

• SAP ECC / SAP S/4HANA data integration
• REST API integrations
• Event-driven architectures
• Unity Catalog
• Microsoft Fabric
• Retail or Automotive domain experience

Experience

• Overall 5-8 years of experience , 3-6 years of Data Engineering experience.
• 2+ years of experience with Azure Databricks.
• 2+ years of experience with Azure Data Factory.
• Experience building enterprise-scale cloud data pipelines.

Technical skills

• Azure Data Factory (ADF)
• Azure Databricks
• Python and PySpark
• SQL
• Azure Data Lake Storage (ADLS)
• Delta Lake

Data engineering concepts

• ETL / ELT development
• Data warehousing
• Medallion Architecture
• Incremental loading
• Data modeling
• Data quality management

Reporting & analytics

• Power BI
• Dataset optimization
• Dimensional modeling

Version control & devops

• Git
• Azure DevOps
• CI/CD concepts

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.