Data Engineer - Synapse Analytics

Forward Eye Technologies; Pune District; On-site; INR 1,200,000 - 2,200,000; Full time

Job summary

Forward Eye Technologies is seeking an experienced Data Engineer to design, build, and optimize ETL pipelines using Azure Data Factory and Databricks. You will develop PySpark workflows, automate data processing with Python, and model data for Azure storage and analytics platforms.

You will collaborate with data scientists and analysts to translate requirements into scalable cloud data solutions, ensuring performance, security, and governance best practices across the Azure ecosystem.

Qualifications

• Minimum of 5 years of experience in data engineering or related roles.
• Strong hands-on experience with Azure Data Factory for data integration and orchestration.
• Proven expertise in using Databricks for data processing and analytics.
• Proficiency in Python scripting, particularly in data processing and automation.

Responsibilities

• Design, develop, and maintain ETL pipelines with Azure Data Factory to integrate data into data lake or data warehouse.
• Implement data workflows and orchestrate data movement and transformation processes within the ADF environment.
• Utilize Databricks to develop and manage scalable data processing pipelines.
• Write and optimize PySpark scripts for data transformation, aggregation, and analysis within Databricks.
• Perform real-time and batch data processing on large datasets to support analytics and reporting needs.
• Develop Python scripts to automate data processing tasks, enhance data pipelines, and integrate with other Azure services.
• Implement data validation and error-handling mechanisms within Python scripts to ensure data accuracy and reliability.
• Design and implement data models that support efficient querying, storage, and retrieval of data in the Azure ecosystem.
• Optimize data storage solutions within Azure, including Azure Data Lake, Azure SQL Data Warehouse, and other related services.
• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Provide technical expertise and guidance on best practices for data engineering and cloud-based solutions.
• Monitor, troubleshoot, and optimize data pipelines for performance, scalability, and cost efficiency.
• Implement best practices for data security, governance, and compliance within the Azure environment.

Skills

Python scripting

Problem-solving

Communication

Independent work

Tools

Azure Data Factory

Databricks

PySpark

Azure Data Lake

Azure SQL Data Warehouse

Azure Synapse Analytics

Git

Apache Airflow

Job description

Job Responsibilities

Data Integration and Orchestration

• Design, develop, and maintain ETL pipelines using Azure Data Factory (ADF) to integrate data from various sources into the data lake or data warehouse.
• Implement data workflows and orchestrate data movement and transformation processes within the ADF environment.

Data Processing and Analytics

• Utilize Databricks to develop and manage scalable data processing pipelines.
• Write and optimize PySpark scripts for data transformation, aggregation, and analysis within Databricks.
• Perform real-time and batch data processing on large datasets to support analytics and reporting needs.

Python Scripting and Automation

• Develop Python scripts to automate data processing tasks, enhance data pipelines, and integrate with other Azure services.
• Implement data validation and error-handling mechanisms within Python scripts to ensure data accuracy and reliability.

Data Modeling and Storage

• Design and implement data models that support efficient querying, storage, and retrieval of data in the Azure ecosystem.
• Optimize data storage solutions within Azure, including Azure Data Lake, Azure SQL Data Warehouse, and other related services.

Collaboration and Stakeholder Engagement

• Work closely with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Provide technical expertise and guidance on best practices for data engineering and cloud-based solutions.

Performance Tuning and Optimization

• Monitor, troubleshoot, and optimize data pipelines for performance, scalability, and cost efficiency.
• Implement best practices for data security, governance, and compliance within the Azure environment.

Required Qualifications

Experience

• Minimum of 5 years of experience in data engineering or related roles.
• Strong hands-on experience with Azure Data Factory for data integration and orchestration.
• Proven expertise in using Databricks for data processing and analytics.
• Proficiency in Python scripting, particularly in the context of data processing and automation.

Technical Skills

• In-depth knowledge of PySpark for large-scale data processing.
• Familiarity with Azure services such as Azure Data Lake, Azure SQL Data Warehouse, and Azure Synapse Analytics.
• Experience with data modeling, ETL processes, and data architecture design in cloud environments.
• Strong understanding of CI/CD pipelines and version control using Git.

Soft Skills

• Excellent problem-solving skills with the ability to debug and resolve complex data engineering challenges.
• Strong communication skills for effective collaboration with technical and non-technical stakeholders.
• Ability to work independently and manage multiple priorities in a fast-paced environment.

Preferred Qualifications

• Experience with data orchestration tools such as Apache Airflow.
• Familiarity with other cloud platforms (e.g., AWS, GCP) is a plus.
• Understanding of data governance and security practices in the cloud.

Keywords

• Azure Data Factory (ADF)
• Databricks
• PySpark
• Python
• Data Engineering
• Cloud Data Solutions
• ETL Pipelines
• Data Integration
• Data Modeling
• Azure Data Lake.

Location

Hyderabad , Ahmedabad , pune , chennai , kolkata.