Azure Data Engineer

Signature IT World Inc; New Jersey; Presencial; USD 120 000 - 150 000; Tempo integral

Resumo da oferta

Signature IT World Inc. is seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data solutions using Microsoft Azure.

The ideal candidate will have strong expertise in Azure Data Factory, Azure Databricks, PySpark, SQL, and data warehousing concepts. The role involves building ETL/ELT pipelines, optimizing data workflows, and ensuring data quality and security across environments.

Qualificações

• Experience designing and implementing large-scale ETL/ELT pipelines on Azure.
• Strong SQL skills and data warehousing concepts.
• Knowledge of Delta Lake and lakehouse architectures.

Responsabilidades

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory.
• Build and optimize data processing workflows with Azure Databricks and PySpark.
• Develop complex SQL queries, stored procedures, and data transformation logic.
• Integrate data from databases, APIs, files, and cloud storage.
• Implement data lake and lakehouse solutions with Azure Data Lake Storage Gen2.
• Work with Delta Lake and efficient data processing strategies.
• Ensure data quality, security, and reliability across pipelines.
• Optimize pipelines and SQL queries for performance and cost.
• Implement monitoring, logging, and error-handling mechanisms.
• Collaborate with data architects, analysts, and stakeholders to deliver data solutions.
• Support CI/CD pipelines and deployment automation for data workloads.
• Follow data governance, access control, and cloud security best practices.

Conhecimentos

ETL/ELT design

SQL proficiency

Data modeling

Cloud data engineering

Ferramentas

Azure Data Factory

Azure Databricks

PySpark

Azure Data Lake Storage Gen2

Delta Lake

Lakehouse architecture

Terraform

Descrição da oferta de emprego

We are looking for an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data solutions using Microsoft Azure. The ideal candidate will have strong expertise in Azure Data Factory, Azure Databricks, PySpark, SQL, and data warehousing concepts.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory.
• Build and optimize data processing workflows using Azure Databricks and PySpark.
• Develop complex SQL queries, stored procedures, and data transformation logic.
• Integrate data from multiple sources, including databases, APIs, files, and cloud storage.
• Implement data lake and lakehouse solutions using Azure Data Lake Storage Gen2.
• Work with Delta Lake and implement efficient data processing and storage strategies.
• Ensure data quality, accuracy, security, and reliability across data pipelines.
• Optimize data pipelines and SQL queries for performance and cost efficiency.
• Implement monitoring, logging, and error-handling mechanisms.
• Collaborate with data architects, analysts, and business stakeholders to deliver data solutions.
• Support CI/CD pipelines and deployment automation for data engineering workloads.
• Follow data governance, access control, and cloud security best practices.

Required Skills

• Strong experience with Microsoft Azure data services.
• Hands-on experience with Azure Data Factory (ADF).
• Experience with Azure Databricks and PySpark.
• Strong SQL skills and database concepts.
• Experience with Azure Data Lake Storage Gen2 (ADLS).
• Knowledge of ETL/ELT processes and data pipeline development.
• Experience with data modeling, data warehousing, and data transformation.
• Understanding of Delta Lake and lakehouse architecture.
• Experience with data pipeline monitoring, troubleshooting, and optimization.

Nice-to-Have Skills

• Experience with Azure Synapse Analytics or Microsoft Fabric.
• Knowledge of Python and Scala.
• Experience with Azure DevOps, Git, and CI/CD pipelines.
• Familiarity with Terraform or other infrastructure-as-code tools.
• Understanding of data governance, security, and compliance.
• Experience with streaming technologies such as Azure Event Hubs or Kafka.

Experience Requirements

• 10+ years of experience in data engineering.
• Strong hands-on experience developing and supporting Azure-based data solutions.
• Experience working on enterprise-scale data integration and transformation projects.