Azure Data Engineer

Lavance LLC; New York (NY); On-site; USD 120,000 - 150,000; Full time

Job summary

Lavance LLC in New York, NY is seeking a Data Engineer to design, develop, and maintain scalable data pipelines using Azure services. You will build ETL/ELT processes with Azure Data Factory and Databricks, develop data transformation logic with PySpark and Python, and manage Delta Lake and Lakehouse architectures.

You will optimize SQL queries and Spark workloads, integrate data from APIs and databases, ensure data quality and governance, and collaborate with analysts and business stakeholders

Qualifications

• Experience designing scalable data pipelines on Azure.
• Strong SQL and Spark performance tuning skills.
• Experience with Delta Lake and Lakehouse architectures.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure services.
• Build ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
• Develop data transformation logic using PySpark, Spark SQL, and Python.
• Manage Delta Lake and Lakehouse architectures.
• Optimize SQL queries and Spark workloads for performance.
• Integrate data from APIs, databases, and third-party applications.
• Implement data quality, validation, and monitoring processes.
• Collaborate with analysts, data scientists, and business stakeholders.
• Ensure data security, governance, and compliance.

Tools

Azure Databricks

Azure Data Factory (ADF)

Azure Data Lake Storage (ADLS Gen2)

Azure Synapse Analytics

PySpark

Python

SQL

Delta Lake

Git

Power BI

REST APIs

Job description

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure services.


• Build ETL/ELT pipelines using Azure Data Factory and Azure Databricks.


• Develop data transformation logic using PySpark, Spark SQL, and Python.


• Manage Delta Lake and Lakehouse architectures.


• Optimize SQL queries and Spark workloads for performance.


• Integrate data from APIs, databases, and third-party applications.


• Implement data quality, validation, and monitoring processes.


• Collaborate with analysts, data scientists, and business stakeholders.


• Ensure data security, governance, and compliance.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure services.


• Build ETL/ELT pipelines using Azure Data Factory and Azure Databricks.


• Develop data transformation logic using PySpark, Spark SQL, and Python.


• Manage Delta Lake and Lakehouse architectures.


• Optimize SQL queries and Spark workloads for performance.


• Integrate data from APIs, databases, and third-party applications.


• Implement data quality, validation, and monitoring processes.


• Collaborate with analysts, data scientists, and business stakeholders.


• Implement CI/CD pipelines using Azure DevOps.


• Ensure data security, governance, and compliance.

Required Skills

• Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, PySpark, Python, SQL, Delta Lake, Git, Power BI, REST APIs