Azure Data Architect with Databricks

Arkhya Tech. Inc.; New Jersey; On-site; USD 140,000 - 190,000; Full time

Job summary

Arkhya Tech. Inc. in New Jersey is seeking an experienced Azure Data Architect to design, implement, and optimize large-scale data pipelines.

You will work with Databricks, Delta Lake, and Unity Catalog to ensure scalable, secure data flows that support dashboards and analytics across the business. The role requires strong SQL and Python skills, deep knowledge of Spark/PySpark, and collaboration with multiple teams to translate complex requirements into robust data solutions while focusing on

Qualifications

• Experience designing and implementing large-scale data pipelines.
• Proficiency with Databricks Delta Lake and Unity Catalog.
• Strong SQL and Python skills for data transformation.
• Familiarity with governance and security standards.
• Experience collaborating with cross-functional teams.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks.
• Leverage PySpark/Spark and SQL to transform and process large datasets.
• Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems.
• Work Closely with multiple teams to prepare data for dashboard and BI Tools.
• Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.
• Performance & Optimization: Optimize Databricks workloads for cost efficiency and performance.
• Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
• Governance & Security: Implement and manage data security, access controls and governance standards using Unity Catalog.
• Ensure compliance with organizational and regulatory data policies.
• Deployment: Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks and configurations across environments.
• Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.

Skills

SQL

Python

Big data processing

Data governance

Cloud ETL

Databricks

Tools

Databricks

Delta Lake

Unity Catalog

Lakehouse Architecture

GitLab

Azure Cloud Services

Spark

PySpark

Azure Data Factory

Job description

We are seeking highly skilled Azure Data Architect with strong expertise in SQL, Python, Datawarehouse, Cloud ETL tools to join our data team. The ideal candidate will design, implement and optimize large-scale data pipeline, ensuring scalability, reliability and performance. This role involves working closely with multiple teams and business stakeholders to deliver cutting-edge data solutions.

Key Responsibilities:

• Build and maintain scalable ETL/ELT pipelines using Databricks.
• Leverage PySpark/Spark and SQL to transform and process large datasets.
• Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems.
• Work Closely with multiple teams to prepare data for dashboard and BI Tools.
• Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.
• Performance & Optimization:
• Optimize Databricks workloads for cost efficiency and performance.
• Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
• Governance & Security:
• Implement and manage data security, access controls and governance standards using Unity Catalog.
• Ensure compliance with organizational and regulatory data policies.
• Deployment:
• Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks and configurations across environments.
• Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.

Technical Skills:

• Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Delta Live Pipelines, Databricks Runtime etc.)
• Proficiency in Azure Cloud Services.
• Solid Understanding of Spark and PySpark for big data processing.
• Experience in relational databases.
• Knowledge on Databricks Asset Bundles and GitLab.