Azure Data Lead

Derex Technologies Inc; Jersey City (NJ); On-site; USD 150,000 - 200,000; Full time

Job summary

Derex Technologies Inc. is seeking an experienced Azure Data Lead to design, build, and support scalable data engineering solutions on Microsoft Azure in Jersey City, NJ.

You will own end-to-end data pipelines across batch and near-real-time ingestion, transformation, and lakehouse integration, using Azure-native services. The role requires strong hands-on expertise with PySpark, Python, SQL, and Azure Data Factory, Databricks, and Synapse.

Qualifications

• 10+ years in data engineering or large-scale data processing.
• Hands-on Azure data pipelines using ADF, Databricks, and Synapse.
• Proficient in PySpark and Python for data transformation.
• Experience with data lake/warehouse integration on Azure.
• Strong SQL skills for data modeling and validation.
• Experience with CI/CD, Git, and production data pipelines.

Responsibilities

• Design and build scalable Azure data pipelines and workflows.
• Develop batch and near-real-time ingestion patterns.
• Implement data quality controls and monitoring.
• Collaborate with stakeholders to translate requirements into data models.
• Optimize performance, cost, and security of data systems.

Skills

PySpark

Python

Azure Data Factory

Azure Databricks

Azure Data Lake Storage Gen2

Azure Synapse Analytics

Azure Cosmos DB

SQL

Data pipelines

CI/CD

Git

Tools

Azure Data Factory

Azure Databricks

Azure Data Lake Storage Gen2

Azure Synapse Analytics

Azure Cosmos DB

PySpark

Python

SQL

Job description

Title: Azure Data Lead

Location: NJ (Day 1 Onsite)

OverviewThis role is for an experienced Azure Data Lead who can design, build, and support scalable data engineering solutions on Microsoft Azure. The individual will work on modern data platforms involving batch and near-real-time ingestion, data transformation, data lake and warehouse integration, and operational data workloads using Azure-native services.The role requires strong hands-on engineering capability in PySpark, Python, SQL, Azure Data Factory, Azure Databricks, Azure Data Lake Storage, Azure Synapse Analytics, and Azure Cosmos DB. The candidate should be able to convert business and data requirements into reliable, secure, performant, and production-ready data pipelines.

Required skills

• 10+ years of experience in data engineering, cloud data platforms, ETL/ELT development, or large-scale data processing
• Strong hands-on experience in designing, developing, testing, and maintaining Azure-based data pipelines and data processing solutions
• Must have strong hands-on experience with PySpark and Python for large-scale data transformation, automation, data quality checks, and reusable data engineering frameworks
• Azure Data Factory for data ingestion, orchestration, parameterized pipelines, triggers, and monitoring
• Azure Databricks and Apache Spark for scalable data processing using PySpark notebooks, jobs, workflows, and optimized Spark transformations
• Azure Data Lake Storage Gen2 for lakehouse-style storage, folder structures, file formats, access control, and lifecycle management
• Azure Synapse Analytics or Azure SQL for analytical workloads, SQL development, data modeling, performance tuning, and reporting integration
• Azure Cosmos DB for NoSQL data modeling, partition key design, indexing strategy, throughput optimization, change feed processing, and integration with analytics pipelines

Required technical skills

• Strong Python programming skills, including data structures, functions, exception handling, logging, reusable modules, API integration, and automation scripts
• Strong PySpark development experience using DataFrame APIs, joins, aggregations, window functions, UDFs, partitioning, caching, broadcast joins, and performance optimization
• Good SQL skills for querying, transformation, data validation, stored procedures, performance tuning, and troubleshooting data issues
• Experience with Git, Azure DevOps, CI/CD practices, unit testing, deployment pipelines, monitoring, and production support for data engineering workloads

Responsibilities

Data Engineering Design & DevelopmentDesign, develop, and maintain scalable Azure data engineering solutions across:Batch, incremental, and near-real-time data ingestion from databases, APIs, files, applications, and streaming sourcesAzure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse Analytics, Azure SQL, and Azure Cosmos DBBuild and optimize data pipelines for:Data extraction, cleansing, transformation, enrichment, validation, and loading into curated data layersReusable PySpark frameworks, parameterized notebooks, modular Python components, and metadata-driven processing patternsData quality controls, exception handling, audit logging, reconciliation, restartability, and operational monitoringDevelop Cosmos DB-based data solutions by:Designing containers, partition keys, indexing policies, consistency levels, TTL, and throughput configuration based on access patternsImplementing ingestion and integration patterns between Cosmos DB, Azure Data Factory, Databricks, ADLS, and analytical storesUsing Cosmos DB change feed, bulk operations, query tuning, partition-aware design, and cost optimization practicesPipeline Delivery, Optimization & SupportOwn hands-on delivery across:PySpark-based ETL/ELT jobs for large-scale structured, semi-structured, and unstructured data processingPython-based automation, data validation utilities, reusable transformation logic, and integration scriptsAzure Data Factory pipelines, Databricks jobs, Synapse SQL workloads, Cosmos DB integrations, and downstream analytics data productsDrive engineering discipline through:Code reviews, unit testing, version control, CI/CD, deployment automation, and environment configuration managementPipeline monitoring, failure handling, performance tuning, cost optimization, and production incident resolution

Preferred Qualifications

Microsoft Azure Data Engineer certification or equivalent hands-on Azure project experienceExperience with Delta Lake, lakehouse patterns, medallion architecture, and data warehouse modelingExposure to event-driven or streaming patterns using Event Hubs, Kafka, Stream Analytics, or Databricks Structured StreamingUnderstanding of data security, RBAC, managed identities, private endpoints, encryption, and compliance-driven data handling

Key Attributes

Strong analytical and problem-solving skills with the ability to troubleshoot complex data and pipeline issuesAbility to work with business analysts, architects, QA teams, and client stakeholders to clarify requirements and deliver reliable data solutionsGood communication skills with the ability to explain technical designs, pipeline behavior, and production issues clearlyOwnership mindset with focus on quality, maintainability, performance, security, and operational stability Must Have skills:Cosmos DBAzure Data Lake StorageAzure Synapse AnalyticsAzure Data FactoryPostgre SQL