Senior Data Engineer

NorthBay Solutions LLC; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

NorthBay Solutions seeks an experienced Senior Data Engineer with 8–10+ years of designing and managing modern data platforms and pipelines on Microsoft Azure. You will work on batch and real-time data ingestion, data lake/warehouse/lakehouse architectures, and cloud-native ETL/ELT frameworks to support enterprise analytics and BI initiatives.

The role requires hands-on Azure expertise, data governance, and collaboration with data scientists, analysts, and application teams to deliver scalable

Qualifications

• 8–10+ years of Data Engineering and Enterprise Data Platform development.
• Hands-on experience with Microsoft Azure Data Services (ADF, Synapse, ADLS Gen2, Azure SQL).
• Experience building large-scale ETL/ELT pipelines and data integration solutions.

Responsibilities

• Design, develop, and maintain scalable data platforms and pipelines on Azure.
• Build real-time and batch data ingestion from multiple sources.
• Design data lake, data warehouse, and lakehouse architectures for analytics.
• Develop and manage ETL/ELT processes using cloud-native practices.
• Implement data quality, governance, and security across Azure environments.
• Collaborate with stakeholders and data scientists to deliver scalable solutions.
• Optimize storage, processing, and query performance across platforms.
• Support integration with AI/ML, BI, and enterprise apps.
• Participate in reviews, debugging, and architectural discussions.
• Ensure high availability and operational excellence of data platforms.

Skills

Azure Data Services

Azure Data Factory

Azure Synapse Analytics

Azure Data Lake Storage

SQL

PySpark

Scala

Java

Data Warehousing

ETL/ELT

CI/CD

DevOps

Security & Governance

Tools

Databricks

Delta Lake

Power BI

Job description

Location: Remote – India

Employment Type: Full-Time, Permanent

Job Summary

We are seeking an experienced Senior Data Engineer with 8–10+ years of experience in designing, developing, and managing modern data platforms, data pipelines, and cloud-based analytics solutions. The ideal candidate will have strong expertise in Azure Data Services, large-scale data processing, data warehousing, ETL/ELT frameworks, and cloud-native data architectures. The role requires hands-on experience in building scalable, secure, and high-performance data solutions that support enterprise analytics, reporting, AI, and business intelligence initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data platforms and data pipelines on Microsoft Azure.
• Build and optimize batch and real-time data ingestion frameworks from multiple structured and unstructured data sources.
• Design and implement data lake, data warehouse, and lakehouse architectures to support analytics and reporting workloads.
• Develop and manage ETL/ELT processes using modern cloud-native data engineering practices.
• Implement data transformation, cleansing, validation, and quality frameworks to ensure data accuracy and reliability.
• Collaborate with business stakeholders, data analysts, data scientists, and application teams to understand data requirements and deliver scalable solutions.
• Optimize data storage, processing, and query performance across enterprise data platforms.
• Implement security, governance, monitoring, and compliance best practices across Azure environments.
• Support integration of data platforms with AI/ML, business intelligence, and enterprise applications.
• Participate in architecture reviews, code reviews, troubleshooting, and technical mentoring activities.
• Ensure high availability, scalability, and operational excellence of data platforms and pipelines.

Required Skills & Qualifications

• 8–10+ years of experience in Data Engineering, Data Warehousing, and Enterprise Data Platform development.
• Strong hands-on experience with Microsoft Azure Data Services.
• Expertise in Azure Data Factory (ADF), Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), and Azure SQL Database.
• Experience building and managing large-scale ETL/ELT pipelines and data integration solutions.
• Strong proficiency in SQL, query optimization, and database performance tuning.
• Hands-on experience with PySpark, Apache Spark, and distributed data processing frameworks.
• Strong programming skills in Python, Scala, or Java.
• Experience with dimensional modeling, data warehousing concepts, and modern lakehouse architectures.
• Experience working with structured, semi-structured, and unstructured data.
• Strong understanding of data governance, data quality, metadata management, and security best practices.
• Experience with REST APIs, data integration patterns, and enterprise system connectivity.
• Hands-on experience with Git, CI/CD pipelines, and DevOps practices.
• Strong analytical, problem-solving, and communication skills.

Preferred Qualifications

• Experience with Microsoft Fabric, OneLake, Dataflows, and Fabric Data Engineering workloads.
• Experience with Databricks, Delta Lake, and lakehouse implementations.
• Knowledge of real-time streaming technologies such as Azure Event Hubs, Apache Kafka, or Azure Stream Analytics.
• Experience supporting AI/ML and advanced analytics workloads through enterprise data platforms.
• Familiarity with Power BI datasets, semantic models, and enterprise reporting architectures.
• Experience with data governance tools such as Microsoft Purview.
• Microsoft Azure Data Engineering certifications are highly preferred.
• Experience working in Agile/Scrum environments.

Nice to Have

• Experience with Microsoft Fabric Data Engineering and Analytics solutions.
• Exposure to MLOps and DataOps practices.
• Knowledge of containerization technologies such as Docker and Kubernetes.
• Experience with Infrastructure as Code (Terraform, ARM Templates, or Bicep).
• Familiarity with Snowflake, BigQuery, or other cloud data warehouse platforms.
• Experience with enterprise-scale data migration and modernization projects.
• Understanding of Generative AI, Vector Databases, and data platforms supporting AI workloads.