Senior Data Engineer

Advantmed; India; Remote; INR 2,500,000 - 3,600,000; Full time

Job summary

Advantmed is seeking a Senior Data Engineer to build a robust data platform on Azure Cloud. You will design and maintain scalable data pipelines for ingestion, transformation, and analytics, ensuring data quality and reliability.

You will work with Azure Data Factory, Databricks, and Azure Synapse Analytics, applying best practices in data modeling, governance, and cost management. Strong Python/SQL skills and collaboration across teams are required.

Qualifications

• Develop and maintain data ingestion pipelines using Azure Data Factory, Databricks, and Azure Event Hubs.

Responsibilities

• Data Ingestion and Integration: Develop and maintain ingestion pipelines; integrate data from APIs, databases, files, and streams.
• ETL/ELT Development: Design and implement ETL/ELT workflows for data lake/warehouse storage.
• Data Modeling and Storage: Design models for Azure Data Lake Storage and Synapse.
• Quality Assurance: Data validation, monitoring, reconciliation to ensure accuracy.
• Collaboration and Documentation: Work with architects/analysts and document configurations.
• Cloud Services and Infrastructure: Leverage Azure Data Factory, Databricks, Functions, and Logic Apps.
• Security and Governance: Ensure pipelines meet security policies; use IAM, encryption, Key Vault.
• Continuous Improvement: Monitor pipelines and adopt improvements; stay updated on tech.

Skills

Azure Data Factory

Databricks

Azure Synapse Analytics

Python

SQL

Spark

ETL/ELT processes

Data modeling

Data warehousing

Data lake architectures

REST APIs

Streaming data (Kafka, Event Hubs)

Batch processing

Kafka

Event Hubs

Education

Bachelor’s degree in Computer Science, Data Science, or related field

8+ years of experience in data engineering or similar role

Tools

Azure Data Factory

Databricks

Azure Synapse Analytics

Job description

We are seeking a talented and experienced Senior Data Engineer to join our team and contribute to building a robust data platform on Azure Cloud. The ideal candidate will have hands-on experience designing and managing data pipelines, ensuring data quality, and leveraging cloud technologies for scalable and efficient data processing. The Data Engineer will design, develop, and maintain scalable data pipelines and systems to support the ingestion, transformation, and analysis of large datasets. The role requires a deep understanding of data workflows, cloud platforms (Azure), and strong problem-solving skills to ensure efficient and reliable data delivery.

Key Responsibilities

Data Ingestion and Integration

• Develop and maintain data ingestion pipelines using tools like Azure Data Factory, Databricks, and Azure Event Hubs.
• Integrate data from various sources, including APIs, databases, file systems, and streaming data.

ETL/ELT Development

• Design and implement ETL/ELT workflows to transform and prepare data for analysis and storage in the data lake or data warehouse.
• Automate and optimize data processing workflows for performance and scalability.

Data Modeling and Storage

• Design data models for efficient storage and retrieval in Azure Data Lake Storage and Azure Synapse Analytics.
• Implement best practices for partitioning, indexing, and versioning in data lakes and warehouses.

Quality Assurance

• Implement data validation, monitoring, and reconciliation processes to ensure data accuracy and consistency.
• Troubleshoot and resolve issues in data pipelines to ensure seamless operation.

Collaboration and Documentation

• Work closely with data architects, analysts, and other stakeholders to understand requirements and translate them into technical solutions.
• Document processes, workflows, and system configurations for maintenance and onboarding purposes.

Cloud Services and Infrastructure

• Leverage Azure services like Azure Data Factory, Databricks, Azure Functions, and Logic Apps to create scalable and cost-effective solutions.
• Monitor and optimize Azure resources for performance and cost management.

Security and Governance

• Ensure data pipelines comply with organizational security and governance policies.
• Implement security protocols using Azure IAM, encryption, and Azure Key Vault.

Continuous Improvement

• Monitor existing pipelines and suggest improvements for better efficiency, reliability, and scalability.
• Stay updated on emerging technologies and recommend enhancements to the data platform.

Skills

• Strong experience with Azure Data Factory, Databricks, and Azure Synapse Analytics.
• Proficiency in Python, SQL, and Spark.
• Hands-on experience with ETL/ELT processes and frameworks.
• Knowledge of data modeling, data warehousing, and data lake architectures.
• Familiarity with REST APIs, streaming data (Kafka, Event Hubs), and batch processing.

Good To Have

• Experience with tools like Azure Purview, Delta Lake, or similar governance frameworks.
• Understanding of CI/CD pipelines and DevOps tools like Azure DevOps or Terraform.
• Familiarity with data visualization tools like Power BI.

Competency

• Analytical Thinking
• Clear and effective communication
• Time Management
• Team Collaboration
• Technical Proficiency
• Supervising Others
• Problem Solving
• Risk Management
• Organizing & Task Management
• Creativity/innovation
• Honesty/Integrity

Education

• Bachelors degree in Computer Science, Data Science, or a related field.
• 8+ years of experience in a data engineering or similar role.