Cloud Data Engineer

Nidus Softdev; India; Presencial; INR 1.500.000 - 2.500.000; Jornada completa

Descripción de la vacante

Nidus Softdev is seeking a Cloud Data Engineer to help migrate on‑prem ETL workloads to Azure Cloud, Databricks and Snowflake, and to design scalable data platforms for modern data processing.

You will collaborate with cross‑functional teams, implement CI/CD pipelines with GitHub Actions, and focus on data quality, security, and operations to ensure reliable data flows across the organization.

Formación

• 3+ years of experience as a Cloud Data Engineer.
• Hands-on with Databricks, Azure data tools, Spark, PySpark and Snowflake.
• Experience migrating data to cloud and modernizing ETL.
• Strong ETL development experience using on-prem databases.
• Proficient in Python or scripting languages for data processing.
• Advanced SQL with complex queries and ETL transformations.
• Experience with GitHub Actions and PRs.
• Commitment to code reviews, testing and data quality.

Responsabilidades

• Contribute to migration of ETL workloads to Azure Cloud, Databricks and Snowflake.
• Develop data platform solutions using Databricks, ADF, SHIR, Logic Apps, ADLS Gen2, Blob Storage and Snowflake.
• Review and analyze existing on-prem ETL processes.
• Implement CI/CD pipelines using GitHub Actions.
• Collaborate with cross-functional teams for seamless data flow.
• Tune workloads for performance and scalability.
• Implement data quality validation, monitoring and observability.
• Support automated reconciliation and data validation processes.
• Ensure data security and compliance with standards.

Conocimientos

Cloud Data Engineer

Databricks

Azure Data Factory

SHIR

Logic Apps

ADLS Gen2

Blob Storage

Apache Spark

PySpark

Snowflake

Data migration

ETL development

Python scripting

SQL development

GitHub Actions

Code reviews

Data quality

Agile

Adaptability

AI-assisted development

Azure/Databricks cert

Descripción del empleo

Role & responsibilities

• Contribute to the migration of the ETL workloads from on-premises platforms to Azure Cloud, Databricks and Snowflake.
• Develop and implement data platform solutions using Databricks, Azure Data Factory (ADF), Self-hosted Integration Runtime (SHIR), Logic Apps, Azure Data Lake Storage Gen2 (ADLS Gen2), Blob Storage, and Snowflake.
• Review and analyze existing on-premises ETL processes.
• Implement DevOps practices and CI/CD pipelines using GitHub Actions.
• Collaborate with cross-functional teams to ensure seamless integration and data flow.
• Tune and optimize workloads for performance scalability.
• Implement data quality validation, monitoring, and observability processes
• Support the development and maintenance of automated reconciliation and data validation processes.
• Ensure data security and compliance with industry standards.

Required Qualifications:

• Minimum of 3+ years of experience as a Cloud Data Engineer.
• Hands-on experience with Databricks, Azure Cloud data tools (ADF, SHIR, Logic Apps, ADLS Gen2, Blob Storage), Apache Spark, PySpark and Snowflake.
• Experience developing or supporting data migration and cloud modernization initiatives.
• Strong experience in ETL development using on-premises databases and ETL technologies.
• Experience with Python or other scripting languages for data processing.
• Strong SQL development experience including complex queries, stored procedures, views, and ETL transformations.
• Experience with GitHub branching, pull requests, and GitHub Actions
• Experience with code reviews and engineering best practices.
• Experience supporting automated testing and data quality processes.
• Experience working within Agile delivery methodologies.
• Ability to adapt to new technologies and learn quickly.
• Experience using AI-assisted development tools (GitHub Copilot, Databricks Assistant, ChatGPT, Claude, etc.) to improve engineering productivity.

Preferred Qualifications:

• Experience with other cloud platforms and data solutions.
• Experience with data modeling and database design.
• Experience implementing Bronze/Silver/Gold (Medallion) architecture patterns.
• Experience with Delta Lake and Lakehouse architectures and optimization techniques.
• Knowledge of data governance and data quality best-practices.
• Experience with development in Snowflake for data engineering and analytics workloads.
• Experience with healthcare payer data domains including Claims, Membership, Enrollment, Provider, Clinical, and Financial data.
• Experience with Airflow or other orchestration frameworks.
• Experience with performance optimization in Databricks and Snowflake.
• Experience leveraging AI/ML solutions for data engineering workflows and automation.
• Certification in Azure or Databricks.
• Excellent analytical, problem-solving, communication, and collaboration skills.
• Demonstrated ability to work independently, manage ambiguity, and deliver high-quality solutions.
• Demonstrated ability to work within tight deadlines and effectively raise critical issues.Preferred Qualifications:
• Experience with other cloud platforms and data solutions.
• Experience with data modeling and database design.
• Experience implementing Bronze/Silver/Gold (Medallion) architecture patterns.
• Experience with Delta Lake and Lakehouse architectures and optimization techniques.
• Knowledge of data governance and data quality best-practices.
• Experience with development in Snowflake for data engineering and analytics workloads.
• Experience with healthcare payer data domains including Claims, Membership, Enrollment, Provider, Clinical, and Financial data.
• Experience with Airflow or other orchestration frameworks.
• Experience with performance optimization in Databricks and Snowflake.
• Experience leveraging AI/ML solutions for data engineering workflows and automation.
• Certification in Azure or Databricks.