Etl Developer

Connected Value Health Solutions; Coimbatore District; On-site; INR 600,000 - 1,200,000; Full time

Job summary

Connected Value Health Solutions in Coimbatore is seeking an experienced ETL Developer to build robust data pipelines using Apache Airflow, Redshift, and SQL.

The role focuses on cloud-native ETL design, data integration from diverse sources, and performance tuning for scalable analytics. You will collaborate with data architects and BI teams in a fast-paced healthcare-focused environment.

Qualifications

• 3–6 years of hands-on ETL development experience.
• Proven experience with Apache Airflow, Amazon Redshift, and strong SQL.
• Strong understanding of data warehousing concepts and cloud-based data ecosystems.
• Familiarity with handling flat files, APIs, and external sources.
• Experience with job orchestration, error handling, and scalable transformation patterns.
• Ability to work independently and meet deadlines.

Responsibilities

• Design and develop scalable ETL pipelines using Apache Airflow with orchestration and monitoring.
• Integrate data from multiple sources including Redshift, flat files, APIs, Excel, and relational databases.
• Write performant SQL queries for data staging and transformation within Redshift and Snowflake.
• Optimize data pipelines and orchestrations for large-scale data volumes.
• Implement robust error handling, logging, and retry logic in Airflow DAGs.
• Document data flows, DAG configurations, transformation logic, and operational procedures.
• Collaborate with data architects, analysts, and BI teams to fulfill data needs.
• Ensure ETL processes adhere to data security, governance, and privacy regulations.

Skills

Apache Airflow

Amazon Redshift

SQL

ETL development

Data warehousing

Cloud data platforms

API integration

Education

Bachelor’s degree in Computer Science or related

Tools

Git

Azure DevOps

CI/CD pipelines

Job description

Job Title: ETL Developer FTE
Location: Coimbatore
Start Date: ASAP

Job Summary

We are looking for an experienced ETL Developer with strong expertise in Apache Airflow, Redshift, and SQL-based data pipelines, with upcoming transitions to Snowflake. This role based in Coimbatore, ideal for professionals who can independently deliver high-quality ETL solutions in a cloud-native, fast-paced environment.

Key Responsibilities

1. ETL Design and Development

• Design and develop scalable and modular ETL pipelines using Apache Airflow, with orchestration and monitoring capabilities.
• Translate business requirements into robust data transformation pipelines across cloud data platforms.
• Develop reusable ETL components to support a configuration-driven architecture.

2. Data Integration and Transformation

• Integrate data from multiple sources: Redshift, flat files, APIs, Excel, and relational databases.
• Implement transformation logic such as cleansing, standardization, enrichment, and deduplication.
• Manage incremental and full loads, along with SCD handling strategies.

3. SQL and Database Development

• Write performant SQL queries for data staging and transformation within Redshift and Snowflake.
• Utilize joins, window functions, and aggregations effectively.
• Ensure indexing and query tuning for high-performance workloads.

4. Performance Tuning

• Optimize data pipelines and orchestrations for large-scale data volumes.
• Tune SQL queries and monitor execution plans.
• Implement best practices in distributed data processing and cloud-native optimizations.

5. Error Handling and Logging

• Implement robust error handling and logging in Airflow DAGs.
• Enable retry logic, alerting mechanisms, and failure notifications.

6. Testing and Quality Assurance

• Conduct unit and integration testing of ETL jobs.
• Validate data outputs against business rules and source systems.
• Support QA during UAT cycles and help resolve data defects.

7. Deployment and Scheduling

• Deploy pipelines using Git-based CI/CD practices.
• Schedule and monitor DAGs using Apache Airflow and integrated tools.
• Troubleshoot failures and ensure data pipeline reliability.

8. Documentation and Maintenance

• Document data flows, DAG configurations, transformation logic, and operational procedures.
• Maintain change logs and update job dependency charts.

9. Collaboration and Communication

• Work closely with data architects, analysts, and BI teams to define and fulfill data needs.
• Participate in stand‑ups, sprint planning, and post‑deployment reviews.

10. Compliance and Best Practices

• Ensure ETL processes adhere to data security, governance, and privacy regulations (HIPAA, GDPR, etc.).
• Follow naming conventions, version control standards, and deployment protocols.

Required Skills & Experience

• 3–6 years of hands‑on experience in ETL development.
• Proven experience with Apache Airflow, Amazon Redshift, and strong SQL.
• Strong understanding of data warehousing concepts and cloud‑based data ecosystems.
• Familiarity with handling flat files, APIs, and external sources.
• Experience with job orchestration, error handling, and scalable transformation patterns.
• Ability to work independently and meet deadlines.

Preferred Skills

• Exposure to Snowflake or plans to migrate to Snowflake platforms.
• Experience in healthcare, life sciences, or regulated environments is a plus.
• Familiarity with Azure Data Factory, Power BI, or other cloud BI tools.
• Knowledge of Git, Azure DevOps, or other version control and CI/CD platforms.