Consultant - Cloud Data Engineer

enGen Global; Chennai District; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

enGen Global seeks a Cloud Data Engineer to build data pipelines on Databricks/PySpark, loading data into Google Cloud and collaborating with India and USA teams to ensure accuracy and timely execution.

The role emphasizes strong SQL skills, experience with BigQuery/Airflow, and Python/PySpark proficiency, plus ownership and proactive communication to drive project success.

Qualifications

• Bachelor's degree from a reputed university.
• 7–12 years total experience.
• Excellent SQL/data knowledge with 6+ years in complex SQL queries.
• Experience in enterprise-grade ETL applications.
• Experience in Google BigQuery or any cloud database.
• Experience in Google Cloud services like Dataproc, Airflow, etc.
• Working knowledge of PySpark/Python.
• Strong communication and a proactive ownership mindset.

Responsibilities

• Design, develop, and implement robust, high-performance data pipelines using Databricks/PySpark to meet data integration needs.
• Utilize SQL for complex data manipulation, analysis, and query optimization.
• Leverage Google Cloud services such as Dataproc, BigQuery, and Airflow.
• Create custom Python/PySpark scripts.
• Handle minor development/enhancement tasks for new requirements.
• Validate data between source and target and rerun jobs as needed.
• Lead other developers, review code, and ensure quality and efficiency.
• Provide regular updates on tasks, status, and risks to the project manager.

Skills

SQL proficiency

Pyspark/Python

Data pipelines

Cloud data engineering

SQL optimization

Education

Bachelor's degree

Tools

Databricks

BigQuery

Dataproc

Airflow

Python

Job description

Cloud Data Engineer

Role Summary

This job involves creating data pipelines using a configurable framework (developed using Databricks / pyspark), which are used to load data from multiple sources to Google Cloud. It also involves extensive collaboration with other team members (India USA) to follow a plan, execute workflows, ensure data load accuracy and debug in case of issues.

Essential Responsibilities

• Design, develop, and implement robust, high-performance data pipelines using Databricks/PySpark to meet diverse data integration requirements. Understand data and its relation / significance from business perspective
• Leverage extensive SQL expertise for complex data manipulation, analysis, and query optimization.
• Experience in Google Cloud services like Dataproc, BigQuery, Airflow, etc.
• Create custom scripts in python / pyspark
• Minor development and enhancement activities as needed to handle new / custom requirements
• Validate data between source and target highlight issues and rerun jobs as needed
• Lead other developers, review their code and ensure quality optimal code is being delivered
• Provide regular updates on the tasks, status and risks to project manager

Required

• Bachelors degree from a reputed university
• 7 to 12 years overall experience
• Excellent SQL / data knowledge (6+ years) in creating complex SQL queries, Stored Procedures etc.
• Experience in developing enterprise grade ETL applications (any tool)
• Experience in Google BigQuery or any cloud database
• Experience in Google Cloud services like Managed service for Apache Spark(Dataproc), Airflow etc.
• Working knowledge in Pyspark / Python
• Good communication and positive attitude
• Attitude to take ownership and drive tasks without follow-ups

Good to have

• Experience in Big data technologies like Hadoop, Hive, etc.
• Knowledge in Iceberg, Starburst, DBT
• Knowledge in US Healthcare domain
• Knowledge in Agile