Data Engineer- Apache Airflow

Astek Group

Ernakulam

On-site

INR 1,200,000 - 1,800,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Astek Group in India seeks a Senior Data Engineer to design, build, and maintain robust data workflows and pipelines across systems. You’ll work with Airflow, PySpark, Hive, AWS, and Snowflake to deliver reliable data solutions with a strong focus on quality and security.

You’ll optimize performance, implement data modeling concepts, and collaborate with cross-functional teams to ensure scalable, compliant data architectures in a fast-paced environment.

Qualifications

  • Bachelor’s degree in Computer Science, Information Technology, or related field.
  • Proficient in Python (2.7 and 3.6+).
  • Experience in unit testing and test-driven development.
  • Familiarity with Secrets Management tools and techniques.
  • Knowledge of data modeling principles and practices.
  • Strong understanding of Git version control and GitLab workflows.
  • Excellent problem-solving and communication skills.
  • Awareness of IT security best practices (ISO / SOC or equivalent).
  • Experience with SSIS for data migration workflows.
  • Experience with Snowflake (external/internal tables, views, materialized views).
  • Familiarity with Hive on Tez for optimized query execution.
  • Experience with AWS command line tools and S3 operations.
  • Knowledge of Livy for efficient Spark job execution.

Responsibilities

  • Design, implement, and manage complex DAGs and SubDAGs in Apache Airflow.
  • Utilize Operators, macros, connections, variables, and XCom for efficient task execution.
  • Conduct unit testing to ensure the robustness of Airflow workflows.
  • Perform file operations and data movement between HDFS and S3 (and vice versa).
  • Manage Hive external tables, update table DDL, and handle partition operations.
  • Execute basic Hadoop commands and navigate/traverse HDFS.
  • Work with Hive on Tez for optimized query execution.
  • Utilize AWS command line tools for tasks such as listing and copying files from S3.
  • Implement and manage secrets using tools like Ansible Vault and AWS Secrets Manager.
  • Develop and optimize data processing tasks using PySpark.
  • Understand and leverage Livy for efficient Spark job execution.
  • Import and export data between Microsoft SQL Server and HDFS using Sqoop.
  • Apply dimension modeling principles, define facts and dimensions, and manage surrogate keys.
  • Implement effective branching strategies in Git.
  • Participate in the process of raising and reviewing Merge Requests in GitLab.
  • Utilize SSIS for migrating data workflows, ensuring a smooth transition.
  • Model data in Snowflake, differentiating between external and internal tables.
  • Understand the use of views and materialized views in Snowflake.

Skills

Python (3.x)
Unit testing
Test-driven development
Data modeling
Git / GitLab workflows
Communication skills
IT security awareness
SSIS
Snowflake experience
Hive/Tez knowledge
AWS CLI / S3 operations
Livy for Spark

Education

Bachelor's degree in Computer Science/IT or related field

Tools

Apache Airflow
PySpark
Hive
Snowflake
AWS CLI
S3
HDFS
Sqoop
Ansible Vault
AWS Secrets Manager
Livy
Tez
SSIS
SQL Server
GitLab

Job description

As a Senior Data Engineer you will play a key role in designing, developing, and maintaining data workflows and pipelines, ensuring the efficient and reliable movement of data across various systems. You will work with modern data technologies including Apache Airflow, PySpark, Hive, AWS, and Snowflake, while collaborating with cross-functional teams to deliver high-quality data solutions. This role requires a strong focus on data quality, performance optimization, and adherence to security and compliance standards.

What You’ll Do
  • Design, implement, and manage complex DAGs and SubDAGs in Apache Airflow.
  • Utilize Operators, macros, connections, variables, and XCom for efficient task execution.
  • Conduct unit testing to ensure the robustness of Airflow workflows.
  • Perform file operations and data movement between HDFS and S3 (and vice versa).
  • Manage Hive external tables, update table DDL, and handle partition operations.
  • Execute basic Hadoop commands and navigate/traverse HDFS.
  • Work with Hive on Tez for optimized query execution.
  • Utilize AWS command line tools for tasks such as listing and copying files from S3.
  • Implement and manage secrets using tools like Ansible Vault and AWS Secrets Manager.
  • Develop and optimize data processing tasks using PySpark.
  • Understand and leverage Livy for efficient Spark job execution.
  • Import and export data between Microsoft SQL Server and HDFS using Sqoop.
  • Apply dimension modeling principles, define facts and dimensions, and manage surrogate keys.
  • Implement effective branching strategies in Git.
  • Participate in the process of raising and reviewing Merge Requests in GitLab.
  • Utilize SSIS for migrating data workflows, ensuring a smooth transition.
  • Model data in Snowflake, differentiating between external and internal tables.
  • Understand the use of views and materialized views in Snowflake.
What You’ll Need
  • Bachelor’s degree in Computer Science, Information Technology, or a related field.
  • Proficient in Python (both Python 2.7 and 3.6+).
  • Experience in unit testing and test-driven development.
  • Familiarity with Secrets Management tools and techniques.
  • Knowledge of data modeling principles and practices.
  • Strong understanding of Git version control and GitLab workflows.
  • Excellent problem-solving and communication skills.
  • Awareness or knowledge of IT security best practices as defined by ISO / SOC or similar standards.
  • Experience with SSIS for data migration workflows.
  • Experience with Snowflake (external/internal tables, views, materialized views).
  • Familiarity with Hive on Tez for optimized query execution.
  • Experience with AWS command line tools and S3 operations.
  • Knowledge of Livy for efficient Spark job execution.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer- Apache Airflow
Senior Data Engineer- Apache Airflow

Simelabs - An Astek Company • Ernakulam

On-site
INR 1,500,000 - 3,000,000
AWS & Airflow Developer
AWS & Airflow Developer

LPL Financials • Hyderabad

Hybrid
INR 1,200,000 - 1,800,000
Data Engineer
Data Engineer

Amplicomm • Bengaluru

On-site
INR 1,500,000 - 2,800,000
Data Engineer
Data Engineer

Staples India • Chennai District

On-site
INR 800,000 - 1,200,000
Airflow Python Developer
Airflow Python Developer

Sutherland • Hyderabad

On-site
INR 1,000,000 - 1,600,000
Data Engineer
Data Engineer

Grid Dynamics • Bengaluru

Hybrid
INR 900,000 - 1,400,000
Senior Developer, Data Engineer
Senior Developer, Data Engineer

Intercontinental Exchange Holdings, Inc. • Hyderabad

On-site
INR 2,500,000 - 4,500,000
Data Engineer - ETL/PySpark
Data Engineer - ETL/PySpark

Forward Eye Technologies • Pune District

On-site
INR 900,000 - 1,300,000
Data Engineer - SQL/PySpark
Data Engineer - SQL/PySpark

Forward Eye Technologies • Dadri

On-site
INR 1,200,000 - 1,800,000
Data Engineer - ETL/PySpark
Data Engineer - ETL/PySpark

Forward Eye Technologies • Dadri

On-site
INR 1,500,000 - 2,400,000