Data Engineer II

DAT Freight & Analytics; Bengaluru; Sur place; INR 1 200 000 - 2 400 000; Plein temps

Résumé du poste

DAT Freight & Analytics is seeking a data engineer to develop and maintain cloud-native ELT pipelines, focusing on Snowflake and dbt.

You will build and optimize Airflow-based workflows, implement automated testing, and contribute to CI/CD practices. Strong Python skills and experience with data warehousing are essential for this role.

Qualifications

• 3–5 years of professional experience in data engineering, analytics engineering or a similar role.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related quantitative field.
• Hands-on experience with cloud data warehouses, preferably Snowflake.
• Familiarity with data transformation concepts and dbt is a bonus.
• Experience creating or running jobs/workflows using Apache Airflow or similar orchestration tools.

Responsabilités

• ELT pipeline development, testing and maintenance for Snowflake and S3 Iceberg layers.
• Design scalable data pipelines supporting incremental loads, backfills, and reprocessing.
• Develop data models with dbt for cleaning, aggregation, and transformation.
• Build Python scripts for extraction, automation, and monitoring.
• Collaborate with team on CI/CD, version control, and automated testing.

Connaissances

Python programming

Team collaboration

Problem solving

Formation

Bachelor's degree in CS/IT/Engineering or related field

Outils

Snowflake

dbt

Apache Airflow

Git

GitHub Actions

Description du poste

The core responsibilities for the job include the following:

Data Pipeline Development and Engineering:

• ELT Process Implementation: Assist senior engineers in designing, building, testing, and maintaining cloud-native ELT (Extract, Load, Transform) data pipelines, ensuring data is reliably loaded into the Snowflake and S3 Iceberg layers.
• Pipeline Architecture: Contribute to the design of scalable, modular data pipelines that support incremental loads, backfills, and reprocessing with minimal manual intervention.
• Transformation with dbt: Develop and maintain data models using dbt (data build tool) for data cleaning, aggregation, and transformation and utilize the Snowflake data warehouse.
• Python Scripting: Utilize Python to build custom data extraction scripts, implement monitoring tools, and contribute to general automation efforts.
• Pipeline Reliability: Build in retry logic, alerting, and failure-handling patterns so pipelines degrade gracefully and self-recover where possible.

Workflow Orchestration and Automation:

• Airflow DAGs: Learn to author, schedule, and monitor data workflows defined as Directed Acyclic Graphs (DAGs) in Apache Airflow.
• Pipeline Scheduling: Integrate and orchestrate dbt runs and other pipeline tasks within Airflow to manage dependencies and execution timing.
• Automation-First Mindset: Actively look for repetitive, manual, or error-prone steps across the data lifecycle and automate them—from data ingestion to deployment to reporting.
• Automated Testing and Deployment: Contribute to automated test suites and deployment scripts that reduce manual QA and release effort.

CI/CD and Version Control:

• Git Workflow: Use Git for version control, following branching, code review, and pull request best practices.
• GitHub Actions: Build and maintain GitHub Actions workflows to automate testing, linting, dbt builds, and deployment of data pipeline code.
• Continuous Integration: Ensure new pipeline and model changes are automatically tested and validated before merging, catching issues before they reach production.
• Continuous Deployment: Support automated promotion of dbt models and pipeline code across environments (dev, staging, and production).

AI-Enabled Engineering:

• AI-Assisted Development: Use AI coding assistants to accelerate development, generate boilerplate, and speed up code review and refactoring.
• AI-Augmented Documentation: Leverage AI tools to help draft and maintain technical documentation for data models, DAGs, and pipeline logic.
• Applied Curiosity: Stay current on emerging AI-assisted data engineering tools and workflows, and bring suggestions for improving team efficiency.

Data Quality, Testing, and Monitoring:

• Data Quality: Implement data validation and testing frameworks using features of dbt (e. g., uniqueness, non-null checks) to ensure high data quality and accuracy within Snowflake data marts.
• Troubleshooting: Monitor data pipeline health, troubleshoot failed Airflow tasks and dbt runs, and quickly resolve data flow issues.
• Documentation: Maintain clear and current technical documentation for data models, Airflow DAGs, and pipeline logic.

Requirements:

• Experience: 3-5 years of professional experience in a data engineering, analytics engineering, or similar technical role (including relevant internship experience).
• Education: Bachelor's Degree in Computer Science, Information Technology, Engineering, or a Related Quantitative Field.

Technical Proficiency:

• Required: Strong proficiency in Python.
• Hands-on experience with a cloud data warehouse, preferably Snowflake.
• Familiarity with data transformation concepts and tools, DBT a bonus.
• Basic experience creating or running jobs/workflows using an orchestration tool like Apache Airflow.
• Experience with Git and CI/CD pipelines, ideally GitHub Actions.
• Comfort using AI-assisted coding tools in a professional engineering workflow.
• Soft Skills: Strong problem-solving abilities, excellent attention to detail, and a proactive, collaborative approach to teamwork.

Key Technologies:

• Data Warehouse: Snowflake/Data Lake.
• Transformation: dbt (data build tool).
• Orchestration: Apache Airflow.
• CI/CD: Git, GitHub Actions.
• Programming: Python.
• AI-Assisted Development: LLM-based coding assistants (e. g., Claude, Copilot) for code generation, review, and documentation.