Data Engineer ID89384

AgileEngine; Ernakulam; Hybrid; INR 1,500,000 - 2,100,000; Full time

Benefits offered by this job

Remote work

Local presence in India

Competitive INR compensation

Exciting projects

Job summary

AgileEngine is seeking a Data Engineer to design, build, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, and SQL. You will productionize agentic workflows for data engineering automation and integrate with GCP (BigQuery) and Delta Lake.

The role emphasizes handling data pipelines, quality checks, and performance optimization within a modern platform for global clients. Remote work is supported, with local India presence required for compliance and collaboration.

Qualifications

• 5+ years of hands-on Databricks and PySpark experience.
• Advanced SQL and data-processing skills.
• Hands-on experience with GCP, particularly BigQuery.
• Experience with Delta Lake and modern data lake/lakehouse architectures.
• Strong ETL/ELT design, data pipeline design, performance optimization, and data quality.
• Experience building reliable, scalable, production-grade data solutions.
• Strong analytical and troubleshooting skills.
• Understanding of software engineering practices, including testing, version control, deployment, monitoring, and production support.
• Upper-intermediate English level.

Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Build and operationalize agentic workflows that automate data validation, issue identification, troubleshooting, and workflow execution.
• Integrate agentic capabilities with Databricks, GCP, BigQuery, and Delta Lake environments while supporting new business requirements and datasets.
• Build reusable frameworks and components that can support multiple data engineering and business use cases.
• Implement data quality checks, monitoring, validation, exception handling, and production controls.
• Optimize PySpark and SQL workloads and support testing, deployment, productionization, and ongoing enhancement of data and agentic solutions.
• Troubleshoot complex production issues and collaborate with business, data engineering, and platform teams to identify additional automation opportunities.

Skills

Databricks

PySpark

SQL

GCP

BigQuery

Delta Lake

ETL/ELT

Data pipelines

Data quality

Testing

Version control

Deployment

Tools

Databricks

PySpark

BigQuery

Delta Lake

GCP

Job description

Job Description

We are looking for a Data Engineer with strong Databricks and GCP experience to build scalable ETL/ELT pipelines and productionize agentic workflows for data engineering automation.

mandatory requirements

are 5+ years of hands‑on Databricks and PySpark experience, advanced SQL skills, strong GCP and BigQuery knowledge, and experience with Delta Lake and lakehouse architectures.

WHAT YOU’LL GAIN

• Remote work - Work from where you feel most productive.
• Local presence in India - Work in a structured and compliant environment aligned with Indian regulations.
• Competitive compensation in INR - Receive compensation in INR, plus support for learning, education, and wellness.
• Exciting projects - Work with modern technologies for global clients and fast-growing companies.

MUST HAVES

• 5+ years of strong hands‑on experience with Databricks and PySpark.
• Advanced SQL and data-processing skills.
• Hands‑on experience with GCP, particularly BigQuery.
• Experience with Delta Lake and modern data lake/lakehouse architectures.
• Strong understanding of ETL/ELT, data pipeline design, performance optimization, and data quality.
• Experience building reliable, scalable, production-grade data solutions.
• Strong analytical and troubleshooting skills.
• Understanding of software engineering practices, including testing, version control, deployment, monitoring, and production support.
• Upper-intermediate English level.

NICE TO HAVES

• Experience developing or integrating AI/agentic workflows, AI agents, or workflow automation solutions.
• Experience applying AI to automate data engineering, validation, troubleshooting, or operational processes.
• Familiarity with orchestration and automation frameworks.
• Experience developing reusable data engineering frameworks and platform components.
• Exposure to productionizing AI-enabled solutions with appropriate validation, monitoring, and human oversight.

WHAT YOU WILL DO

• Design, develop, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, and SQL.
• Build and operationalize agentic workflows that automate data validation, issue identification, troubleshooting, and workflow execution.
• Integrate agentic capabilities with Databricks, GCP, BigQuery, and Delta Lake environments while supporting new business requirements and datasets.
• Build reusable frameworks and components that can support multiple data engineering and business use cases.
• Implement data quality checks, monitoring, validation, exception handling, and production controls.
• Optimize PySpark and SQL workloads and support testing, deployment, productionization, and ongoing enhancement of data and agentic solutions.
• Troubleshoot complex production issues and collaborate with business, data engineering, and platform teams to identify additional automation opportunities.