Data Engineer: Build Scalable Pipelines on Databricks & GCP

AgileEngine, LLC.; La Coruña; Sur place; EUR 65 000 - 90 000; Plein temps

Avantages offerts par ce poste

Professional growth

Competitive compensation

Exciting projects

Flextime

Résumé du poste

AgileEngine, LLC. is seeking a Data Engineer with strong Databricks and GCP experience to build scalable ETL/ELT pipelines and productionize agentic workflows for data automation. The role focuses on Databricks, PySpark, SQL, and Delta Lake within a modern data lakehouse architecture.

You will integrate with BigQuery and GCP environments, implement data quality measures, and optimize workloads for performance, reliability, and scalability across multiple business use cases.

Qualifications

• 5+ years hands-on Databricks and PySpark experience.
• Advanced SQL and data-processing skills.
• GCP experience, especially BigQuery.
• Experience with Delta Lake and data lakehouse architectures.
• Ability to design robust ETL/ELT pipelines and data quality checks.
• Experience building production-grade data solutions.
• Strong analytical and troubleshooting skills.
• Knowledge of testing, version control, deployment, monitoring, and production support.

Responsabilités

• Design, develop, and optimize scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
• Build and operationalize agentic workflows to automate data engineering and operational processes.
• Integrate agentic capabilities with Databricks, GCP, BigQuery, and Delta Lake environments.
• Develop data pipelines and processing solutions to support new business requirements.
• Create reusable frameworks and components for multiple data use cases.
• Implement data quality checks, monitoring, and production controls.
• Optimize PySpark and SQL workloads for performance and reliability.
• Support testing, deployment, productionization, and ongoing enhancements.
• Troubleshoot complex data and production issues and implement sustainable fixes.
• Collaborate with business and platform teams to identify automation opportunities.

Connaissances

Databricks

PySpark

SQL

ETL/ELT design

Data quality

Production-grade data solutions

Troubleshooting

Software engineering practices

Outils

BigQuery

Delta Lake

Description du poste

AgileEngine, LLC. is seeking a Data Engineer with strong Databricks and GCP experience to build scalable ETL/ELT pipelines and productionize agentic workflows for data automation. The role focuses on Databricks, PySpark, SQL, and Delta Lake within a modern data lakehouse architecture.

You will integrate with BigQuery and GCP environments, implement data quality measures, and optimize workloads for performance, reliability, and scalability across multiple business use cases.