Data Engineer: Databricks, GCP & AI-Driven Pipelines

AgileEngine, LLC.; Braga; Hybrid; EUR 60,000 - 95,000; Full time

Benefits offered by this job

Professional growth

Competitive USD-based compensation and

education budgets

Flextime

Job summary

AgileEngine is seeking a Data Engineer with strong Databricks and GCP experience to design and deploy scalable ETL/ELT pipelines. You will build agentic workflows that automate data engineering tasks and integrate with Databricks, GCP, BigQuery, and Delta Lake environments.

You will optimize data pipelines for performance and reliability, implement data quality controls, and collaborate with cross-functional teams to drive data-driven decisions.

Qualifications

• 5+ years of hands-on Databricks and PySpark experience.
• Advanced SQL and data-processing skills.
• Hands-on GCP experience, esp. BigQuery.
• Delta Lake and modern lakehouse architectures.
• Strong understanding of ETL/ELT, data pipeline design and optimization.
• Experience building reliable, scalable data solutions.
• Strong analytical and troubleshooting skills.
• Familiar with testing, version control, deployment, monitoring and production support.

Responsibilities

• Design, develop, and optimize scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
• Build and operationalize agentic workflows to automate data engineering and operational processes.
• Integrate agentic capabilities with Databricks, GCP, BigQuery, and Delta Lake environments.
• Develop data pipelines and processing solutions to support new datasets and business requirements.
• Create reusable data engineering frameworks and platform components.
• Implement data quality checks, monitoring, validation, and production controls.
• Optimize PySpark and SQL workloads for performance, reliability, and scalability.
• Support testing, deployment, productionization, and ongoing enhancement of data solutions.
• Troubleshoot complex data and production issues and implement durable solutions.
• Collaborate with business, data engineering, and platform teams to identify automation opportunities.

Skills

Databricks

PySpark

SQL

BigQuery

Delta Lake

ETL/ELT design

Data quality

Troubleshooting

Testing & deployment

Job description

AgileEngine is seeking a Data Engineer with strong Databricks and GCP experience to design and deploy scalable ETL/ELT pipelines. You will build agentic workflows that automate data engineering tasks and integrate with Databricks, GCP, BigQuery, and Delta Lake environments.

You will optimize data pipelines for performance and reliability, implement data quality controls, and collaborate with cross-functional teams to drive data-driven decisions.