Senior Data Engineer — Databricks, GCP, AI Pipelines
AgileEngine, LLC.
Destaca en este puesto — crea un currículum adaptado y una carta de presentación en aproximadamente un minuto.
Ventajas ofrecidas por este puesto de trabajo
Descripción de la vacante
AgileEngine, LLC. is seeking a Data Engineer with strong Databricks and GCP experience to develop scalable ETL/ELT pipelines and productionize agentic workflows for data engineering automation.
You will design, build, and optimize data pipelines using Databricks, PySpark, and SQL, while integrating with BigQuery and Delta Lake in production environments. Collaborate across teams to automate data processes and ensure data quality.
Formación
- 5+ years of hands-on Databricks and PySpark experience.
- Advanced SQL and data-processing skills.
- Hands-on GCP experience, especially BigQuery.
- Experience with Delta Lake and modern lakehouse architectures.
- Proven ability to design scalable ETL/ELT pipelines and data quality checks.
Responsabilidades
- Design, develop, and optimize scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
- Build and operationalize agentic workflows to automate data engineering and operational processes.
- Integrate agentic capabilities with Databricks, GCP, BigQuery, and Delta Lake environments.
- Develop data pipelines and processing solutions to support new business requirements and datasets.
- Build reusable frameworks and components across data engineering use cases.
- Implement data quality checks, monitoring, and production controls.
- Optimize PySpark and SQL workloads for performance and reliability.
- Support testing, deployment, and productionization of data solutions.
- Troubleshoot complex data issues and implement sustainable solutions.
- Collaborate with teams to identify further automation opportunities.
Conocimientos
Herramientas
Descripción del empleo
AgileEngine, LLC. is seeking a Data Engineer with strong Databricks and GCP experience to develop scalable ETL/ELT pipelines and productionize agentic workflows for data engineering automation.
You will design, build, and optimize data pipelines using Databricks, PySpark, and SQL, while integrating with BigQuery and Delta Lake in production environments. Collaborate across teams to automate data processes and ensure data quality.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
