Data Engineer - Databricks, PySpark on GCP
AgileEngine, LLC.
Una candidatura hecha para este puesto de trabajo — un currículum y una carta de presentación adaptados que responden directamente a la oferta.
Ventajas ofrecidas por este puesto de trabajo
Descripción de la vacante
AgileEngine, LLC. is seeking a Data Engineer with strong Databricks and GCP experience to design and productionize scalable ETL/ELT pipelines and agentic workflows for data automation.
You will build reusable data frameworks, ensure data quality, and optimize workloads across Databricks, PySpark, SQL, and BigQuery in a modern data lake environment. The role emphasizes collaboration with cross-functional teams to deliver robust data solutions and automation opportunities, while supporting
Formación
- 5+ years hands-on Databricks and PySpark experience.
- Advanced SQL and data-processing skills.
- Hands-on GCP experience, especially BigQuery.
- Experience with Delta Lake and modern data lake architectures.
- Strong ETL/ELT design and data quality focus.
- Proven ability to build scalable, production-grade data solutions.
- Strong analytical and troubleshooting skills.
- Familiarity with software engineering practices (testing, version control, deployment, monitoring).
Responsabilidades
- Design, develop, and optimize scalable ETL/ELT data pipelines using Databricks, PySpark, and SQL.
- Build and operationalize agentic workflows to automate data engineering and operational processes such as data validation, issue identification, troubleshooting, and workflow execution.
- Integrate agentic capabilities with existing Databricks, GCP, BigQuery, and Delta Lake environments.
- Develop data pipelines and processing solutions to support new business requirements and datasets.
- Build reusable frameworks and components across data engineering and business use cases.
- Implement data quality checks, monitoring, validation, exception handling, and production controls.
- Optimize PySpark and SQL workloads for performance, reliability, and scalability.
- Support testing, deployment, productionization, and ongoing enhancement of data and agentic solutions.
- Troubleshoot complex data and production issues and implement sustainable solutions.
- Collaborate with business, data engineering, and platform teams to identify automation opportunities.
Conocimientos
Herramientas
Descripción del empleo
AgileEngine, LLC. is seeking a Data Engineer with strong Databricks and GCP experience to design and productionize scalable ETL/ELT pipelines and agentic workflows for data automation.
You will build reusable data frameworks, ensure data quality, and optimize workloads across Databricks, PySpark, SQL, and BigQuery in a modern data lake environment. The role emphasizes collaboration with cross-functional teams to deliver robust data solutions and automation opportunities, while supporting
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
