Senior Databricks Engineer: Lakehouse & AI Analytics
AgileEngine, LLC.
Consigue una respuesta de este empleador — un currículum y una carta de presentación adaptados exactamente a lo que busca para contratar.
Ventajas ofrecidas por este puesto de trabajo
Descripción de la vacante
AgileEngine, LLC. is seeking a Senior Data Engineer to design and operate batch and streaming pipelines on Databricks using PySpark and Delta Lake. The role focuses on migrating legacy ETL workloads to a governed Lakehouse with medallion architecture to power analytics and ML use cases.
You will work with SQL, Python, Unity Catalog governance, and collaborate with DevOps and analytics engineers to ensure observability, security, and data quality.
Formación
- 3+ years in data engineering with Spark and cloud data architectures.
- Strong SQL and Python; data modeling across Lakehouse patterns.
- Experience with streaming and batch data pipelines on Databricks.
Responsabilidades
- Design, build, and operate batch and streaming pipelines on Databricks (PySpark, Delta Lake).
- Model medallion architecture for analytics, reporting, and ML users.
- Migrate ETL workloads to Lakehouse with minimal disruption.
- Write tests, documentation, and generate code scaffolding with AI tools.
- Optimize Spark jobs and Delta tables for performance and cost.
Conocimientos
Herramientas
Descripción del empleo
AgileEngine, LLC. is seeking a Senior Data Engineer to design and operate batch and streaming pipelines on Databricks using PySpark and Delta Lake. The role focuses on migrating legacy ETL workloads to a governed Lakehouse with medallion architecture to power analytics and ML use cases.
You will work with SQL, Python, Unity Catalog governance, and collaborate with DevOps and analytics engineers to ensure observability, security, and data quality.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
