Senior Databricks Engineer — Hybrid, Growth & Impact
AgileEngine, LLC.
Una candidatura completa en un minuto: currículum y carta de presentación adaptados, listos para enviar.
Ventajas ofrecidas por este puesto de trabajo
Descripción de la vacante
AgileEngine is seeking a Senior Data Engineer to build batch and streaming data pipelines on Databricks using PySpark and Delta Lake. This role migrates legacy data warehouse workloads onto a governed Lakehouse, using a medallion architecture to power analytics and AI use cases.
You will collaborate with DevOps and analytics engineers, optimize Spark jobs, implement data quality and governance with Unity Catalog, and write clean Python and SQL.
Formación
- 3+ years of data engineering experience with Spark and cloud data architectures.
- Proficient in Databricks, PySpark, and Delta Lake for batch/streaming pipelines.
- Strong SQL and Python with data modeling across Lakehouse patterns.
- Experience with Structured Streaming, Auto Loader, Kafka or Event Hubs.
Responsabilidades
- Build and operate batch and streaming pipelines on Databricks using PySpark and Delta Lake.
- Model a medallion architecture to serve analytics and ML consumers.
- Migrate legacy ETL workloads to a governed Lakehouse with data parity.
- Write clean Python and SQL; ensure tests and docs are maintained.
- Optimize Spark jobs and Delta tables for performance and cost.
- Implement data quality, lineage, and governance with Unity Catalog.
Conocimientos
Herramientas
Descripción del empleo
AgileEngine is seeking a Senior Data Engineer to build batch and streaming data pipelines on Databricks using PySpark and Delta Lake. This role migrates legacy data warehouse workloads onto a governed Lakehouse, using a medallion architecture to power analytics and AI use cases.
You will collaborate with DevOps and analytics engineers, optimize Spark jobs, implement data quality and governance with Unity Catalog, and write clean Python and SQL.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
