Senior Data Platform Engineer — Databricks, AI & Streaming

UP.Labs; A distancia; MXN 800,000 - 1,100,000; Jornada completa

Descripción de la vacante

A dynamic venture studio is seeking a Lead Data Engineer to spearhead the design and evolution of its data infrastructure. You will collaborate with Data Scientists and ML Engineers to ensure high-quality data for analytics and AI projects. The role involves building data pipelines using Databricks, Snowflake, and Python while also providing technical leadership within the team. Ideal candidates have extensive experience in data engineering, with significant expertise in cloud environments and data processing technologies.

Formación

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field (or equivalent practical experience).
• 5+ years of experience building and maintaining data pipelines and platforms in production.
• Demonstrated ability to lead technical initiatives and mentor other engineers.
• Hands-on expertise with Databricks for large-scale data processing and analytics.
• Experience working with Snowflake as a cloud data warehouse.
• Advanced SQL skills for complex querying, modeling, and performance optimization.
• Strong proficiency in Python for data engineering workflows.
• Practical experience with dbt for analytics engineering and data transformations.
• Experience building data platforms on Azure.
• Solid knowledge of relational databases, particularly PostgreSQL.
• Familiarity with AWS.
• Experience with Kafka or similar messaging/streaming platforms.
• Strong understanding of Apache Spark for distributed data processing.
• Exposure to GenAI-related data use cases.
• Proven experience collaborating with Data Science and ML teams.
• Strong communication skills.

Responsabilidades

• Design, build, and maintain scalable data pipelines and data platforms using Databricks.
• Develop and manage data models and transformations using dbt and SQL.
• Implement and optimize data workflows across Snowflake and relational databases.
• Build robust ETL/ELT pipelines using Python and Apache Spark.
• Design and operate cloud-native data solutions primarily on Azure.
• Implement and manage streaming data pipelines using Kafka.
• Support GenAI and ML use cases with high-quality data.
• Implement CI/CD best practices for data workflows.
• Monitor and optimize data pipelines for performance and reliability.
• Collaborate with cross-functional teams.
• Provide technical leadership and mentorship.
• Organize work for the team and drive progress.
• Contribute to strategic decisions around data architecture.

Descripción del empleo

A dynamic venture studio is seeking a Lead Data Engineer to spearhead the design and evolution of its data infrastructure. You will collaborate with Data Scientists and ML Engineers to ensure high-quality data for analytics and AI projects. The role involves building data pipelines using Databricks, Snowflake, and Python while also providing technical leadership within the team. Ideal candidates have extensive experience in data engineering, with significant expertise in cloud environments and data processing technologies.