Senior Databricks Engineer

Infinite Computer Solutions; Colombia; Presencial; COP 90.000.000 - 130.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Health insurance

Descripción de la vacante

Infinite Computer Solutions is seeking a Senior Data Engineer to define data requirements, build and optimize pipelines, and collaborate with data science and BI teams in Colombia. You will mentor junior engineers and drive data quality and analytics initiatives.

The role focuses on scalable data infrastructure, cloud platforms (AWS/GCP), and strong SQL/Python skills to enable insights and machine learning workflows.

Formación

• 7+ years of IT experience in data engineering/ETL roles.
• 5+ years in data engineering/ETL handling large datasets.
• 3+ years with Big Data tools (Hadoop, Spark, Kafka, Hive, S3) or cloud platforms (AWS/GCP).
• Databricks UI, Delta Lake, Python/Spark SQL proficiency.
• High-velocity streaming with Kafka and Spark Streaming.

Responsabilidades

• Define data requirements, wrangle large structured and unstructured data.
• Ingest, analyze, validate, normalize and clean data for BI and ML use.
• Develop data policies and retention models; anonymize data as needed.
• Implement data quality procedures and support analytics teams.
• Design and support scalable data infrastructure and pipelines.
• Collaborate with data science and BI teams on models and reports.
• Mentor junior data engineers and lead data engineering projects.
• Present insights and strategic recommendations to leadership.

Conocimientos

SQL

Python

PySpark

Spark SQL

Databricks

Kafka

ETL Development

Data Pipelines

Cloud AWS

Hadoop

GIT

Delta Lake

Spark Streaming

SQL Optimization

Tibco

IBM DataStage

Educación

Bachelor’s Degree

Herramientas

Databricks

Hadoop

Spark

Kafka

IBM DataStage

Delta Lake

Teradata

BigQuery

AWS

GCP

S3

Kinesis

Glue

TIBCO EMS

IBM MQ

Descripción del empleo

• Defines data requirements, gather, and wrangle large scale of structured and unstructured data, and validate data by running various data tools in the Data Environment.
• Supports the standardization, customization, and ad-hoc data analysis, and will develop the mechanisms to ingest, analyze, validate, normalize and clean data.
• Creates data policy and develop interfaces and retention models which requires synthesizing or anonymizing data.
• Implements statistical data quality procedures on new data sources, and by applying rigorous iterative data analytics, supports Data Scientists and analytics and insights creation in data sourcing and preparation to visualize data and synthesize insights of commercial value.
• Develops and maintains data engineering best practices and contributes to Insights on data analytics and visualization concepts, methods and techniques.
• Works closely with the data science and business intelligence teams to develop data models and pipelines for research, reporting, and machine learning.
• Design, implement, and support scalable data infrastructure solutions to integrate with multi-heterogeneous data sources, aggregate and retrieve Big Data in a fast and safe mode, curate data that can be used in BI reporting, analysis, machine learning models and ad-hoc data requests.
• Build data pipelines that clean, transform, and aggregate data from disparate sources.
• Engages with business teams to gather requirements and design data solutions.
• Mentors team of more Junior Data Engineers.
• Collaborates across multiple projects to provide data engineering expertise across teams.
• Analyzes most relevant insights and shares with leadership to provide strategic recommendations for the business
• Lead a team of data engineers and act as a key senior contributor to a data engineering project.

Skills and Experience:

• 7+ years of overall IT experience
• 5+ years of experience in a data engineering/ETL role with a track record of manipulating, processing, and extracting value from large datasets
• 3+ years of experience with Big Data tools/technologies like Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS, or Cloud platforms e.g. AWS, GCP, etc.
• 3+ years building, testing, and optimizing data ingestion pipelines, architectures, and data sets with Tibco, IBM or others.
• Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog.
• High-velocity high-volume stream processing with Apache Kafka and Spark Streaming.
• Strong SQL skills with ability to write intermediate complexity queries.
• ETL experience with PySpark, Spark SQL , IBM Data Stage or similar.
• Agile Scrum, Kanban or SAFe experience.
• Databricks, Python (and/or Scala) and PySpark/Scala-Spark.
• Database solutions like Databricks, Teradata, Mainframe, DB2 or BigQuery.
• Azure, AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue.
• Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series.
• Strong SQL skills with ability to write intermediate complexity queries
• Experience with GIT code versioning software

Education

Bachelor’s Degree required.Preferably in Information Systems, Computer Science, Computer Information Systems or related field