Data Engineer Databricks | Mid/Senior

Compass UOL; Brasil; Híbrido; BRL 80.000 - 120.000; Tempo integral

Resumo da oferta

A technology company in Brazil is looking for a Data Engineer to manage and optimize data pipelines utilizing tools such as Apache Spark and Azure Data Factory. The successful candidate will have strong skills in SQL, ETL processes, and Python, along with experience in data governance and scalable architecture. The position offers remote work options and requires early morning shifts.

Serviços

Remote work options

Continuous learning opportunities

Qualificações

• Advanced knowledge in SQL and relational databases.
• Experience with ETL workflows.
• Familiarity with Cloud environments and Azure tools.
• Experience in distributed large-scale data processing.
• Knowledge of data governance and modeling.

Responsabilidades

• Optimize data pipelines for real-time analysis.
• Manage data ingestion processes using Azure tools.
• Implement data governance practices to ensure data quality.
• Monitor platform performance and make necessary adjustments.
• Collaborate with teams to identify improvement opportunities.
• Maintain documentation for procedures and workflows.
• Provide support for data infrastructures and resolve issues.

Conhecimentos

SQL

ETL processes

Python

Apache Spark

Azure Data Factory

Fivetran

Airflow

Docker

Kubernetes

Prometheus

Grafana

Azure Monitor

Descrição da oferta de emprego

Job description

We are seeking a Data Engineer to optimize and manage data pipelines, ensuring efficient processing of large volumes of data using tools like Apache Spark, Databricks, and Azure Data Factory. The role involves data ingestion, quality assurance, security, and collaboration with cross-disciplinary teams to implement and maintain scalable data solutions. Responsibilities also include documentation, support, and continuous improvement of data infrastructure.

Main responsibilities

• Optimize data pipelines with Apache Spark and Databricks for real-time and large-scale data analysis.
• Configure and manage data ingestion processes using Azure Data Factory, Fivetran, and other tools from internal and external sources to Data Lake.
• Implement data governance practices to ensure data quality, security, and compliance.
• Monitor platform performance, making adjustments to ensure high scalability and efficiency based on performance indicators.
• Collaborate with business, infrastructure, and data analysis teams to identify improvement opportunities and implement solutions.
• Maintain documentation of procedures, architecture, and workflows to facilitate understanding and continuity.
• Provide support and maintenance for data infrastructures, resolving issues and ensuring operational continuity.

Requirements and skills

Languages and Tools:

• SQL: Advanced knowledge in relational databases and querying.
• ETL processes: Experience with ETL workflows.

Preferred experience in Cloud environments using Azure Data Factory, Fivetran, Apache Spark, and Databricks.

• Python: For automation and data manipulation.
• Apache Spark: Experience in distributed large-scale data processing.
• Azure: Knowledge of Azure Data Lake, Data Factory, Synapse Analytics, among others.
• Fivetran: Experience with data integration and automation from various sources.

Data Orchestration:

• Experience with Airflow and Control-M for pipeline automation.

Containerization and Orchestration:

• Experience with Docker and Kubernetes for scalable data applications.

Data Governance and Architecture:

• Knowledge in data governance, modeling, and scalable architecture implementation.

System Monitoring:

• Experience with tools like Prometheus, Grafana, Azure Monitor for system performance monitoring.

Note: If you do not meet all requirements, we encourage continuous learning and development at Compass UOL.

Additional information

Work shift during the early morning hours, on-call schedule, remote work options.