Sr. Data Engineer Azure Databricks

Winnin; São Paulo; Teletrabalho; BRL 80.000 - 120.000; Tempo integral

Resumo da oferta

An innovative firm is seeking a Senior Data Engineer to lead the design and maintenance of data infrastructures for analytics in the Aviation Industry. This remote contract role emphasizes building scalable data solutions using Python, SQL, and Azure technologies. The ideal candidate will thrive in an Agile environment, mentoring junior staff while collaborating across multi-disciplined teams. Responsibilities include architecting data systems, ensuring data quality, and optimizing performance. Join a forward-thinking company where your expertise will drive impactful data solutions and contribute to the evolution of data engineering practices.

Qualificações

• Strong background in data engineering with expertise in Python, SQL, and Azure.
• Experience in building and maintaining data architectures for large-scale applications.

Responsabilidades

• Design and maintain complex data architectures for integration and analytics.
• Collaborate with teams to develop data solutions and ensure data quality.

Conhecimentos

Python

SQL

PySpark

Azure

Databricks

DevOps

Data Quality

Performance Optimization

Agile Methodologies

Ferramentas

Azure Data Lake

Apache Spark

Descrição da oferta de emprego

This is a remote, contract position responsible for designing, building, and maintaining the infrastructure required for data integration, storage, processing, and analytics (BI, visualization and Advanced Analytics).

We are looking for a skilled Senior Data Engineer with a strong background in Python, SQL, PySpark, Azure, Databricks, Synapse, Azure Data Lake, DevOps and cloud-based large scale data applications with a passion for data quality, performance and cost optimization. The ideal candidate will develop in an Agile environment, contributing to the architecture, design, and implementation of Data products in the Aviation Industry, including migration from Synapse to Azure Data Lake. This role involves hands-on coding, mentoring junior staff and collaboration with multi-disciplined teams to achieve project objectives.

Responsibilities

1. Architect, design, develop, test and maintain high-performance, large-scale, complex data architectures, which support data integration (batch and real-time, ETL and ELT patterns from heterogeneous data systems: APIs and platforms), storage (data lakes, warehouses, data lake houses, etc), processing, orchestration and infrastructure. Ensuring the scalability, reliability, and performance of data systems, focusing on Databricks and Azure.
2. Contribute to detailed design, architectural discussions, and customer requirements sessions.
3. Actively participate in the design, development, and testing of big data products.
4. Construct and fine-tune Apache Spark jobs and clusters within the Databricks platform.
5. Migrate out of Azure Synapse to Azure Data Lake or other technologies.
6. Assess best practices and design schemas that match business needs for delivering a modern analytics solution (descriptive, diagnostic, predictive, prescriptive).
7. Design and implement data models and schemas that support efficient data processing and analytics.
8. Design and develop clear, maintainable code with automated testing using Pytest, unittest, integration tests, performance tests, regression tests, etc.
9. Collaborate with cross-functional teams and Product, Engineering, Data Scientists and Analysts to understand data requirements and develop data solutions, including reusable components meeting product deliverables.
10. Evaluate and implement new technologies and tools to improve data integration, data processing, storage and analysis.
11. Evaluate, design, implement and maintain data governance solutions: cataloging, lineage, data quality and data governance frameworks that are suitable for a modern analytics solution, considering industry-standard best practices and patterns.
12. Continuously monitor and fine-tune workloads and clusters to achieve optimal performance.
13. Provide guidance and mentorship to junior team members, sharing knowledge and best practices.
14. Maintain clear and comprehensive documentation of the solutions, configurations, and best practices implemented.
15. Promote and enforce best practices in data engineering, data governance, and data quality.
16. Ensure data quality and accuracy.
17. Design, implement and maintain data security and privacy measures.
18. Be an active member of an Agile team, participating in all ceremonies and continuous improvement activities, being able to work independently as well as collaboratively.