DATA ENGINEER

Job Responsibilities

• Lead the design, development, and maintenance of scalable data pipelines and systems according to the customers’ needs.
• Collaborate with cross-functional teams to understand data requirements and deliver solutions.
• Ensure the performance and security of the data infrastructure and data quality/integrity, following best practices of Data Engineering.
• Optimize data workflows for performance and efficiency to enrich and transform large volumes of data with complex business rules.
• Automate data pipelines and streamline data ingestion through the implementation of different orchestrators and scheduling processes (mainly Airflow as a Service).
• Design and implement scalable and secure data processing pipelines using Scala, Spark, and Cloud Object Storage (COS).
• Set up CI/CD pipelines to automate deployments, unit testing, and development management.
• Write and conduct unit and validation tests to ensure accuracy and integrity of developed code.
• Write technical documentation (specifications and operational documents) to ensure knowledge capitalization.
• Support different teams in troubleshooting and resolving data-related issues.
• Assure the creation and maintenance of comprehensive documentation of data pipelines, systems, and processes, being accessible to team members.
• Communicate effectively with team members and stakeholders to ensure alignment on data initiatives.
• Provide technical guidance and mentorship to junior data engineers.
• Foster a culture of continuous learning and improvement within the team.

Qualifications

Good knowledge of:

• Spark on Scala
• S3 storage/COS and Parquet format
• Strong knowledge of SQL and NoSQL databases
• Hadoop

Full understanding of:

• Shell scripting
• Software Development Lifecycle (SDLC) awareness
• Agile principles and ceremonies
• Elasticsearch and Kibana

Optionally / as a plus:

• HVault
• Dremio as a tool to virtualize data

Additional Information

Business and Transversal Skills:

• Knowledge of banking industry and processes
• Excellent problem-solving skills and attention to detail
• Decision-making ability
• Business/IT relationship (including IT OPS)
• Strong communication and interpersonal skills
• Ability to understand, explain, and support change