Description
Your role will involve collaboration with cross-functional teams, leveraging cutting‑edge technologies, and ensuring scalable, efficient, and secure data engineering practices.
Description
Your role will involve collaboration with cross-functional teams, leveraging cutting‑edge technologies, and ensuring scalable, efficient, and secure data engineering practices.
Responsibilities
- Provide Technical Leadership: Offer technical leadership to ensure clarity between ongoing projects and facilitate collaboration across teams to solve complex data engineering challenges.
- Build and Maintain Data Pipelines: Design, build, and maintain scalable, efficient, and reliable data pipelines to support data ingestion, transformation, and integration across diverse sources and destinations, using tools such as Kafka, Databricks, and similar toolsets.
- Drive Innovation: Leverage innovative technologies and approaches to modernize and extend core data assets, including SQL-based, NoSQL-based, cloud-based, and real-time streaming data platforms.
- Implement Automated Testing: Design and implement automated unit, integration, and performance testing frameworks to ensure data quality, reliability, and compliance with organizational standards.
- Optimize Data Workflows: Optimize data workflows for performance, cost efficiency, and scalability across large datasets and complex environments.
- Mentor Team Members: Mentor team members in data principles, patterns, processes, and practices to promote best practices and improve team capabilities.
- Draft and Review Documentation: Draft and review architectural diagrams, interface specifications, and other design documents to ensure clear communication of data solutions and technical requirements.
Requirements
- Bachelor’s degree typically in Computer Science, Management Information Systems, Mathematics, Business Analytics or another STEM degree.
- Proven experience working within cross-functional teams and managing complex projects from inception to completion
- 4+ years of professional Data Development experience.
- 3+ years of experience with SQL and NoSQL technologies.
- 3+ years of experience building and maintaining data pipelines and workflows.
- 3+ years of experience developing with Python & PySpark.
- Experience developing within databricks.
- Experience applying object-oriented design patterns and principles.
- Experience with CI/CD pipelines and processes.
- Experience with automated unit, integration, and performance testing.
- Experience with version control software such as Git.
- Full understanding of ETL and Data Warehousing concepts.
- Strong understanding of Agile principles (Scrum).
Preferred Qualifications
- Knowledge of Structured Streaming (Spark, Kafka, EventHub, or similar technologies).
- Experience with GitHub SaaS/GitHub Actions.
- Experience with Service Oriented Architecture.
- Experience with containerization technologies such as Docker and Kubernetes.