DevOps Lead

Pyramid Consulting, Inc; Leeds; On-site; USD 70,000 - 120,000; Full time

Job summary

An innovative consulting firm is looking for a Kafka Real-Time Architect to lead the design of high-performance data processing systems. In this dynamic role, you'll architect scalable Kafka clusters, ensuring robust real-time analytics and data ingestion. Collaborate with cross-functional teams to integrate Kafka with leading data processing tools and optimize performance. This position offers the chance to stay at the forefront of technology, implementing best practices and security measures while mentoring others. Join a forward-thinking company that values your expertise in shaping the future of data processing.

Qualifications

• Deep understanding of Kafka and related technologies required.
• Proactive approach to addressing business data needs essential.

Responsibilities

• Architect and design scalable, real-time streaming systems using Apache Kafka.
• Integrate Kafka with key data processing tools and platforms.
• Provide technical leadership and mentorship to development teams.

Skills

Apache Kafka

Kafka Streams

Kafka Connect

Apache Spark Streaming

Apache Flink

Apache Beam

Schema Registry

Data Security

Performance Monitoring

Technical Leadership

Job description

5 days ago Be among the first 25 applicants

Direct message the job poster from Pyramid Consulting, Inc

Talent Acquisition UK/Europe at Pyramid IT Consulting

We are seeking an experienced Kafka Real-Time Architect to design and implement scalable, high-performance real-time data processing systems leveraging Apache Kafka. In this role, you will be responsible for architecting and managing Kafka clusters, ensuring system scalability and availability, and integrating Kafka with various data processing platforms. This position requires a deep understanding of Kafka and related technologies, along with a proactive approach to addressing business data needs and ensuring optimal system performance.

Key Responsibilities:

• Architect and design scalable, real-time streaming systems using Apache Kafka, ensuring they are robust, highly available, and meet business requirements for data ingestion, processing, and real-time analytics.
• Configure, deploy, and maintain highly available Kafka clusters, ensuring scalability and fault tolerance across distributed environments. Manage configurations, optimize cluster performance, and troubleshoot issues to maintain smooth operations.
• Integration & Data Processing:
• Integrate Kafka with key data processing tools and platforms, including Kafka Streams, Kafka Connect, Apache Spark Streaming, Apache Flink, Apache Beam, and Schema Registry. This integration will facilitate data stream processing, event-driven architectures, and continuous data pipelines.
• Cross-functional Collaboration:
• Work closely with product and engineering teams to understand data requirements, design and implement solutions, and ensure seamless integration with existing infrastructure and tools. Provide strategic guidance on best practices for real-time data processing.
• Security & Compliance:
• Implement and enforce security measures to protect Kafka clusters and data streams, including encryption, access control, and data integrity. Ensure compliance with organizational and regulatory standards.
• Performance Monitoring & Optimization:
• Proactively monitor Kafka cluster performance, analyze logs, identify bottlenecks, and resolve issues to ensure high availability and low-latency data streaming. Leverage monitoring tools and metrics to enhance decision-making and operational efficiency.
• Technical Leadership & Support:
• Provide technical leadership and mentorship to development and operations teams. Help troubleshoot and resolve Kafka-related issues in production environments. Drive the continuous improvement of Kafka usage, providing insights into new tools, technologies, and practices.
• Staying Current with Industry Trends:
• Keep up to date with the latest Apache Kafka features, updates, and emerging industry best practices. Participate in Kafka-related forums, conferences, and workshops to maintain a cutting-edge understanding of the technology.

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Contract

Job function

• Industries

  IT Services and IT Consulting

Referrals increase your chances of interviewing at Pyramid Consulting, Inc by 2x

Get notified about new DevOps Manager jobs in Leeds, England, United Kingdom.

South Yorkshire, England, United Kingdom 2 weeks ago

Leeds, England, United Kingdom 1 month ago

Leeds, England, United Kingdom 2 weeks ago

South Yorkshire, England, United Kingdom 1 week ago

Leeds, England, United Kingdom 1 week ago

South Yorkshire, England, United Kingdom 1 week ago

South Yorkshire, England, United Kingdom 1 week ago

Leeds, England, United Kingdom $70,000 - $120,000 1 month ago

Leeds, England, United Kingdom 1 week ago

Senior Software Developer (Remote, United Kingdom)

Gildersome, England, United Kingdom 4 days ago

Senior Software Developer (Remote, United Kingdom)

Gildersome, England, United Kingdom 4 days ago

Leeds, England, United Kingdom 4 days ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.