LinkedIn Created Kafka, But It is Ditching It for Something Better

Fifteen years ago, LinkedIn engineers gave the world Kafka, a resilient, distributed event streaming platform. Subsequently, it was open-sourced and widely adopted by the industry. Notably, the company is trying to replace it. After pushing Kafka to its operational limits while serving over a billion users and processing trillions of events, LinkedIn has unveiled Northguard and Xinfra. These systems are designed to take ordered data and the pattern of separating data producers from data consumers (Pub/Sub) further. Kafka isn’t going away overnight. It still works, and it works well. However, LinkedIn believes that the scale and complexity of its operations have grown too large for its original design.  Why Kafka Couldn’t Keep Up Kafka was originally designed for a much