HA and Scalability
- Partitions can have copies to increase durability and availability and enable Kafka to failover to a broker with a replica of the partition if the broker with the leader partition fails. This is called the Replication Factor and can be 1 or more.
The following diagrams (from the insidebigdata series we published last year on Apache Kafka architecture) illustrate how Kafka partitions and leaders/followers work for a simple example (1 topic and 4 partitions), enable Kafka write scalability (including replication), and read scalability:
Reference
- Kafka The definition Guide
- https://www.instaclustr.com/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/
- https://www.confluent.io/blog/okay-store-data-apache-kafka/
- https://stackoverflow.com/questions/38013266/apache-kafka-persist-all-data
- https://towardsdatascience.com/using-kafka-as-a-temporary-data-store-and-data-loss-prevention-tool-in-the-data-lake-5472f2b586e
FEATURED TAGS
algorithm
algorithmproblem
architecturalpattern
architecture
aws
c#
cachesystem
codis
concurrentcontrol
database
dataformat
datastructure
debug
design
designpattern
distributedsystem
django
docker
domain
engineering
freebsd
git
golang
grafana
hackintosh
hadoop
hardware
hexo
http
hugo
ios
iot
java
javaee
javascript
kafka
kubernetes
linux
linuxcommand
linuxio
lock
macos
markdown
microservices
mysql
nas
network
networkprogramming
nginx
node.js
npm
oop
openwrt
operatingsystem
padavan
performance
programming
prometheus
protobuf
python
redis
router
security
shell
software-testing
spring
sql
systemdesign
truenas
ubuntu
vmware
vpn
windows
wmware
wordpress
zookeeper