西维蜀黍

【Kafka】Best Practice

Naming

  • Valid characters for Kafka topics are the following:
    • ASCII alphanumerics
    • .
    • _
    • -
  • Avoid using any non-English words.

Retention Time

  • The retention time means how long the msg data will be kept on Kafka. By default it is 3 days and NOT recommended to exceed 7 days.
  • It is recommended to reduce the retention time in case the data size becomes larger.
  ...

Posted by Wei on 2021-11-03, Last Modified on 2025-08-11

【Design Pattern】Concurrency - Copy-on-write Pattern

Copy-on-write

于在传统的使用读写锁以保证互斥操作的实现中，通常会在进行写操作时加锁，因而在写线程操作的时候，其他的读线程都必须等待。

**写时复制（copy on write）**是指多线程对保护对象进行读写操作时，执行写操作的线程在写时，会把该对象复制一份（并对复制后的对象实例进行写修改操作），最终就可以做到写操作进行的同时，不阻塞其他线程的读操作。

当在把对象复制一份完成之后，写操作完成之前，读操作来了，这时候需要先读复制后的对象（因为它是最新的），再读复制前的对象。

写时复制适合读多写少的应用场景，比如白名单，黑名单，商品类目的访问和更新场景，而不适合内存敏感以及对实时性要求很高的场景。

  ...

Posted by Wei on 2021-11-02, Last Modified on 2023-02-22

【Golang】Golang 中的零拷贝

  ...

Posted by Wei on 2021-11-02, Last Modified on 2023-02-20

【Linux】I/O - Page Cache

Page Cache / Buffer Cache

In computing, a page cache, sometimes also called disk cache, is a transparent cache for the pages originating from a secondary storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). The operating system keeps a page cache in otherwise unused portions of the main memory (RAM), resulting in quicker access to the contents of cached pages and overall performance improvements. A page cache is implemented in kernels with the paging memory management, and is mostly transparent to applications.

Usually, all physical memory not directly allocated to applications is used by the operating system for the page cache. Since the memory would otherwise be idle and is easily reclaimed when applications request it, there is generally no associated performance penalty and the operating system might even report such memory as “free” or “available”.

When compared to main memory, hard disk drive read/writes are slow and random accesses require expensive disk seeks; as a result, larger amounts of main memory bring performance improvements as more data can be cached in memory. Separate disk caching is provided on the hardware side, by dedicated RAM or NVRAM chips located either in the disk controller (in which case the cache is integrated into a hard disk drive and usually called disk buffer[3]), or in a disk array controller. Such memory should not be confused with the page cache.

  ...

Posted by Wei on 2021-11-02, Last Modified on 2025-08-11

【Linux】I/O - DMA

Linux I/O读写方式

Linux 提供了轮询、I/O 中断以及 DMA 传输这 3 种磁盘与主存之间的数据传输机制。其中轮询方式是基于死循环对 I/O 端口进行不断检测。I/O 中断方式是指当数据到达时，磁盘主动向 CPU 发起中断请求，由 CPU 自身负责数据的传输过程。 DMA 传输则在 I/O 中断的基础上引入了 DMA 磁盘控制器，由 DMA 磁盘控制器负责数据的传输，降低了 I/O 中断操作对 CPU 资源的大量消耗。

  ...

Posted by Wei on 2021-11-02, Last Modified on 2025-08-11

【Linux】内存布局（memory Layout）

Background

When the program runs, the processing is performed in two spaces called Kernel Space and User Space on the system. The two processing spaces implicitly interfere with each other and the processing of the program proceeds.

  • Kernel Space

The kernel space can be accessed by user processes only through the use of system calls that are requests in a Unix-like operating system such as input/output (I/O) or process creation.

  • User Space

The user space is a computational resource allocated to a user, and it is a resource that the executing program can directly access. This space can be categorized into some segments.

Overview

High Addresses ---> .----------------------.
                    |      Kernel Space    |
                    |----------------------|
                    |                      |   Functions and variable are declared
                    |         STACK        |   on the stack.
base pointer ->     | - - - - - - - - - - -|
                    |           |          |
                    |           v          |
                    :                      :
                    .                      .   The stack grows down into unused space
                    .         Empty        .   while the heap grows up. 
                    .                      .
                    .                      .   (other memory maps do occur here, such 
                    .                      .    as dynamic libraries, and different memory
                    :                      :    allocate)
                    |           ^          |
                    |           |          |
 brk point ->       | - - - - - - - - - - -|   Dynamic memory is declared on the heap
                    |          HEAP        |
                    |                      |
                    |----------------------|
                    |          BSS         |   Uninitialized data (BSS)
                    |----------------------|   
                    |          Data        |   Initialized data (DS)
                    |----------------------|
                    |          Text        |   Binary code
Low Addresses ----> '----------------------'
  ...

Posted by Wei on 2021-11-02, Last Modified on 2025-08-11

【MySQL】Best Practice

Naming

Generic

Do

   ...

Posted by Wei on 2021-11-01, Last Modified on 2025-08-11

【MySQL】Logs - Redo Logs (Write-ahead Logging)

Background

我们都知道，事务的四大特性里面有一个是持久性（durability） ，具体来说就是

  • 只要事务提交成功，那么对数据库做的修改就会被永久保存下来了，不可能因为任何原因再回到原来的状态 。

那么 MySQL 是如何保证一致性的呢？最简单的做法是在每次事务提交的时候，将该事务涉及修改的数据页全部刷新到磁盘中。但是这么做会有严重的性能问题，主要体现在两个方面：

  1. 因为 InnoDB 是以**页（page）**为单位进行磁盘交互的，而一个事务很可能只修改一个数据页里面的几个字节，这个时候将完整的数据页刷到磁盘的话，太浪费资源了！
  2. 一个事务可能涉及修改多个数据页，并且这些数据页在物理上并不连续，因而是随机 I/O（random I/IO） ，随机 I/O写入性能相对很差

因此 MySQL设计了 Redo Log， 具体来说就是只记录事务对数据页做了哪些修改，这样就能在尽可能的提高性能的同时，保证持久性（durability）。注意到，写入 Redo Log 写入位于磁盘中的 Redo Log file 是顺序 I/O（即通过append-only的方式把修改追加到 Redo Log file 中） ，而真正去磁盘更新存储的数据库数据是随机I/O。

  ...

Posted by Wei on 2021-10-31, Last Modified on 2025-08-11

【MySQL】Logs - Relay Logs

A replica server creates several repositories of information to use for the replication process:

  • The replica’s relay log, which is written by the replication I/O (receiver) thread, contains the transactions read from the replication source server’s binary log. The transactions in the relay log are applied on the replica by the replication SQL (applier) thread.
  • The replica’s connection metadata repository contains information that the replication receiver thread needs to connect to the replication source server and retrieve transactions from the source’s binary log. The connection metadata repository is written to the mysql.slave_master_info table.
  • The replica’s applier metadata repository contains information that the replication applier thread needs to read and apply transactions from the replica’s relay log. The applier metadata repository is written to the mysql.slave_relay_log_info table.
  ...

Posted by Wei on 2021-10-31, Last Modified on 2025-08-11

【MySQL】搭建基于 Binlog Position 的主从复制

Demo

在我的demo中，我是在docker 中创建了两个docker container。

# 这里：
# 1. --restart=unless-stopped：container 会自动启动
# 2. MYSQL_ROOT_HOST=%：root用户可以在任何host上登录
# 3. MYSQL_ROOT_PASSWORD=1234：root 密码为1234
# 4. 3307:3306：将本机的 3307 映射到 container 内部的 3306
$ docker run --name sw-mysql1 --restart=unless-stopped -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=1234 -p 3307:3306 -d mysql

$ docker run --name sw-mysql2 --restart=unless-stopped -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=1234 -p 3308:3306 -d mysql

值得一提的是，这两个container在默认情况下并不能相互通讯

$ docker network create myNetwork
$ docker network connect myNetwork sw-mysql1
$ docker network connect myNetwork sw-mysql2
  ...

Posted by Wei on 2021-10-30, Last Modified on 2023-03-29

FEATURED TAGS
algorithm algorithmproblem architecturalpattern architecture aws blockchain c# cachesystem codis compile concurrentcontrol database dataformat datastructure debug design designpattern distributedsystem django docker domain engineering freebsd git golang grafana hackintosh hadoop hardware hexo http hugo ios iot java javaee javascript kafka kubernetes linux linuxcommand linuxio lock macos markdown microservices mysql nas network networkprogramming nginx node.js npm oop openwrt operatingsystem padavan performance programming prometheus protobuf python redis router security shell software testing spring sql systemdesign truenas ubuntu vmware vpn windows wmware wordpress xml zookeeper
ABOUT ME
avatar

🐒 Software engineer | 📷 Photographer | 👹 Urban explorer

FRIENDS