西维蜀黍

【Prometheus】Prometheus - Node Exporter

安装Node Exporter

在Prometheus的架构设计中,Prometheus Server并不直接服务监控特定的目标,其主要任务负责数据的收集,存储并且对外提供数据查询支持。因此为了能够能够监控到某些东西,如主机的CPU使用率,我们需要使用到Exporter。Prometheus周期性的从Exporter暴露的HTTP服务地址(通常是/metrics)拉取监控样本数据。

xporter可以是一个相对开放的概念,其可以是一个独立运行的程序独立于监控目标以外,也可以是直接内置在监控目标中。只要能够向Prometheus提供标准格式的监控样本数据即可。

这里为了能够采集到主机的运行指标如CPU, 内存,磁盘等信息。我们可以使用Node Exporter

Node Exporter同样采用Golang编写,并且不存在任何的第三方依赖,只需要下载,解压即可运行。可以从 https://prometheus.io/download/ 获取最新的node exporter版本的二进制包。

  ...


【Prometheus】Prometheus 初入

Prometheus

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

Prometheus’s main distinguishing features as compared to other monitoring systems are:

  • a multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
  • a flexible query language to leverage this dimensionality
  • no dependency on distributed storage; single server nodes are autonomous
  • timeseries collection happens via a pull model over HTTP
  • pushing timeseries is supported via an intermediary gateway
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboarding support
  • support for hierarchical and horizontal federation

Architecture overview

Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be used to visualize the collected data.

强大的查询语言PromQL

Prometheus内置了一个强大的数据查询语言PromQL。 通过PromQL可以实现对监控数据的查询、聚合。同时PromQL也被应用于数据可视化(如Grafana)以及告警当中。

通过PromQL可以轻松回答类似于以下问题:

  • 在过去一段时间中95%应用延迟时间的分布范围?
  • 预测在4小时后,磁盘空间占用大致会是什么情况?
  • CPU占用率前5位的服务有哪些?(过滤)

ref https://swsmile.info/post/prometheus-promql/ for more details.

  ...


【Grafana】配置

Grafana

grafana是一个非常酷的数据可视化平台,常常应用于显示监控数据。

配置数据源(Data Sources)

本篇主要介绍了Grafana基于MySQL数据源的安装及常用姿势,也支持其他数据源如ElasticSearch、InfluxDB、Zabbix、Prometheus、Druid、influxDb、MySQL、graphite等。

Supported data sources

The following data sources are officially supported:

  ...


【Software Testing】App 自动化测试框架 - Appium

Background

XCUITest Driver

Appium’s primary support for automating iOS apps is via the XCUITest driver. (New to Appium? Read our introduction to Appium drivers). This driver leverages Apple’s XCUITest libraries under the hood in order to facilitate automation of your app . This access to XCUITest is mediated by the WebDriverAgent server.

WebDriverAgent

WebDriverAgent (also referred to as “WDA”) is a project managed by Facebook, to which the Appium core team contributes heavily. WDA is a WebDriver-compatible server that runs in the context of an iOS simulator or device and exposes the XCUITest API. Appium’s XCUITest driver manages WDA as a subprocess opaque to the Appium user, proxies commands to/from WDA, and provides a host of additional functionality (like simulator management and other methods, for example).

WebDriver Protocol

When all is said and done, Appium is just an HTTP server. It sits and waits for connections from a client, which then instructs Appium what kind of session to start and what kind of automation behaviors to enact once a session is started. This means that you never use Appium just by itself. You always have to use it with a client library of some kind (or, if you’re adventurous, cURL!).

Luckily, Appium speaks the same protocol as Selenium, called the WebDriver Protocol. You can do a lot of things with Appium just by using one of the standard Selenium clients. You may even have one of these on your system already. It’s enough to get started, especially if you’re using Appium for the purpose of testing web browsers on mobile platforms.

  ...


【Software Testing】App 自动化测试框架

iOS

对于 iOS 9.2 及更低版本,苹果唯一的自动化技术被称为UIAutomation,它运行在 Instruments 中。从 iOS 10 开始,苹果已经完全删除了 UIAutomation 工具。

同时,苹果推出了一款名为XCUITest 的新型自动化技术,从 iOS 9.3iOS 10 及以上版本,这将是苹果唯一支持的自动化框架。

  ...


【Protobuf】Protocol Buffers 2中使用map

Maps

If you want to create an associative map as part of your data definition, protocol buffers provides a handy shortcut syntax:

map<key_type, value_type> map_field = N;

…where the key_type can be any integral or string type (so, any scalar type except for floating point types and bytes). Note that enum is not a valid key_type. The value_type can be any type except another map.

So, for example, if you wanted to create a map of projects where each Project message is associated with a string key, you could define it like this:

map<string, Project> projects = 3;

The generated map API is currently available for all proto2 supported languages. You can find out more about the map API for your chosen language in the relevant API reference.

  ...


【Cache System】Redis Cluster

Redis Cluster

Redis Cluster 是Redis3.0之后,官方提供的Redis集群解决方案,由 Redis 官方团队来实现。

在3.0之前,为了解决容量高可用用方面的需求,基本上只能通过客户端分片 + Redis sentinel或者添加 proxy(twemproxy、codis)方案解决。

Redis Cluster最大的特性,就是对Redis集群提供了水平扩展的能力(horizential scalibility),即当整个 Redis 集群出现存储容量或者性能 bottleneck 时,使用 Redis Cluster 可以通过增加新的 master Redis node从而快速解决bottleneck。

除此之外,Redis Cluster提供了强大的高可用机制(即 failure failover)。即当集群中任何一个master Redis node无法正常工作时(比如因为底层依赖的硬件故障、网络问题),它对应的 slave Redis node就会自动代替它(当然,这里有一个前提,是我们设置了slave Redis node)。

与Codis 和 Twemproxy 不同的是:Redis Cluster并非使用Porxy模式来连接集群节点,而是使用无中心节点的模式来组建集群(即没有coordinator)。

Redis Cluster实现在多个节点之间进行数据共享,即使部分节点失效或者无法进行通讯时,Cluster仍然可以继续处理请求。若每个主节点都有一个从节点支持,在主节点下线或者无法与集群的大多数节点进行通讯的情况下, 从节点提升为主节点,并提供服务,保证Cluster正常运行。

  ...


【Cache System】Redis 集群方案

Redis 集群方案

Redis 的集群解决方案有社区的,也有官方的,社区的解决方案有 CodisTwemproxy

Codis

Codis 由我国的豌豆荚团队开源

Twemproxy

Twemproxy 由Twitter团队开源。

  ...


【Redis】Codis Pipeline

codis proxy 处理 “pipeline” 的逻辑:只是在从client 到proxy 有效率的提升(因为使用一次 pipeline意味着只有一次RTT)。而从 proxy 到 redis node的这个过程,依然是对 pipeline 中的每个命令串行地分别进行处理,即使这些命令对应的key位于不同的slot。

By right,可以实现成当这些命令对应的key位于不同的slot,以slot为单位异步地去执行,以如果有两个命令,他们对应的key位于不同的slot,则这两条命令会被并行的去执行,而当他们对应的key位于同一个slot,则自然而然地串行的去执行(因为Redis是单线程的)。

  ...


【Mattermost】webhook

Incoming Webhooks

Mattermost supports webhooks to easily integrate external applications into the server.

Use incoming webhooks to post messages to Mattermost public channels, private channels and direct messages. Messages are sent via an HTTP POST request to a Mattermost URL generated for each application and contain a specifically formatted JSON payload in the request body.

  ...