`Error on ingesting samples that are too old or are too far into the futur`

Logs:

Jan  4 20:32:46 box prometheus: level=warn ts=2020-01-04T19:32:46.888Z caller=scrape.go:1170 component="scrape manager" scrape_pool=cpanel target=http://target_host:2089/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=7
Jan  4 20:32:46 box prometheus: level=warn ts=2020-01-04T19:32:46.889Z caller=scrape.go:945 component="scrape manager" scrape_pool=cpanel target=http://target_host:2089/metrics msg="appending scrape report failed" err="out of bounds"

Solution 1

The metrics that its trying to scrape appear to be timestamped (possibly the problem!) and are either too old or too far into the future.

Generally, prometheus metrics don’t include a timestamp.

If you can drop the timestamps, that would solve the problem.
If you can’t drop the timestamps, then correct them so that they’re ~current.

How to solve:

basically the docker containers use UTC time, while the node experter’s target uses local time.
thus need to let containers use local time as well, or rather say they should be the same timezone

# "--restart unless-stopped" 如果希望设置为在宿主机开机后，这个 prometheus Docker instance 也一直运行
$ docker run -d \
--name prometheus \
-v /etc/localtime:/etc/localtime:ro \
-v /etc/timezone:/etc/timezone \
--restart unless-stopped \
-p 9090:9090 \
-v /home/sw/prometheus_config:/etc/prometheus \
prom/prometheus

$ docker run -d \
--restart unless-stopped \
-p 3000:3000 \
--name grafana \
-v /etc/localtime:/etc/localtime:ro \
-v /etc/timezone:/etc/timezone \
grafana/grafana

How to check curreset set timezone and set it

Checking Your Current Timezone

$ cat /etc/timezone
Asia/Singapore
# To get the timezone name and offset, you can use the data command with the +"%Z %z" argument. The uppercase Z prints the timezone name, while the lowercase z outputs the time offset.
$ date +"%Z %z"
$ timedatectl
               Local time: Sat 2022-02-12 12:57:40 +08
           Universal time: Sat 2022-02-12 04:57:40 UTC
                 RTC time: Sat 2022-02-12 04:57:39
                Time zone: Asia/Singapore (+08, +0800)
System clock synchronized: yes
              NTP service: n/a
          RTC in local TZ: no

Set the timezone

To set your timezone the timedatectl command will be used. Before you set your new timezone you may want to know what timezones are available. You can list all available timezones using the timedatectl list-timezones command.

$ timedatectl list-timezones | grep America

# Once you know the time zone you want to set your Ubuntu server to, you use the timedatactl set-timezone command to set it.
$ sudo timedatectl set-timezone America/Toronto

Solution 2

Delete all files under the data folder.

If in a Docker container

# enter the container
$ cd /prometheus
$ rm -rf *

# restart the container

Reference

FEATURED TAGS

algorithm algorithmproblem architecturalpattern architecture aws c# cachesystem codis concurrentcontrol database dataformat datastructure debug design designpattern distributedsystem django docker domain engineering freebsd git golang grafana hackintosh hadoop hardware hexo http hugo ios iot java javaee javascript kafka kubernetes linux linuxcommand linuxio lock macos markdown microservices mysql nas network networkprogramming nginx node.js npm oop openwrt operatingsystem padavan performance programming prometheus protobuf python redis router security shell software-testing spring sql systemdesign truenas ubuntu vmware vpn windows wmware wordpress zookeeper

【Prometheus】Troubleshooting