基于consul 的服务发现 prometheus.yaml ## 操作系统监控 动态发现 - job_name: 'os_system' #metrics_path: "/metrics" consul_sd_configs: - server: 127.0.0.1:8500 services: - os_system scheme: http tags: - "node" relabel_configs: - source_labels: [__meta_consul_tags] regex: .*node.* action: keep - regex: __meta_consul_service_metadata_(.+) action: labelmap 注册服务 方式一 curl ## prometheus 与 consul 配置之间的对应关系 ## tags - tags ## service - name ## label - meta ## curl -X PUT -d '{"id": "node-1","name": "os_system","address": "10.1.x.x","port": 9100, "tags": ["node"],"meta":{&quo……

编译安装Haproxy 2.4 下载预备 sudo wget https://www.haproxy.org/download/2.4/src/haproxy-2.4.15.tar.gz -O /usr/local/src/haproxy-2.4.15.tar.gz sudo wget http://www.lua.org/ftp/lua-5.3.5.tar.gz -O /usr/local/src/lua-5.3.5.tar.gz sudo yum install make gcc build-essential libssl-devel zlib1g-devel pcre3 pcre3-devel systemd-devel readline-devel openssl openssl-devel -y 编译安装 lua cd /usr/local/src && tar -zxvf lua-5.3.5.tar.gz cd /usr/local/src/lua-5.3.5 && make linux src/lua -v haproxy cd /usr/local/src && tar -zxvf haproxy-2.4.15.tar.gz cd /usr/local/src/haproxy-2.4.15 make -j `lscpu |awk 'NR==4{print $2}'` ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_CPU_AFFINITY=1 USE_LUA=1 LUA_INC=/usr/local/src/lua-5.3.5/src/ LUA_LIB=/usr/local/src/lua-5.3.5/src/ PREFIX=/usr/local/haproxy make install PREFIX=/usr/local/haproxy /usr/local/haproxy/sbin/haproxy -v 如果想添加 openssl 支持参考 In order to link OpenSSL statically against HAProxy, first download OpenSSL from https://www.openssl.org/ then build it with the "no-shared" keyword and install it to a local directory, so your system is not affected : $ export STATICLIBSSL=/tmp/staticlibssl $ ./config --prefix=$STATICLIBSSL no-shared……

免密登陆 以下几种PG配置免密的方法。 方法一：设置pg_hab.conf 认证方式为trust #Type database user address method host all postgres 127.0.0.1/32 trust 该方式最不安全，导致通过指定IP连接数据库的连接都可以任意登录数据，毫无安全保障。禁止在生产环境使用。 方法二：设置PG环境变量PGPASSWORD PGPASSWORD是P……

限流目的 清理是在后台运行的维护任务，对用户查询的影响最小。不应该消耗太多的资源(CPU和磁盘I/O)。 清理成本计算 清理过程相当简单，它从数据文件中读取页面(8kB的数据块)，并检查是否需要清理。如果没有dead tuples，页面将被简单地丢弃，而不进行任何更改。否则，它将被清理(……

库内存命中率 SELECT 'index hit rate' AS name, (sum(idx_blks_hit)) / nullif(sum(idx_blks_hit + idx_blks_read),0) AS ratio FROM pg_statio_user_indexes UNION ALL SELECT 'table hit rate' AS name, sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read),0) AS ratio FROM pg_statio_user_tables; 表内存命中率 SELECT relname AS relation, heap_blks_read AS heap_read, heap_blks_hit AS heap_hit, ((heap_blks_hit*100) / NULLIF((heap_blks_hit + heap_blks_read), 0)) AS ratio FROM pg_statio_user_tables order by ratio; 读IO内存占比 磁盘中读取和在内存中直接读取之间的数字和比 SELECT relname AS "relation", heap_blks_read AS heap_read, heap_blks_hit AS heap_hit, ( (heap_blks_hit*100) / NULLIF((heap_blks_hit + heap_blks_read), 0)) AS ratioFROM pg_statio_user_tables; 表中索引命中率 SELECT relname, CASE idx_scan WHEN 0 THEN NULL ELSE round(100.0 * idx_scan / (seq_scan + idx_scan), 5) END……

介绍 类似Linux top 命令 查看数据实时运行情况 https://github.com/dalibo/pg_activity 安装 测试环境centos7 postgresql10 查看已安装的PG版本 如果有9.2 版本，清理,如果没有postgresql10-devel 需要安装 yum list installed | grep postgres 设置环境变量 export PG_HOME=/usr/pgsql-10 export PATH=$PATH:$PG_HOME/bin/ 安装pg_activity python3 -m pip install pg_activity psycopg2-binary 使用 与psql 连接方式相同 pg_activity -U xxx -p xxx 更多……

生成大量文件 mkdir lostfiles cd lostfiles cat create_files.sh for i in $(seq 1 1500000) do echo test >>$i.txt done time sh create_files.sh real 3m44.841s user 0m13.208s sys 3m14.523s 快速删除方式对比 time rsync --delete-before -d /tmp/null/ lostfiles/ real 1m14.937s user 0m1.769s sys 0m54.957s time rm lostfiles/ -rf real 1m4.221s user 0m0.776s sys 0m40.334s time find ./lostfiles/ -type f -delete real 1m0.695s user 0m0.851s sys 0m41.294s time perl -e 'for(<*>){((stat)[9]<(unlink))}' real 1m9.959s user 0m2.955s sys 0m50.202s 应用 快速清除minio bucket 数据 清除bucket数据 time find ./bucket001/ -type f -delete 清除bucket元数据 time find .minio.sys/buckets/bucket001 -type f -delete……

收集数据 sudo perf record -F 99 -p 13204 -g -- sleep 30 -F 99表示每秒99次， -p 13204是进程号 -g表示记录调用栈 sleep 30则是持续30秒 数据解析 perf script -i perf.data &> perf.unfold 数据展现 git clone https://github.com/brendangregg/FlameGraph.git 将perf.unfold中的符号进行折叠 ./stackcollapse-perf.pl perf.unfold &> perf.folded 生成svg图 ./flamegraph.pl perf.folded > perf.svg 用浏览器打开查看 结果解读 https://www.ruanyifeng.com/blog/2017/09/flame-graph.html……

压缩比 xz >biz2 > gzip 安装 默认系统自带 yum install epel-release yum install xz 解压缩 xz -d --threads=`nproc` -k -v hits_v1.tsv.xz 压缩 xz -z --threads=`nproc` -k -v hits_v1.tsv 参数说明 # xz --help Usage: xz [OPTION]... [FILE]... Compress or decompress FILEs in the .xz format. -z, --compress force compression -d, --decompress force decompression -t, --test test compressed file integrity -l, --list list information about .xz files -k, --keep keep (don't delete) input files -f, --force force overwrite of output file and (de)compress links -c, --stdout write to standard output and don't delete input files -0 ... -9 compression preset; default is 6; take compressor *and* decompressor memory usage into account before using 7-9! -e, --extreme try to improve compression ratio by using more CPU time; does not affect……

调研总结 不适用于多个大表的join操作 没有事务保障机制 merage最终一致性 不适合财务相关系统 对数据精度要求不高，实时性有要求的场景比较适合。……