概述

可以通过 rest api 添加或删除后端服务器，实现 nginx 后端服务器的动态更新

背景:

• 服务实例的 ip 会频繁更换, 需要想办法让发布后的实例 ip 自动更新到 nginx 的配置中去,并且能够让其自动生效
• nginx reload 会重新开一组进程来接收请求，让老进程完事后再退出. 切记如果你线上流量较大，不要频繁地 reload,因为 nginx 新老配置交替的时候会掉 15%左右的 qps; nginx reload 也会造成 nginx 与 client 和 backend 的长链接关闭

组件版本

• nginx-1.20.1
• etcd-3.4.16
• nginx-upsync-module_v2.1.3 当前仅支持 etcd V2 API, 从 consul 或 etcd 同步后端服务器配置，动态修改后端服务器的属性, 不需要重新加载 nginx
• nginx_upstream_check_module: 主动检查后端服务器是否健康

Consul

1、二进制方式部署

# 下载预编译二进制安装
wget https://releases.hashicorp.com/consul/1.8.15/consul_1.8.15_linux_amd64.zip
unzip consul_1.8.15_linux_amd64.zip
mv consul /usr/local/bin/

2、创建运行环境及启动脚本

# 创建运行用户及相关目录权限
useradd -M -s /sbin/nologin consul && mkdir /var/lib/consul/ && chown -R consul.consul /var/lib/consul/
mkdir /var/run/consul && chown -R consul.consul /var/run/consul/
mkdir /etc/consul

# 启动脚本
cat /etc/systemd/system/consul.service
[Unit]
Description=Consul service discovery agent
Requires=network-online.target
After=network-online.target

[Service]
User=consul
Group=consul
PIDFile=/var/run/consul/consul.pid
EnvironmentFile=-/etc/default/consul
Environment=GOMAXPROCS=2
Restart=on-failure
ExecStart=/usr/local/bin/consul agent -dev -bind=192.168.31.69 -client 0.0.0.0 -config-dir /etc/consul/ -data-dir=/var/lib/consul -pid-file=/var/run/consul/consul.pid
ExecReload=/usr/local/bin/consul reload
KillSignal=SIGTERM
TimeoutStopSec=5

[Install]
WantedBy=multi-user.target

3、启动 consul

systemctl start consul
# consul web 界面:
http://192.168.31.69:8500/ui

Etcd

1、下载解压安装

wget https://github.com/coreos/etcd/releases/download/v3.4.16/etcd-v3.4.16-linux-amd64.tar.gz
tar xzf etcd-v3.4.16-linux-amd64.tar.gz
mv etcd-v3.4.16-linux-amd64 /srv/etcd && cd /srv/etcd

2、设置工作及配置目录

mkdir -p /var/lib/etcd/ && mkdir -p /srv/etcd/config/
cat <<EOF | sudo tee /srv/etcd/config/etcd.conf
#节点名称
ETCD_NAME=$(hostname -s)
# 数据存放位置
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_PEER_URLS=http://192.168.31.69:2380
ETCD_LISTEN_CLIENT_URLS=http://192.168.31.69:2379
ETCD_ADVERTISE_CLIENT_URLS=http://192.168.31.69:2379
EOF

3、设置 systemd 启动脚本

; 添加运行用户、文件目录
useradd -M -r -s /sbin/nologin etcd
mkdir /var/lib/etcd/ && chown etcd.etcd /var/lib/etcd/ -R
; 启动脚本
cat <<EOF | sudo tee /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
Documentation=https://github.com/coreos/etcd
After=network.target

[Service]
Type=notify
User=etcd
Group=etcd
EnvironmentFile=-/srv/etcd/config/etcd.conf
ExecStart=/srv/etcd/etcd --enable-v2=true
ExecReload=/bin/kill -HUP \$MAINPID
Restart=on-failure
RestartSec=10s
KillMode=process
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target
EOF

4、启动 systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

Nginx

1、下载扩展模块 nginx-upsync-module

yum -y install gcc-c++ zlib-devel openssl-devel pcre-devel gd-devel patch
wget https://github.com/weibocom/nginx-upsync-module/archive/v2.1.3.tar.gz -O nginx-upsync-module_v2.1.3.tar.gz
wget https://github.com/xiaokai-wang/nginx_upstream_check_module/archive/refs/heads/master.zip
wget http://nginx.org/download/nginx-1.20.1.tar.gz
tar xf nginx-1.20.1.tar.gz && cd nginx-1.20.1 && unzip master.zip

# 编译之前部分模块需要先给 nginx 打补丁
patch -p1 < patch -p1 < /root/src/nginx_upstream_check_module-master/check_1.12.1+.patch

./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' \
--add-dynamic-module=/root/src/nginx-upsync-module-2.1.3 \
--add-module=/root/src/nginx_upstream_check_module-master

# nginx 编译选项说明:
# --add-module    静态加载, 编译后需要替换正式环境中的二进制文件 nginx
# --add-dynamic-module 动态加载, 将模块编译为 objs/*.so(make 编译后即可生成) 然后在 nginx 的配置文件中加载就行, 不需要替换 nginx

# 编译安装 nginx_upstream_check_module
make && make install

# 添加运行用户及必要的文件目录信息
useradd -M -r -s /sbin/nologin nginx && mkdir /etc/nginx/conf.d/ && touch /etc/nginx/conf.d/servers
mkdir /var/cache/nginx/client_temp -p

2、nginx 配置文件主要部分

load_module "/usr/lib64/nginx/modules/ngx_http_upsync_module.so";

# http 模块下
upstream test {
    # 后端使用 consul 存储
    upsync 192.168.31.69:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;

    # 后端使用 etcd 存储
    #upsync 192.168.31.69:2379/v2/keys/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=etcd strong_dependency=off;

    upsync_dump_path /etc/nginx/conf.d/servers;
    include /etc/nginx/conf.d/servers;

    check interval=5000 rise=2 fall=3 timeout=3000 type=http;
    check_http_send "HEAD /test HTTP/1.0\r\n\r\n";
    check_http_expect_alive http_2xx http_3xx;
}

server {
    listen 80;
    server_name c22.it123.com;

    location /test {
        proxy_pass http://test/;
    }
    location /upstream_show {
        upstream_show;
    }
    # 查看后端服务器状态
    location /backend_status {
        check_status;
        # check_status json;
        access_log off;
        #allow 172.1.1.1;
        #deny all;
    }
}

3、初始后端服务器列表文件

cat /etc/nginx/conf.d/servers
> server 192.168.31.17:8081 weight=1 fail_timeout=10 max_fails=3;

4、启动 nginx systemctl start nginx

验证

1、etcd 方式

# 添加后端上游服务器, 对应配置项会立即写入文件 /etc/nginx/conf.d/servers
curl -X PUT -d value="{\"weight\":3, \"max_fails\":3, \"fail_timeout\":15}" http://192.168.31.69:2379/v2/keys/upstreams/test/192.168.31.17:8082
curl -X PUT -d value="{\"weight\":3, \"max_fails\":3, \"fail_timeout\":15}" http://192.168.31.69:2379/v2/keys/upstreams/test/192.168.31.17:8083

# 同步上游服务器配置到文件
cat /etc/nginx/conf.d/servers
server 192.168.31.17:8081 weight=1 fail_timeout=10 max_fails=3;
server 192.168.31.17:8082 weight=1 fail_timeout=10 max_fails=3;
server 192.168.31.17:8083 weight=1 fail_timeout=10 max_fails=3;

# 查看已添加的配置
export ETCDCTL_API=2
etcdctl --endpoints=//192.168.31.69:2379 -r ls /

2、Consul 方式

# 增加一个后端服务器
curl -X PUT http://192.168.31.69:8500/v1/kv/upstreams/test/192.168.31.17:8081
curl -X PUT http://192.168.31.69:8500/v1/kv/upstreams/test/192.168.31.17:8082
curl -X PUT http://192.168.31.69:8500/v1/kv/upstreams/test/192.168.31.17:8083

# 同步上游服务器配置到文件
cat /etc/nginx/conf.d/servers
server 192.168.31.17:8081 weight=1 fail_timeout=10 max_fails=3;
server 192.168.31.17:8082 weight=1 fail_timeout=10 max_fails=3;
server 192.168.31.17:8083 weight=1 fail_timeout=10 max_fails=3;

# 删除一个服务器
curl -X DELETE http://192.168.31.69:8500/v1/kv/upstreams/test/192.168.31.17:8081

3、业务访问测试

http://192.168.31.69/test/index.html

http://192.168.31.69/upstream_show
http://192.168.31.69/backend_status
http://192.168.31.69/backend_status?format=json