环境说明

测试环境，一共4个节点，两个用作gtm，另外两个用作gtm_proxy/coordinator和datanode。

具体安装不说，参考相关文档即可。

环境规划如下所示：

目录规划：

主机名	目录
pgxc-lvs	-
pgxc-gtm1	/data/gtm
pgxc-gtm2	/data/gtm
pgxc-n1	/data/coord, /data/data, /data/coord_s, /data/data_s, /data/gtm_proxy
pgxc-n2	/data/coord, /data/data, /data/coord_s, /data/data_s, /data/gtm_proxy

端口规划：

主机名/ROLE	端口
pgxc-lvs	5432(DB)
pgxc-gtm1	6666
pgxc-gtm2	6666
gtm_proxy1	6667
gtm_proxy2	6667
coord1/coord1_pooler Port	6611/6612
coord2/coord2_pooler Port	6611/6612
coord1_slave/coord1_slave_pooler Port	6621/6622
coord2_slave/coord2_slave_pooler Port	6621/6622
data1/data1_pooler Port	6631/6632
data2/data2_pooler Port	6631/6632
data1_slave/data1_slave_pooler Port	6641/6642
data1_slave/data1_slave_pooler Port	6641/6642

软件版本

我是用的是Postgres-XL提供的版本，所有节点都安装相同版本的软件包，如下所示为其中一个节点：

[pgxc@pgxc-gtm1 ~]$ rpm -qa |grep postgres
postgres-xl95-libs-9.5-1.6.el7.x86_64
postgres-xl95-gtm-9.5-1.6.el7.x86_64
postgres-xl95-contrib-9.5-1.6.el7.x86_64
postgres-xl95-test-9.5-1.6.el7.x86_64
postgres-xl95-9.5-1.6.el7.x86_64
postgres-xl95-server-9.5-1.6.el7.x86_64
postgres-xl95-devel-9.5-1.6.el7.x86_64

说明：官方手册是使用源码编译安装，懒得用源码编译，所以提前用源码编译了RPM包，一劳永逸。

环境变量

所有节点配置相同的环境变量设置，如下：

[pgxc@pgxc-gtm1 ~]$ tail -n4 ~/.bashrc
export PGHOME=/usr/postgres-xl-9.5
export export PGUSER=pgxc
export LD_LIBRARY_PATH=$PGHOME/lib:$LD_LIBRARY_PATH
export PATH=$PGHOME/bin:$PATH

操作过程

生成模版配置文件

选取任一节点，如pgxc-gtm1节点上操作：

$ ssh root@10.128.0.31
$ pgxc_ctl

$ pgxc_ctl
/usr/bin/bash
Installing pgxc_ctl_bash script as /home/pgxc/pgxc_ctl/pgxc_ctl_bash.
ERROR: File "/home/pgxc/pgxc_ctl/pgxc_ctl.conf" not found or not a regular file. No such file or directory
Installing pgxc_ctl_bash script as /home/pgxc/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/pgxc/pgxc_ctl/pgxc_ctl_bash --home /home/pgxc/pgxc_ctl --configuration /home/pgxc/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/pgxc/pgxc_ctl
PGXC prepare config empty   # 生成空的配置文件模板
PGXC exit                   # 退出，也可以执行`quit`。

往集群中添加节点

配置gtm节点（master/standby）

pgxc_ctl
add gtm master gtm1 pgxc-gtm1 6666 /data/gtm
add gtm slave  gtm2 pgxc-gtm2 6666 /data/gtm
monitor all
quit

设置附加配置文件

生成附加配置文件，这两个文件的内容分别需要添加到各节点实例的postgresql.conf和pg_hba.conf文件中：

cat > ~/pgxc_ctl/coordExtraConfig.conf <<EOF
#================================================
# Added to all the coordinator postgresql.conf
# Original: \$coordExtraConfig
log_destination = 'stderr'
logging_collector = on
log_directory = 'pg_log'
listen_addresses = '*'
max_connections = 256
EOF

cat > ~/pgxc_ctl/coordExtraPgHba.conf <<EOF
#================================================
# Added to all the coordinator master's pg_hba.conf
# Original: \$coordExtraPgHba
host    all    all    10.128.0.30/32  trust
host    all    all    10.128.0.31/32  trust
host    all    all    10.128.0.32/32  trust
host    all    all    10.128.0.33/32  trust
host    all    all    10.128.0.34/32  trust
host    all    all    10.128.0.35/32  trust
host    all    all    10.128.0.36/32  trust
host    all    all    0.0.0.0/0       md5
EOF

添加协调节点（Master/Slave）

pgxc_ctl
add coordinator master coord1 pgxc-n1 6611 6612 /data/coord coordExtraConfig.conf coordExtraPgHba.conf
add coordinator master coord2 pgxc-n2 6611 6612 /data/coord coordExtraConfig.conf coordExtraPgHba.conf
add coordinator slave coord1 pgxc-n2 6621 6622 /data/coord_s /data/coord_s_arc
add coordinator slave coord2 pgxc-n1 6621 6622 /data/coord_s /data/coord_s_arc
monitor coordinator all
exit

Check backgroud coordinator processes

for i in `seq 2`; do echo "===> pgxc-n$i: " && ssh pgxc@pgxc-n$i "ls -ld /data/coord*" && echo; done
for i in `seq 2`; do echo "===> pgxc-n$i: " && ssh pgxc@pgxc-n$i ps -ef |grep -v grep |grep coordinator && echo; done
for i in `seq 2`; do echo "===> pgxc-n$i: " && ssh pgxc@pgxc-n$i ps -ef |grep -v grep |grep -E '(coordinator|sender|receiver)' && echo; done

添加数据节点（Master/Slave）

pgxc_ctl
add datanode master data1 pgxc-n1 6631 6632 /data/data none coordExtraConfig.conf coordExtraPgHba.conf
add datanode master data2 pgxc-n2 6631 6632 /data/data none coordExtraConfig.conf coordExtraPgHba.conf
add datanode slave data1 pgxc-n2 6641 6642 /data/data_s none /data/data_s_arc
add datanode slave data2 pgxc-n1 6641 6642 /data/data_s none /data/data_s_arc
monitor datanode all
exit

Check backgroud datanode processes

for i in `seq 2`; do echo "===> pgxc-n$i: " && ssh pgxc@pgxc-n$i "ls -ld /data/data*" && echo; done
for i in `seq 2`; do echo "===> pgxc-n$i: " && ssh pgxc@pgxc-n$i ps -ef |grep -v grep |grep datanode && echo; done
for i in `seq 2`; do echo "===> pgxc-n$i: " && ssh pgxc@pgxc-n$i ps -ef |grep -v grep |grep -E '(datanode|sender|receiver)' && echo; done

添加GTM proxy节点

首先将GTM proxy节点添加到集群中：

pgxc_ctl
add gtm_proxy gtm_proxy1 pgxc-n1 6667 /data/gtm_proxy
add gtm_proxy gtm_proxy2 pgxc-n2 6667 /data/gtm_proxy
quit

默认添加后可能起不来，但集群会注册这两个节点信息。

我们需要分别连接到pgxc-n1和pgxc-n2节点上修改配置文件，然后启动GTM proxy。

首先，配置pgxc-n1上的GTM proxy，如下所示：

ssh pgxc-n1
vi /data/gtm_proxy/gtm_proxy.conf      # 根据需要修改参数配置。

参数配置如下所示：

nodename = 'gtm_proxy1'
listen_addresses = '*'
port = 6667
worker_threads = 2
gtm_host = 'pgxc-gtm1'
gtm_port = 6667
gtm_connect_retry_interval = 1
log_file = 'gtm_proxy.log'
log_min_messages = INFO

启动GTM proxy：

gtm_proxy -h pgxc-n1 -p 6667 -s pgxc-gtm1 -t 6666 -n 2 -D /data/gtm_proxy/ -l /data/gtm_proxy/gtm_proxy.log &
exit

然后，对pgxc-n2节点执行类似步骤来启动该节点上的GTM proxy：

ssh pgxc-n2 
vi /data/gtm_proxy/gtm_proxy.conf      # 根据需要修改参数配置。
gtm_proxy -h pgxc-n2 -p 6667 -s pgxc-gtm1 -t 6666 -n 2 -D /data/gtm_proxy/ -l /data/gtm_proxy/gtm_proxy.log &
exit

然后，回到pgxc_ctl命令行中，观察所有节点状态：

[pgxc@pgxc-gtm1 ~]$ pgxc_ctl
/bin/bash
Installing pgxc_ctl_bash script as /home/pgxc/pgxc_ctl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /home/pgxc/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/pgxc/pgxc_ctl/pgxc_ctl_bash --home /home/pgxc/pgxc_ctl --configuration /home/pgxc/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
******** PGXC_CTL START ***************

Current directory: /home/pgxc/pgxc_ctl
PGXC monitor all
Running: gtm master
Running: gtm slave
Running: gtm proxy gtm_proxy1
Running: gtm proxy gtm_proxy2
Running: coordinator master coord1
Running: coordinator slave coord1
Running: coordinator master coord2
Running: coordinator slave coord2
Running: datanode master data1
Running: datanode slave data1
Running: datanode master data2
Running: datanode slave data2
PGXC

pgxc_ctl使用帮助

pgxc_ctl工具也可以执行节点删除等工作，使用help COMMAND查看帮助信息，如下所示：

PGXC help
You are using pgxc_ctl, the configuration utility for PGXL
Type:
    help <command>
    where <command> is either add, Createdb, Createuser, clean,
        configure, deploy, failover, init, kill, log, monitor,
        prepare, q, reconnect, remove, set, show, start,
        stop or unregister

PGXC help remove

remove gtm slave
remove gtm_proxy nodename [ clean ]
remove coordinator [ master| slave ] nodename [ clean ]
remove datanode [ master| slave ] nodename [ clean ]

Removes the specified node from the cluster
For more details, please see the pgxc_ctl documentation

PGXC