#!/bin/sh # vim: set noexpandtab tabstop=4 shiftwidth=4 softtabstop=4 : name=$0 basename="$(basename $0)" . /usr/lib/unbound/iptools.sh export OMR_TRACKER_STATUS export OMR_TRACKER_STATUS_MSG _log() { logger -p daemon.info -t "${basename}" "$@" } _post_tracking() { [ ! -d /usr/share/omr/server-post-tracking.d/ ] && return for tracker_bin in /usr/share/omr/server-post-tracking.d/*; do [ -x "$tracker_bin" ] && ( _log() { logger -t "post-tracking-${tracker_bin##*/}" "$*" } . "$tracker_bin" 2>&1 ) done } _ping_server() { local host=$1 ret=$(ping \ -w "$OMR_TRACKER_TIMEOUT" \ -c 1 \ -q \ "${host}" ) && echo "$ret" | grep -sq " 0% packet loss" && { server_ping=true } } _check_server() { local host=$1 local port=$2 local k=0 local valid_ip6=$(valid_subnet6 $host) while [ "$server_ping" = false ] && [ "$k" -le "$retry" ]; do if [ "$valid_ip6" != "ok" ]; then ret=$(curl \ --max-time "$OMR_TRACKER_TIMEOUT" \ -s \ -k \ "https://${host}:${port}/" ) else ret=$(curl \ --max-time "$OMR_TRACKER_TIMEOUT" \ -s \ -k \ "https://[${host}]:${port}/" ) fi [ -n "$ret" ] && server_ping=true k=$((k+1)) sleep "${intervaltries}" done } _check_server_intf() { local host=$1 local port=$2 local valid_ip6=$(valid_subnet6 $host) for intf in $(multipath 2>/dev/null | awk '/default/ {print $1}'); do local k=0 while [ "$server_ping" = false ] && [ "$k" -le "$retry" ]; do if [ "$valid_ip6" != "ok" ]; then ret=$(curl \ --max-time "$OMR_TRACKER_TIMEOUT" \ -s \ -k \ --interface $intf \ "https://${host}:${port}/" ) else ret=$(curl \ --max-time "$OMR_TRACKER_TIMEOUT" \ -s \ -k \ "https://[${host}%${intf}]:${port}/" ) fi [ -n "$ret" ] && { server_ping=true } k=$((k+1)) sleep "${intervaltries}" done done } _get_server_name() { local serverid=$1 local serverip=$2 config_get serveripc $serverid server if [ "$serveripc" = "$serverip" ]; then servername=$serverid fi } _disable_current() { local serv=$1 uci -q set openmptcprouter.${server}.current=0 } _disable_redir() { local redir=$1 local serverdisable=$2 local shadowsocks="$3" config_get serverss $redir server if [ "$serverss" = "$serverdisable" ]; then uci -q set ${shadowsocks}.${redir}.disabled=1 fi } _enable_redir() { local redir=$1 local shadowsocks="$2" config_get serverss $redir server if [ "$serverss" = "sss${count}" ]; then uci -q set ${shadowsocks}.${redir}.disabled=0 fi } _check_master() { local name=$1 local count=0 local countips=0 local changes="0" config_get master $1 master config_get ip $1 ip config_get port $1 port "65500" config_get disabled $1 disabled serverip="" [ "$master" = "1" ] && [ -n "$ip" ] && [ "$disabled" != "1" ] && { check_ip() { local ipd=$1 server_ping=false #[ -n "$serverip" ] && [ "$server_ping" = true ] && return ipresolve="$(resolveip -4 $ipd | head -n 1)" if [ -n "$ipresolve" ]; then serverip="$ipresolve" else ip6resolve="$(resolveip -6 $ipd | head -n 1)" [ -n "$ip6resolve" ] && serverip="$ip6resolve" fi [ -z "$serverip" ] && return #_ping_server $serverip #_check_server $serverip $port _check_server_intf $serverip $port [ "$server_ping" = true ] && { oneserverup="1" } countips=$((countips+1)) set_ip } set_ip() { ip="$serverip" if [ "$server_ping" = true ]; then if [ "$(uci -q get shadowsocks-libev.sss${count}.server | tr -d '\n')" != "$ip" ] && [ "$(uci -q get shadowsocks-rust.sss${count}.server | tr -d '\n')" != "$ip" ]; then logger -t "OMR-Tracker-Server" "Master server ${name} up ($ip), set it back" changes="1" #logger -t "OMR-Tracker-Server" "$(uci -q get shadowsocks-libev.sss${count}.server | tr -d '\n') - $ip" uci -q batch <<-EOF >/dev/null set shadowsocks-libev.sss${count}.server=$ip set shadowsocks-rust.sss${count}.server=$ip EOF if [ "$(uci -q get openmptcprouter.settings.proxy)" = "shadowsocks-rust" ]; then uci -q batch <<-EOF >/dev/null set shadowsocks-rust.sss${count}.disabled=0 EOF fi if [ "$(uci -q get openmptcprouter.settings.proxy)" = "shadowsocks" ]; then uci -q batch <<-EOF >/dev/null set shadowsocks-libev.sss${count}.disabled=0 EOF fi if [ -z "$(uci -q get openvpn.omr.remote | grep $ip)" ]; then uci -q batch <<-EOF >/dev/null add_list openvpn.omr.remote=$ip EOF fi if [ "$count" -eq "0" ]; then config_load openmptcprouter config_foreach _disable_current server uci -q batch <<-EOF >/dev/null set xray.omrout.s_vmess_address=$ip set xray.omrout.s_vless_address=$ip set xray.omrout.s_vless_reality_address=$ip set xray.omrout.s_trojan_address=$ip set xray.omrout.s_socks_address=$ip set xray.omrout.s_shadowsocks_address=$ip set v2ray.omrout.s_vmess_address=$ip set v2ray.omrout.s_vless_address=$ip set v2ray.omrout.s_trojan_address=$ip set v2ray.omrout.s_socks_address=$ip commit v2ray commit xray set glorytun.vpn.host=$ip commit glorytun glorytun-udp.vpn.host=$ip commit glorytun set dsvpn.vpn.host=$ip commit dsvpn set mlvpn.general.host=$ip commit mlvpn del openvpn.omr.remote add_list openvpn.omr.remote=$ip commit openvpn set openmptcprouter.${name}.current='1' commit openmptcprouter EOF /etc/init.d/openmptcprouter-vps get_openvpn_key $name >/dev/null 2>/dev/null #/etc/init.d/openmptcprouter-vps restart >/dev/null 2>/dev/null /etc/init.d/v2ray restart >/dev/null 2>/dev/null /etc/init.d/xray restart >/dev/null 2>/dev/null /etc/init.d/glorytun restart >/dev/null 2>/dev/null /etc/init.d/glorytun-udp restart >/dev/null 2>/dev/null /etc/init.d/mlvpn restart >/dev/null 2>/dev/null #/etc/init.d/openvpn restart omr >/dev/null 2>/dev/null /etc/init.d/dsvpn restart >/dev/null 2>/dev/null /etc/init.d/mptcpovervpn restart >/dev/null 2>/dev/null fi fi config_load shadowsocks-libev config_foreach _enable_redir ss_redir "shadowsocks-libev" config_load shadowsocks-rust config_foreach _enable_redir ss_redir "shadowsocks-rust" OMR_TRACKER_STATUS_MSG="Answer to ping and to API check" mail_alert="$(uci -q get omr-tracker.server.mail_alert)" #[ -z "$mail_alert" ] && mail_alert="$(uci -q get omr-tracker.defaults.mail_alert)" [ "$mail_alert" = "1" ] && [ -n "$(uci -q get mail.default.to)" ] && { OMR_SYSNAME="$(uci -q get system.@system[0].hostname)" if [ "$(uci -q get omr-tracker.defaults.mail_up_subject)" != "" ] && [ "$(uci -q get omr-tracker.defaults.mail_up_message)" != "" ]; then mail_subject="$(uci -q get omr-tracker.defaults.mail_up_subject)" mail_subject=`echo $mail_subject | sed -e "s/%SYSNAME%/$OMR_SYSNAME/g" -e "s/%INTERFACE%/Server ${name}/g" -e "s/%DEVICE%/${ip}/g" -e "s/%MESSAGE%/$OMR_TRACKER_STATUS_MSG/g"` mail_message="$(uci -q get omr-tracker.defaults.mail_up_message)" mail_message=`echo $mail_message | sed -e "s/%SYSNAME%/$OMR_SYSNAME/g" -e "s/%INTERFACE%/Server ${name}/g" -e "s/%DEVICE%/${ip}/g" -e "s/%MESSAGE%/$OMR_TRACKER_STATUS_MSG/g"` echo -e "Subject: ${mail_subject}\n\n${mail_message}" | sendmail $(uci -q get mail.default.to) else echo -e "Subject: $OMR_SYSNAME: Server ${name} (${ip}) is UP\n. The reason is \"$OMR_TRACKER_STATUS_MSG\"." | sendmail $(uci -q get mail.default.to) fi } OMR_TRACKER_STATUS="OK" script_alert_up="$(uci -q get omr-tracker.proxy.script_alert_up)" [ -n "$script_alert_up" ] && eval $script_alert_up count=$((count+1)) else [ -n "$ip" ] && [ -z "$(logread | tail -n 10 | grep "Master server ${name} down" | grep "$ip")" ] && logger -t "OMR-Tracker-Server" "Master server ${name} down ($ip)" OMR_TRACKER_STATUS_MSG="No answer to ping and API check" servername="" config_load shadowsocks-libev config_foreach _get_server_name server $ip [ -n "$servername" ] && config_foreach _disable_redir ss_redir $servername "shadowsocks-libev" servername="" config_load shadowsocks-rust config_foreach _get_server_name server $ip [ -n "$servername" ] && config_foreach _disable_redir ss_redir $servername "shadowsocks-rust" if [ -n "$(uci -q get openvpn.omr.remote | grep $ip)" ]; then uci -q batch <<-EOF >/dev/null del_list openvpn.omr.remote=$ip EOF fi OMR_TRACKER_STATUS_MSG="No answer to ping and to API check" mail_alert="$(uci -q get omr-tracker.server.mail_alert)" #[ -z "$mail_alert" ] && mail_alert="$(uci -q get omr-tracker.defaults.mail_alert)" [ "$mail_alert" = "1" ] && [ -n "$(uci -q get mail.default.to)" ] && { OMR_SYSNAME="$(uci -q get system.@system[0].hostname)" if [ "$(uci -q get omr-tracker.defaults.mail_down_subject)" != "" ] && [ "$(uci -q get omr-tracker.defaults.mail_down_message)" != "" ]; then mail_subject="$(uci -q get omr-tracker.defaults.mail_down_subject)" mail_subject=`echo $mail_subject | sed -e "s/%SYSNAME%/$OMR_SYSNAME/g" -e "s/%INTERFACE%/Server ${name}/g" -e "s/%DEVICE%/${ip}/g" -e "s/%MESSAGE%/$OMR_TRACKER_STATUS_MSG/g"` mail_message="$(uci -q get omr-tracker.defaults.mail_down_message)" mail_message=`echo $mail_message | sed -e "s/%SYSNAME%/$OMR_SYSNAME/g" -e "s/%INTERFACE%/Server ${name}/g" -e "s/%DEVICE%/${ip}/g" -e "s/%MESSAGE%/$OMR_TRACKER_STATUS_MSG/g"` echo -e "Subject: ${mail_subject}\n\n${mail_message}" | sendmail $(uci -q get mail.default.to) else echo -e "Subject: $OMR_SYSNAME: Server ${name} (${ip}) is down\n. The reason is \"$OMR_TRACKER_STATUS_MSG\"." | sendmail $(uci -q get mail.default.to) fi } OMR_TRACKER_STATUS="ERROR" script_alert_down="$(uci -q get omr-tracker.proxy.script_alert_down)" [ -n "$script_alert_down" ] && eval $script_alert_down fi } config_load openmptcprouter config_list_foreach $1 ip check_ip #set_ip #if [ "$server_ping" = true ] && [ "$countips" = "1" ]; then # uci -q batch <<-EOF >/dev/null # set shadowsocks-libev.sss${count}.server=$ip # EOF # config_foreach _enable_redir ss_redir #fi if [ "$oneserverup" = "0" ]; then uci -q batch <<-EOF >/dev/null set openmptcprouter.${name}.current='0' commit openmptcprouter EOF fi [ -n "$(uci changes shadowsocks-libev)" ] && changes="1" [ -n "$(uci changes shadowsocks-rust)" ] && changes="1" [ -n "$(uci changes openvpn)" ] && changes="1" uci -q commit shadowsocks-libev uci -q commit shadowsocks-rust uci -q commit openvpn [ "$changes" = "1" ] && { /etc/init.d/shadowsocks-libev restart >/dev/null 2>/dev/null /etc/init.d/shadowsocks-rust restart >/dev/null 2>/dev/null /etc/init.d/openvpn restart omr >/dev/null 2>/dev/null } break } } _check_backup() { local name=$1 local count=0 local countips=0 local changes="0" config_get backup $1 backup config_get ip $1 ip config_get port $1 port config_get disabled $1 disabled serverip="" [ "$backup" = "1" ] && [ -n "$ip" ] && [ "$disabled" != "1" ] && { check_ip() { local ipd=$1 server_ping=false #[ -n "$serverip" ] && [ "$server_ping" = true ] && return ipresolve="$(resolveip -4 $ipd | head -n 1)" if [ -n "$ipresolve" ]; then serverip="$ipresolve" else ip6resolve="$(resolveip -6 $ipd | head -n 1)" [ -n "$ip6resolve" ] && serverip="$ip6resolve" fi [ -z "$serverip" ] && return #_ping_server $serverip #_check_server $serverip $port _check_server_intf $serverip $port [ "$server_ping" = true ] && { oneserverup="1" } countips=$((countips+1)) } set_ip() { ip="$serverip" #[ "$server_ping" = true ] && [ "$(uci -q get shadowsocks-libev.sss${count}.server | tr -d '\n')" = "$ip" ] && break if [ "$server_ping" = true ]; then if [ "$(uci -q get shadowsocks-libev.sss${count}.server | tr -d '\n')" != "$ip" ] && [ "$(uci -q get shadowsocks-rust.sss${count}.server | tr -d '\n')" != "$ip" ]; then logger -t "OMR-Tracker-Server" "Use backup server $1 ($ip)" changes="1" uci -q batch <<-EOF >/dev/null set shadowsocks-libev.sss${count}.server=$ip set shadowsocks-rust.sss${count}.server=$ip EOF if [ "$(uci -q get openmptcprouter.settings.proxy)" = "shadowsocks-rust" ]; then uci -q batch <<-EOF >/dev/null set shadowsocks-rust.sss${count}.disabled=0 EOF fi if [ "$(uci -q get openmptcprouter.settings.proxy)" = "shadowsocks" ]; then uci -q batch <<-EOF >/dev/null set shadowsocks-libev.sss${count}.disabled=0 EOF fi if [ "$count" -eq "0" ]; then config_load openmptcprouter config_foreach _disable_current server uci -q batch <<-EOF >/dev/null set xray.omrout.s_vmess_address=$ip set xray.omrout.s_vless_address=$ip set xray.omrout.s_vless_reality_address=$ip set xray.omrout.s_trojan_address=$ip set xray.omrout.s_socks_address=$ip set xray.omrout.s_shadowsocks_address=$ip set v2ray.omrout.s_vmess_address=$ip set v2ray.omrout.s_vless_address=$ip set v2ray.omrout.s_trojan_address=$ip set v2ray.omrout.s_socks_address=$ip commit v2ray commit xray set glorytun.vpn.host=$ip commit glorytun glorytun-udp.vpn.host=$ip commit glorytun set dsvpn.vpn.host=$ip commit dsvpn set mlvpn.general.host=$ip commit mlvpn del openvpn.omr.remote add_list openvpn.omr.remote=$ip commit openvpn set openmptcprouter.${name}.current='1' commit openmptcprouter EOF /etc/init.d/openmptcprouter-vps get_openvpn_key $name >/dev/null 2>/dev/null #/etc/init.d/openmptcprouter-vps restart >/dev/null 2>/dev/null /etc/init.d/v2ray restart >/dev/null 2>/dev/null /etc/init.d/xray restart >/dev/null 2>/dev/null /etc/init.d/glorytun restart >/dev/null 2>/dev/null /etc/init.d/glorytun-udp restart >/dev/null 2>/dev/null /etc/init.d/mlvpn restart >/dev/null 2>/dev/null #/etc/init.d/openvpn restart omr >/dev/null 2>/dev/null /etc/init.d/dsvpn restart >/dev/null 2>/dev/null /etc/init.d/mptcpovervpn restart >/dev/null 2>/dev/null fi sleep $waittest fi config_load shadowsocks-libev config_foreach _enable_redir ss_redir "shadowsocks-libev" config_load shadowsocks-rust config_foreach _enable_redir ss_redir "shadowsocks-rust" OMR_TRACKER_STATUS_MSG="Answer to ping and to API check" mail_alert="$(uci -q get omr-tracker.server.mail_alert)" #[ -z "$mail_alert" ] && mail_alert="$(uci -q get omr-tracker.defaults.mail_alert)" [ "$mail_alert" = "1" ] && [ -n "$(uci -q get mail.default.to)" ] && { OMR_SYSNAME="$(uci -q get system.@system[0].hostname)" if [ "$(uci -q get omr-tracker.defaults.mail_up_subject)" != "" ] && [ "$(uci -q get omr-tracker.defaults.mail_up_message)" != "" ]; then mail_subject="$(uci -q get omr-tracker.defaults.mail_up_subject)" mail_subject=`echo $mail_subject | sed -e "s/%SYSNAME%/$OMR_SYSNAME/g" -e "s/%INTERFACE%/Server ${name}/g" -e "s/%DEVICE%/${ip}/g" -e "s/%MESSAGE%/$OMR_TRACKER_STATUS_MSG/g"` mail_message="$(uci -q get omr-tracker.defaults.mail_up_message)" mail_message=`echo $mail_message | sed -e "s/%SYSNAME%/$OMR_SYSNAME/g" -e "s/%INTERFACE%/Server ${name}/g" -e "s/%DEVICE%/${ip}/g" -e "s/%MESSAGE%/$OMR_TRACKER_STATUS_MSG/g"` echo -e "Subject: ${mail_subject}\n\n${mail_message}" | sendmail $(uci -q get mail.default.to) else echo -e "Subject: $OMR_SYSNAME: Server ${name} (${ip}) is UP\n. The reason is \"$OMR_TRACKER_STATUS_MSG\"." | sendmail $(uci -q get mail.default.to) fi } script_alert_up="$(uci -q get omr-tracker.proxy.script_alert_up)" [ -n "$script_alert_up" ] && eval $script_alert_up count=$((count+1)) else [ -n "$ip" ] && [ -z "$(logread | tail -n 10 | grep "Backup server ${name} down" | grep "$ip")" ] && logger -t "OMR-Tracker-Server" "Backup server ${name} down ($ip)" uci -q batch <<-EOF >/dev/null set openmptcprouter.${name}.current='0' commit openmptcprouter EOF config_load shadowsocks-libev config_foreach _get_server_name server $ip [ -n "$servername" ] && config_foreach _disable_redir ss_redir $servername "shadowsocks-libev" servername="" config_load shadowsocks-rust config_foreach _get_server_name server $ip [ -n "$servername" ] && config_foreach _disable_redir ss_redir $servername "shadowsocks-rust" OMR_TRACKER_STATUS_MSG="No answer to ping and to API check" mail_alert="$(uci -q get omr-tracker.server.mail_alert)" #[ -z "$mail_alert" ] && mail_alert="$(uci -q get omr-tracker.defaults.mail_alert)" [ "$mail_alert" = "1" ] && [ -n "$(uci -q get mail.default.to)" ] && { OMR_SYSNAME="$(uci -q get system.@system[0].hostname)" if [ "$(uci -q get omr-tracker.defaults.mail_down_subject)" != "" ] && [ "$(uci -q get omr-tracker.defaults.mail_down_message)" != "" ]; then mail_subject="$(uci -q get omr-tracker.defaults.mail_down_subject)" mail_subject=`echo $mail_subject | sed -e "s/%SYSNAME%/$OMR_SYSNAME/g" -e "s/%INTERFACE%/Server ${name}/g" -e "s/%DEVICE%/${ip}/g" -e "s/%MESSAGE%/$OMR_TRACKER_STATUS_MSG/g"` mail_message="$(uci -q get omr-tracker.defaults.mail_down_message)" mail_message=`echo $mail_message | sed -e "s/%SYSNAME%/$OMR_SYSNAME/g" -e "s/%INTERFACE%/Server ${name}/g" -e "s/%DEVICE%/${ip}/g" -e "s/%MESSAGE%/$OMR_TRACKER_STATUS_MSG/g"` echo -e "Subject: ${mail_subject}\n\n${mail_message}" | sendmail $(uci -q get mail.default.to) else echo -e "Subject: $OMR_SYSNAME: Server ${name} (${ip}) is down\n. The reason is \"$OMR_TRACKER_STATUS_MSG\"." | sendmail $(uci -q get mail.default.to) fi } script_alert_down="$(uci -q get omr-tracker.proxy.script_alert_down)" [ -n "$script_alert_down" ] && eval $script_alert_down fi countips=$((countips+1)) } config_load openmptcprouter config_list_foreach $1 ip check_ip set_ip #if [ "$server_ping" = true ] && [ "$countips" = "1" ]; then # uci -q batch <<-EOF >/dev/null # set shadowsocks-libev.sss${count}.server=$ip # EOF # config_foreach _enable_redir ss_redir #fi if [ "$oneserverup" = "0" ]; then uci -q batch <<-EOF >/dev/null set openmptcprouter.${name}.current='0' commit openmptcprouter EOF fi [ -n "$(uci changes shadowsocks-libev)" ] && changes="1" [ -n "$(uci changes shadowsocks-rust)" ] && changes="1" [ -n "$(uci changes openvpn)" ] && changes="1" uci -q commit shadowsocks-libev uci -q commit shadowsocks-rust uci -q commit openvpn [ "$changes" = "1" ] && { /etc/init.d/shadowsocks-libev restart >/dev/null 2>/dev/null /etc/init.d/shadowsocks-rust restart >/dev/null 2>/dev/null /etc/init.d/openvpn restart omr >/dev/null 2>/dev/null } [ "$server_ping" = true ] && break } } . /lib/functions.sh timeout=${OMR_TRACKER_TIMEOUT:-5} interval=${OMR_TRACKER_INTERVAL:-10} intervaltries=${OMR_TRACKER_INTERVAL_TRIES:-2} retry=${OMR_TRACKER_TRIES:-4} waittest=${OMR_TRACKER_WAIT_TEST:-0} while true; do server_ping=false oneserverup="0" config_load openmptcprouter config_foreach _check_master server [ "$oneserverup" = "0" ] && { config_load openmptcprouter config_foreach _check_backup server } sleep "${interval}" done