From 11aa6bfe53bc88fa0899c834cafb400dcb610ed0 Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 18:39:57 -0800 Subject: [PATCH 01/21] Add rcon --- README.md | 34 ++++++++++------ rcon.sh | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+), 11 deletions(-) create mode 100755 rcon.sh diff --git a/README.md b/README.md index e3cf5d3..73a7208 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # Minecraft Backup -Backup script for Linux servers running a Minecraft server in a GNU Screen or tmux +![GitHub Workflow Status](https://img.shields.io/github/workflow/status/nicolaschan/minecraft-backup/CI) +[![codecov](https://codecov.io/gh/nicolaschan/minecraft-backup/branch/master/graph/badge.svg?token=LCbVC4TbYJ)](https://codecov.io/gh/nicolaschan/minecraft-backup) + +Backup script for Linux servers running a Minecraft server. +Supports servers running in [GNU screen](https://en.wikipedia.org/wiki/GNU_Screen), [tmux](https://en.wikipedia.org/wiki/Tmux), or with [rcon](https://wiki.vg/RCON) enabled. ### Disclaimer Backups are essential to the integrity of your Minecraft world. You should automate regular backups and **check that your backups work**. While this script has been used in production for several years, it is up to you to make sure that your backups work and that you have a reliable backup policy. @@ -12,17 +16,26 @@ Please refer to the LICENSE (MIT License) for the full legal disclaimer. - "thin" - keep last 24 hourly, last 30 daily, and use remaining space for monthly backups - "sequential" - delete oldest backup - Choose your own compression algorithm (tested with: `gzip`, `xz`, `zstd`) +- Works on vanilla -- no plugins required - Able to print backup status and info to the Minecraft chat +## Why not just use `tar` directly? +If the Minecraft server is currently running, you need to disable world autosaving, or you will likely get an error like this: +``` +tar: /some/path/here/world/region/r.1.11.mca: file changed as we read it +``` +This script will take care of disabling and then re-enabling autosaving for you, and also alert players in the chat of successful backups or errors. This script also manages deleting old backups. + ## Requirements - Linux computer (tested on Ubuntu) - GNU Screen (running your Minecraft server) -- Minecraft server (tested with Vanilla 1.10.2 only) +- Minecraft server -## Installation -1. Download the script: `$ wget https://raw.githubusercontent.com/nicolaschan/minecraft-backup/master/backup.sh` -2. Mark as executable: `$ chmod +x backup.sh` -3. Use the command line options or configure default values at the top of `backup.sh`: +## Quick Start +```bash +wget https://raw.githubusercontent.com/nicolaschan/minecraft-backup/master/backup.sh` +chmod +x backup.sh +``` Command line options: ```text @@ -47,11 +60,11 @@ Example usage of command line options: ```bash ./backup.sh -c -i /home/server/minecraft-server/world -o /mnt/external-storage/minecraft-backups -s minecraft ``` -This will use show chat messages (`-c`) in the screen called "minecraft" and save a backup of `/home/server/minecraft-server/world` into `/mnt/external-storage/minecraft-backups` using the default thinning delete policy for old backups. +This will show chat messages (`-c`) in the screen called "minecraft" and save a backup of `/home/server/minecraft-server/world` into `/mnt/external-storage/minecraft-backups` using the default thinning delete policy for old backups. -4. Create a cron job to automatically backup: - - Edit the crontab: `$ crontab -e` - - Example for hourly backups: `00 * * * * /path/to/backup.sh` +### Create a cron job to automatically backup: +- Edit the crontab with `crontab -e` +- Example for hourly backups: `00 * * * * /path/to/backup.sh ...` ## Retrieving Backups Always test your backups! Backups are in the `tar` format and compressed depending on the option you choose. To restore, first decompress if necessary and then extract using tar. You may be able to do this in one command if `tar` supports your compression option, as is the case with `gzip`: @@ -70,6 +83,5 @@ Then you can move your restored world (`restored-world` in this case) to your Mi - Make sure your compression algorithm is in the crontab's PATH - Make sure cron has permissions for all the files involved and access to the Minecraft server's GNU Screen - It's surprising how much space backups can take--make sure you have enough empty space -- `SERVER_DIRECTORY` should be the server directory, not the `world` directory - Do not put trailing `/` in the `SERVER_DIRECTORY` or `BACKUP_DIRECTORY` - If "thin" delete method is behaving weirdly, try emptying your backup directory or switch to "sequential" diff --git a/rcon.sh b/rcon.sh new file mode 100755 index 0000000..bf5e42e --- /dev/null +++ b/rcon.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash + +reverse-hex-endian () { + # Given a 4-byte hex integer, reverse endianness + while read -r -d '' -N 8 INTEGER; do + echo "$INTEGER" | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/' + done +} + +decode-hex-int () { + # decode little-endian hex integer + while read -r -d '' -N 8 INTEGER; do + BIG_ENDIAN_HEX=$(echo "$INTEGER" | reverse-hex-endian) + echo "$((16#$BIG_ENDIAN_HEX))" + done +} + +encode-int () { + # Encode an integer as 4 bytes in little endian and return as hex + INT="$1" + # Source: https://stackoverflow.com/a/9955198 + printf "%08x" "$INT" | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/' +} + +encode () { + # Encode a packet type and payload for the rcon protocol + TYPE="$1" + PAYLOAD="$2" + REQUEST_ID="$3" + PAYLOAD_LENGTH="${#PAYLOAD}" + TOTAL_LENGTH="$((4 + 4 + PAYLOAD_LENGTH + 1 + 1))" + + OUTPUT="" + OUTPUT+=$(encode-int "$TOTAL_LENGTH") + OUTPUT+=$(encode-int "$REQUEST_ID") + OUTPUT+=$(encode-int "$TYPE") + OUTPUT+=$(echo -n "$PAYLOAD" | xxd -ps) + OUTPUT+="0000" + + echo -n "$OUTPUT" | xxd -ps -r +} + +read-response () { + # read next response packet and return the payload text + IN_PIPE="$1" + # HEX_LENGTH=$(head -c4 "$IN_PIPE" | xxd -ps | reverse-hex-endian) + HEX_LENGTH=$(head -c4 <&3 | xxd -ps | reverse-hex-endian) + LENGTH=$((16#$HEX_LENGTH)) + + RESPONSE_PAYLOAD=$(head -c $LENGTH <&3 | xxd -ps) + echo -n "$RESPONSE_PAYLOAD" +} + +response-request-id () { + echo -n "${1:0:8}" | decode-hex-int +} + +response-type () { + echo -n "${1:8:8}" | decode-hex-int +} + +response-payload () { + echo -n "${1:16:-4}" | xxd -r -ps +} + +login () { + PASSWORD="$1" + encode 3 "$PASSWORD" 12 >&3 + + RESPONSE=$(read-response "$IN_PIPE") + + RESPONSE_REQUEST_ID=$(response-request-id "$RESPONSE") + if [ "$RESPONSE_REQUEST_ID" -eq -1 ] || [ "$RESPONSE_REQUEST_ID" -eq 4294967295 ]; then + echo "Authentication failed: Wrong RCON password" 1>&2 + return 1 + fi +} + +run-command () { + COMMAND="$1" + + # encode 2 "$COMMAND" 13 >> "$OUT_PIPE" + encode 2 "$COMMAND" 13 >&3 + + RESPONSE=$(read-response "$IN_PIPE") + response-payload "$RESPONSE" +} + +rcon-command () { + HOST="$1" + PORT="$2" + PASSWORD="$3" + COMMAND="$4" + + # Open a TCP socket + # Source: https://www.xmodulo.com/tcp-udp-socket-bash-shell.html + exec 3<>/dev/tcp/"$HOST"/"$PORT" + + login "$PASSWORD" || return 1 + run-command "$COMMAND" + + # Close the socket + exec 3<&- + exec 3>&- +} + + +HOST="$1" +PORT="$2" +PASSWORD="$3" +COMMAND="$4" + +rcon-command "$HOST" "$PORT" "$PASSWORD" "$COMMAND" From 1c4162455f7a1729e78912974c275295bab95dcc Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 18:44:50 -0800 Subject: [PATCH 02/21] Touch up typos --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 73a7208..290faed 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ This script will take care of disabling and then re-enabling autosaving for you, ## Quick Start ```bash -wget https://raw.githubusercontent.com/nicolaschan/minecraft-backup/master/backup.sh` +wget https://raw.githubusercontent.com/nicolaschan/minecraft-backup/master/backup.sh chmod +x backup.sh ``` @@ -58,9 +58,9 @@ Command line options: Example usage of command line options: ```bash -./backup.sh -c -i /home/server/minecraft-server/world -o /mnt/external-storage/minecraft-backups -s minecraft +./backup.sh -c -i /home/nicolas/mcserver/world -o /mnt/storage/backups -s minecraft ``` -This will show chat messages (`-c`) in the screen called "minecraft" and save a backup of `/home/server/minecraft-server/world` into `/mnt/external-storage/minecraft-backups` using the default thinning delete policy for old backups. +This will show chat messages (`-c`) in the screen called "minecraft" and save a backup of `/home/nicolas/mcserver/world` into `/mnt/storage/backups` using the default thinning deletion policy for old backups. ### Create a cron job to automatically backup: - Edit the crontab with `crontab -e` From b8094d758e5f9fb0e2ed53183e53974e020e2587 Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 19:50:47 -0800 Subject: [PATCH 03/21] Reorganize readme --- README.md | 62 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 33 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 290faed..1846d38 100644 --- a/README.md +++ b/README.md @@ -2,13 +2,8 @@ ![GitHub Workflow Status](https://img.shields.io/github/workflow/status/nicolaschan/minecraft-backup/CI) [![codecov](https://codecov.io/gh/nicolaschan/minecraft-backup/branch/master/graph/badge.svg?token=LCbVC4TbYJ)](https://codecov.io/gh/nicolaschan/minecraft-backup) -Backup script for Linux servers running a Minecraft server. -Supports servers running in [GNU screen](https://en.wikipedia.org/wiki/GNU_Screen), [tmux](https://en.wikipedia.org/wiki/Tmux), or with [rcon](https://wiki.vg/RCON) enabled. - -### Disclaimer -Backups are essential to the integrity of your Minecraft world. You should automate regular backups and **check that your backups work**. While this script has been used in production for several years, it is up to you to make sure that your backups work and that you have a reliable backup policy. - -Please refer to the LICENSE (MIT License) for the full legal disclaimer. +Backup script for Minecraft servers on Linux. +Supports servers running in [screen](https://en.wikipedia.org/wiki/GNU_Screen), [tmux](https://en.wikipedia.org/wiki/Tmux), or with [rcon](https://wiki.vg/RCON) enabled. ## Features - Create backups of your world folder @@ -19,24 +14,19 @@ Please refer to the LICENSE (MIT License) for the full legal disclaimer. - Works on vanilla -- no plugins required - Able to print backup status and info to the Minecraft chat -## Why not just use `tar` directly? -If the Minecraft server is currently running, you need to disable world autosaving, or you will likely get an error like this: -``` -tar: /some/path/here/world/region/r.1.11.mca: file changed as we read it -``` -This script will take care of disabling and then re-enabling autosaving for you, and also alert players in the chat of successful backups or errors. This script also manages deleting old backups. - -## Requirements -- Linux computer (tested on Ubuntu) -- GNU Screen (running your Minecraft server) -- Minecraft server - -## Quick Start +## Install ```bash wget https://raw.githubusercontent.com/nicolaschan/minecraft-backup/master/backup.sh chmod +x backup.sh ``` +## Usage +```bash +./backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s minecraft +``` + +This will show chat messages (`-c`) in the screen called "minecraft" and save a backup of `/home/user/mcserver/world` into `/mnt/storage/backups` using the default thinning deletion policy for old backups. + Command line options: ```text -a Compression algorithm (default: gzip) @@ -56,16 +46,13 @@ Command line options: -w Window manager: screen (default) or tmux ``` -Example usage of command line options: -```bash -./backup.sh -c -i /home/nicolas/mcserver/world -o /mnt/storage/backups -s minecraft -``` -This will show chat messages (`-c`) in the screen called "minecraft" and save a backup of `/home/nicolas/mcserver/world` into `/mnt/storage/backups` using the default thinning deletion policy for old backups. - -### Create a cron job to automatically backup: +### Automate backups with cron - Edit the crontab with `crontab -e` -- Example for hourly backups: `00 * * * * /path/to/backup.sh ...` - +- Example for hourly backups: +``` +00 * * * * /path/to/backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s minecraft +``` + ## Retrieving Backups Always test your backups! Backups are in the `tar` format and compressed depending on the option you choose. To restore, first decompress if necessary and then extract using tar. You may be able to do this in one command if `tar` supports your compression option, as is the case with `gzip`: @@ -78,6 +65,13 @@ tar -xzvf /path/to/backups/2019-04-09_02-15-01.tar.gz Then you can move your restored world (`restored-world` in this case) to your Minecraft server folder and rename it (usually called `world`) so the Minecraft server uses it. +## Why not use `tar` directly? +If the Minecraft server is currently running, you need to disable world autosaving, or you will likely get an error like this: +``` +tar: /some/path/here/world/region/r.1.11.mca: file changed as we read it +``` +This script will take care of disabling and then re-enabling autosaving for you, and also alert players in the chat of successful backups or errors. This script also manages deleting old backups. + ## Help - Make sure the compression algorithm you specify is installed on your system. (zstd is not installed by default) - Make sure your compression algorithm is in the crontab's PATH @@ -85,3 +79,13 @@ Then you can move your restored world (`restored-world` in this case) to your Mi - It's surprising how much space backups can take--make sure you have enough empty space - Do not put trailing `/` in the `SERVER_DIRECTORY` or `BACKUP_DIRECTORY` - If "thin" delete method is behaving weirdly, try emptying your backup directory or switch to "sequential" + +## Disclaimer +Backups are essential to the integrity of your Minecraft world. You should automate regular backups and **check that your backups work**. It is up to you to make sure that your backups work and that you have a reliable backup policy. + +Some backup tips: +- Drives get corrupted or fail! Backup to a _different_ drive than the one your server is running on, so if your main drive fails then you have backups. +- _Automate_ backups so you never lose too much progress. +- Check that your backups work from time to time. + +Please refer to the LICENSE (MIT License) for the full legal disclaimer. From a943f093a6ede632ba9478f0c683e82aa233723f Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 19:52:10 -0800 Subject: [PATCH 04/21] Update README --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 1846d38..1be40a6 100644 --- a/README.md +++ b/README.md @@ -8,11 +8,11 @@ Supports servers running in [screen](https://en.wikipedia.org/wiki/GNU_Screen), ## Features - Create backups of your world folder - Manage deletion of old backups - - "thin" - keep last 24 hourly, last 30 daily, and use remaining space for monthly backups + - "thin" - keep last 24 hourly, last 30 daily, and use remaining space for weekly backups - "sequential" - delete oldest backup - Choose your own compression algorithm (tested with: `gzip`, `xz`, `zstd`) -- Works on vanilla -- no plugins required -- Able to print backup status and info to the Minecraft chat +- Works on vanilla (no plugins required) +- Print backup status to the Minecraft chat ## Install ```bash From 22a579a19c1f4db634c9bf516936b93b925df5de Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 22:12:28 -0800 Subject: [PATCH 05/21] Add rcon to backup script --- README.md | 5 +- backup.sh | 125 ++++++++++++++++++++++++++++-- rcon.sh | 188 ++++++++++++++++++++++++---------------------- test/mock_rcon.py | 31 ++++++++ test/test.sh | 24 +++++- 5 files changed, 272 insertions(+), 101 deletions(-) create mode 100644 test/mock_rcon.py diff --git a/README.md b/README.md index 1be40a6..b6bda51 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,6 @@ Supports servers running in [screen](https://en.wikipedia.org/wiki/GNU_Screen), - Manage deletion of old backups - "thin" - keep last 24 hourly, last 30 daily, and use remaining space for weekly backups - "sequential" - delete oldest backup -- Choose your own compression algorithm (tested with: `gzip`, `xz`, `zstd`) - Works on vanilla (no plugins required) - Print backup status to the Minecraft chat @@ -66,11 +65,11 @@ tar -xzvf /path/to/backups/2019-04-09_02-15-01.tar.gz Then you can move your restored world (`restored-world` in this case) to your Minecraft server folder and rename it (usually called `world`) so the Minecraft server uses it. ## Why not use `tar` directly? -If the Minecraft server is currently running, you need to disable world autosaving, or you will likely get an error like this: +If you use `tar` while the server is running, you will likely get an error like this because Minecraft autosaves the world periodically: ``` tar: /some/path/here/world/region/r.1.11.mca: file changed as we read it ``` -This script will take care of disabling and then re-enabling autosaving for you, and also alert players in the chat of successful backups or errors. This script also manages deleting old backups. +To fix this problem, the backup script disables autosaving with the `save-off` Minecraft command before running `tar` and then re-enables autosaving after `tar` is done. ## Help - Make sure the compression algorithm you specify is installed on your system. (zstd is not installed by default) diff --git a/backup.sh b/backup.sh index 9f2ad43..1b0a487 100755 --- a/backup.sh +++ b/backup.sh @@ -47,9 +47,9 @@ while getopts 'a:cd:e:f:hi:l:m:o:p:qs:vw:' FLAG; do echo "-o Output directory" echo "-p Prefix that shows in Minecraft chat (default: Backup)" echo "-q Suppress warnings" - echo "-s Minecraft server screen name" + echo "-s Screen name, tmux session name, or hostname:port:password for rcon" echo "-v Verbose mode" - echo "-w Window manager: screen (default) or tmux" + echo "-w Window manager: screen (default), tmux, rcon" exit 0 ;; i) SERVER_WORLD=$OPTARG ;; @@ -71,6 +71,116 @@ log-warning () { echo -e "\033[0;33mWARNING:\033[0m $*" } +rcon-command () { + HOST="$(echo $1 | cut -d: -f1)" + PORT="$(echo $1 | cut -d: -f2)" + PASSWORD="$(echo $1 | cut -d: -f3-)" + COMMAND="$2" + + reverse-hex-endian () { + # Given a 4-byte hex integer, reverse endianness + while read -r -d '' -N 8 INTEGER; do + echo "$INTEGER" | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/' + done + } + + decode-hex-int () { + # decode little-endian hex integer + while read -r -d '' -N 8 INTEGER; do + BIG_ENDIAN_HEX=$(echo "$INTEGER" | reverse-hex-endian) + echo "$((16#$BIG_ENDIAN_HEX))" + done + } + + stream-to-hex () { + xxd -ps + } + + hex-to-stream () { + xxd -ps -r + } + + encode-int () { + # Encode an integer as 4 bytes in little endian and return as hex + INT="$1" + # Source: https://stackoverflow.com/a/9955198 + printf "%08x" "$INT" | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/' + } + + encode () { + # Encode a packet type and payload for the rcon protocol + TYPE="$1" + PAYLOAD="$2" + REQUEST_ID="$3" + PAYLOAD_LENGTH="${#PAYLOAD}" + TOTAL_LENGTH="$((4 + 4 + PAYLOAD_LENGTH + 1 + 1))" + + OUTPUT="" + OUTPUT+=$(encode-int "$TOTAL_LENGTH") + OUTPUT+=$(encode-int "$REQUEST_ID") + OUTPUT+=$(encode-int "$TYPE") + OUTPUT+=$(echo -n "$PAYLOAD" | stream-to-hex) + OUTPUT+="0000" + + echo -n "$OUTPUT" | hex-to-stream + } + + read-response () { + # read next response packet and return the payload text + HEX_LENGTH=$(head -c4 <&3 | stream-to-hex | reverse-hex-endian) + LENGTH=$((16#$HEX_LENGTH)) + + RESPONSE_PAYLOAD=$(head -c $LENGTH <&3 | stream-to-hex) + echo -n "$RESPONSE_PAYLOAD" + } + + response-request-id () { + echo -n "${1:0:8}" | decode-hex-int + } + + response-type () { + echo -n "${1:8:8}" | decode-hex-int + } + + response-payload () { + echo -n "${1:16:-4}" | hex-to-stream + } + + login () { + PASSWORD="$1" + encode 3 "$PASSWORD" 12 >&3 + + RESPONSE=$(read-response "$IN_PIPE") + + RESPONSE_REQUEST_ID=$(response-request-id "$RESPONSE") + if [ "$RESPONSE_REQUEST_ID" -eq -1 ] || [ "$RESPONSE_REQUEST_ID" -eq 4294967295 ]; then + log-warning "RCON connection failed: Wrong RCON password" 1>&2 + return 1 + fi + } + + run-command () { + COMMAND="$1" + + # encode 2 "$COMMAND" 13 >> "$OUT_PIPE" + encode 2 "$COMMAND" 13 >&3 + + RESPONSE=$(read-response "$IN_PIPE") + response-payload "$RESPONSE" + } + + # Open a TCP socket + # Source: https://www.xmodulo.com/tcp-udp-socket-bash-shell.html + exec 3<>/dev/tcp/"$HOST"/"$PORT" + + login "$PASSWORD" || return 1 + run-command "$COMMAND" + + # Close the socket + exec 3<&- + exec 3>&- +} + if [[ $COMPRESSION_FILE_EXTENSION == "." ]]; then COMPRESSION_FILE_EXTENSION="" fi @@ -78,7 +188,7 @@ fi # Check for missing encouraged arguments if ! $SUPPRESS_WARNINGS; then if [[ $SCREEN_NAME == "" ]]; then - log-warning "Minecraft screen name not specified (use -s)" + log-warning "Minecraft screen/tmux/rcon location not specified (use -s)" fi fi # Check for required arguments @@ -109,9 +219,11 @@ execute-command () { local COMMAND=$1 if [[ $SCREEN_NAME != "" ]]; then case $WINDOW_MANAGER in - "screen") screen -S $SCREEN_NAME -p 0 -X stuff "$COMMAND$(printf \\r)" + "screen") screen -S "$SCREEN_NAME" -p 0 -X stuff "$COMMAND$(printf \\r)" ;; - "tmux") tmux send-keys -t $SCREEN_NAME "$COMMAND" ENTER + "tmux") tmux send-keys -t "$SCREEN_NAME" "$COMMAND" ENTER + ;; + "rcon") rcon-command "$SCREEN_NAME" "$COMMAND" ;; esac fi @@ -238,6 +350,9 @@ delete-thinning () { delete-sequentially } +# Ensure directory exists +mkdir -p "$(dirname $ARCHIVE_PATH)" + # Disable world autosaving execute-command "save-off" diff --git a/rcon.sh b/rcon.sh index bf5e42e..cc4f679 100755 --- a/rcon.sh +++ b/rcon.sh @@ -1,96 +1,102 @@ #!/usr/bin/env bash -reverse-hex-endian () { - # Given a 4-byte hex integer, reverse endianness - while read -r -d '' -N 8 INTEGER; do - echo "$INTEGER" | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/' - done -} - -decode-hex-int () { - # decode little-endian hex integer - while read -r -d '' -N 8 INTEGER; do - BIG_ENDIAN_HEX=$(echo "$INTEGER" | reverse-hex-endian) - echo "$((16#$BIG_ENDIAN_HEX))" - done -} - -encode-int () { - # Encode an integer as 4 bytes in little endian and return as hex - INT="$1" - # Source: https://stackoverflow.com/a/9955198 - printf "%08x" "$INT" | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/' -} - -encode () { - # Encode a packet type and payload for the rcon protocol - TYPE="$1" - PAYLOAD="$2" - REQUEST_ID="$3" - PAYLOAD_LENGTH="${#PAYLOAD}" - TOTAL_LENGTH="$((4 + 4 + PAYLOAD_LENGTH + 1 + 1))" - - OUTPUT="" - OUTPUT+=$(encode-int "$TOTAL_LENGTH") - OUTPUT+=$(encode-int "$REQUEST_ID") - OUTPUT+=$(encode-int "$TYPE") - OUTPUT+=$(echo -n "$PAYLOAD" | xxd -ps) - OUTPUT+="0000" - - echo -n "$OUTPUT" | xxd -ps -r -} - -read-response () { - # read next response packet and return the payload text - IN_PIPE="$1" - # HEX_LENGTH=$(head -c4 "$IN_PIPE" | xxd -ps | reverse-hex-endian) - HEX_LENGTH=$(head -c4 <&3 | xxd -ps | reverse-hex-endian) - LENGTH=$((16#$HEX_LENGTH)) - - RESPONSE_PAYLOAD=$(head -c $LENGTH <&3 | xxd -ps) - echo -n "$RESPONSE_PAYLOAD" -} - -response-request-id () { - echo -n "${1:0:8}" | decode-hex-int -} - -response-type () { - echo -n "${1:8:8}" | decode-hex-int -} - -response-payload () { - echo -n "${1:16:-4}" | xxd -r -ps -} - -login () { - PASSWORD="$1" - encode 3 "$PASSWORD" 12 >&3 - - RESPONSE=$(read-response "$IN_PIPE") - - RESPONSE_REQUEST_ID=$(response-request-id "$RESPONSE") - if [ "$RESPONSE_REQUEST_ID" -eq -1 ] || [ "$RESPONSE_REQUEST_ID" -eq 4294967295 ]; then - echo "Authentication failed: Wrong RCON password" 1>&2 - return 1 - fi -} - -run-command () { - COMMAND="$1" - - # encode 2 "$COMMAND" 13 >> "$OUT_PIPE" - encode 2 "$COMMAND" 13 >&3 - - RESPONSE=$(read-response "$IN_PIPE") - response-payload "$RESPONSE" -} - rcon-command () { - HOST="$1" - PORT="$2" - PASSWORD="$3" - COMMAND="$4" + HOST="$(echo $1 | cut -d: -f1)" + PORT="$(echo $1 | cut -d: -f2)" + PASSWORD="$(echo $1 | cut -d: -f3-)" + COMMAND="$2" + + reverse-hex-endian () { + # Given a 4-byte hex integer, reverse endianness + while read -r -d '' -N 8 INTEGER; do + echo "$INTEGER" | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/' + done + } + + decode-hex-int () { + # decode little-endian hex integer + while read -r -d '' -N 8 INTEGER; do + BIG_ENDIAN_HEX=$(echo "$INTEGER" | reverse-hex-endian) + echo "$((16#$BIG_ENDIAN_HEX))" + done + } + + stream-to-hex () { + xxd -ps + } + + hex-to-stream () { + xxd -ps -r + } + + encode-int () { + # Encode an integer as 4 bytes in little endian and return as hex + INT="$1" + # Source: https://stackoverflow.com/a/9955198 + printf "%08x" "$INT" | sed -E 's/(..)(..)(..)(..)/\4\3\2\1/' + } + + encode () { + # Encode a packet type and payload for the rcon protocol + TYPE="$1" + PAYLOAD="$2" + REQUEST_ID="$3" + PAYLOAD_LENGTH="${#PAYLOAD}" + TOTAL_LENGTH="$((4 + 4 + PAYLOAD_LENGTH + 1 + 1))" + + OUTPUT="" + OUTPUT+=$(encode-int "$TOTAL_LENGTH") + OUTPUT+=$(encode-int "$REQUEST_ID") + OUTPUT+=$(encode-int "$TYPE") + OUTPUT+=$(echo -n "$PAYLOAD" | stream-to-hex) + OUTPUT+="0000" + + echo -n "$OUTPUT" | hex-to-stream + } + + read-response () { + # read next response packet and return the payload text + HEX_LENGTH=$(head -c4 <&3 | stream-to-hex | reverse-hex-endian) + LENGTH=$((16#$HEX_LENGTH)) + + RESPONSE_PAYLOAD=$(head -c $LENGTH <&3 | stream-to-hex) + echo -n "$RESPONSE_PAYLOAD" + } + + response-request-id () { + echo -n "${1:0:8}" | decode-hex-int + } + + response-type () { + echo -n "${1:8:8}" | decode-hex-int + } + + response-payload () { + echo -n "${1:16:-4}" | hex-to-stream + } + + login () { + PASSWORD="$1" + encode 3 "$PASSWORD" 12 >&3 + + RESPONSE=$(read-response "$IN_PIPE") + + RESPONSE_REQUEST_ID=$(response-request-id "$RESPONSE") + if [ "$RESPONSE_REQUEST_ID" -eq -1 ] || [ "$RESPONSE_REQUEST_ID" -eq 4294967295 ]; then + echo "Authentication failed: Wrong RCON password" 1>&2 + return 1 + fi + } + + run-command () { + COMMAND="$1" + + # encode 2 "$COMMAND" 13 >> "$OUT_PIPE" + encode 2 "$COMMAND" 13 >&3 + + RESPONSE=$(read-response "$IN_PIPE") + response-payload "$RESPONSE" + } # Open a TCP socket # Source: https://www.xmodulo.com/tcp-udp-socket-bash-shell.html @@ -110,4 +116,4 @@ PORT="$2" PASSWORD="$3" COMMAND="$4" -rcon-command "$HOST" "$PORT" "$PASSWORD" "$COMMAND" +rcon-command "$HOST:$PORT:$PASSWORD" "$COMMAND" diff --git a/test/mock_rcon.py b/test/mock_rcon.py new file mode 100644 index 0000000..c5f7c35 --- /dev/null +++ b/test/mock_rcon.py @@ -0,0 +1,31 @@ +# Reference: https://docs.python.org/3/library/socketserver.html +import codecs +import socketserver +import sys + +class Handler(socketserver.BaseRequestHandler): + def handle(self): + while True: + length = int.from_bytes(self.request.recv(4), 'little') + if not length: + continue + + self.request.recv(4) + type_ = int.from_bytes(self.request.recv(4), 'little') + self.data = self.request.recv(length - 8)[:-2].decode('utf-8') + if self.data: + if type_ == 2: + print(self.data) + sys.stdout.flush() + try: + if type_ == 3 and self.data != sys.argv[2]: + self.request.sendall(codecs.decode('0a000000ffffffff020000000000', 'hex')) + else: + self.request.sendall(codecs.decode('0a00000010000000020000000000', 'hex')) + except: + break + +if __name__ == "__main__": + HOST, PORT = "localhost", int(sys.argv[1]) + with socketserver.ThreadingTCPServer((HOST, PORT), Handler) as server: + server.serve_forever() diff --git a/test/test.sh b/test/test.sh index 7e3e0c9..ae1b0ce 100755 --- a/test/test.sh +++ b/test/test.sh @@ -5,6 +5,8 @@ TEST_DIR="test" TEST_TMP="$TEST_DIR/tmp" SCREEN_TMP="tmp-screen" +RCON_PORT="8088" +RCON_PASSWORD="supersecret" setUp () { rm -rf "$TEST_TMP" mkdir -p "$TEST_TMP/server/world" @@ -13,6 +15,8 @@ setUp () { echo "file2" > "$TEST_TMP/server/world/file2.txt" echo "file3" > "$TEST_TMP/server/world/file3.txt" + python test/mock_rcon.py "$RCON_PORT" "$RCON_PASSWORD" > "$TEST_TMP/rcon-output" & + echo "$!" > "$TEST_TMP/rcon-pid" screen -dmS "$SCREEN_TMP" bash screen -S "$SCREEN_TMP" -X stuff "cat > $TEST_TMP/screen-output\n" tmux new-session -d -s "$SCREEN_TMP" @@ -21,6 +25,8 @@ setUp () { } tearDown () { + RCON_PID="$(cat "$TEST_TMP/rcon-pid")" + kill "$RCON_PID" >/dev/null 2>&1 || true screen -S "$SCREEN_TMP" -X quit >/dev/null 2>&1 || true tmux kill-session -t "$SCREEN_TMP" >/dev/null 2>&1 || true } @@ -88,7 +94,7 @@ test-missing-options () { OUTPUT="$(./backup.sh 2>&1)" EXIT_CODE="$?" assertEquals 1 "$EXIT_CODE" - assertContains "$OUTPUT" "Minecraft screen name not specified" + assertContains "$OUTPUT" "Minecraft screen/tmux/rcon location not specified (use -s)" assertContains "$OUTPUT" "Server world not specified" assertContains "$OUTPUT" "Backup directory not specified" } @@ -97,7 +103,7 @@ test-missing-options-suppress-warnings () { OUTPUT="$(./backup.sh -q 2>&1)" EXIT_CODE="$?" assertEquals 1 "$EXIT_CODE" - assertNotContains "$OUTPUT" "Minecraft screen name not specified" + assertNotContains "$OUTPUT" "Minecraft screen/tmux/rcon location not specified (use -s)" } test-empty-world-warning () { @@ -129,6 +135,20 @@ test-tmux-interface () { assertEquals "$SCREEN_CONTENTS" "$EXPECTED_CONTENTS" } +test-rcon-interface () { + TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01")" + ./backup.sh -w rcon -i "$TEST_TMP/server/world" -o "$TEST_TMP/backups" -s "localhost:$RCON_PORT:$RCON_PASSWORD" -f "$TIMESTAMP" + EXPECTED_CONTENTS=$(echo -e "save-off\nsave-on\nsave-all") + SCREEN_CONTENTS="$(head -n3 "$TEST_TMP/rcon-output")" + assertEquals "$SCREEN_CONTENTS" "$EXPECTED_CONTENTS" +} + +test-rcon-interface-wrong-password () { + TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01")" + OUTPUT="$(./backup.sh -w rcon -i "$TEST_TMP/server/world" -o "$TEST_TMP/backups" -s "localhost:$RCON_PORT:wrong$RCON_PASSWORD" -f "$TIMESTAMP" 2>&1)" + assertContains "$OUTPUT" "Wrong RCON password" +} + test-sequential-delete () { for i in $(seq 0 99); do TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01 +$i hour")" From a2ff9b0b0bab8d3d8eddec73fd8c4f0cd2cc9d15 Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 22:14:57 -0800 Subject: [PATCH 06/21] Update help --- README.md | 6 +++--- backup.sh | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index b6bda51..694dc5b 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![codecov](https://codecov.io/gh/nicolaschan/minecraft-backup/branch/master/graph/badge.svg?token=LCbVC4TbYJ)](https://codecov.io/gh/nicolaschan/minecraft-backup) Backup script for Minecraft servers on Linux. -Supports servers running in [screen](https://en.wikipedia.org/wiki/GNU_Screen), [tmux](https://en.wikipedia.org/wiki/Tmux), or with [rcon](https://wiki.vg/RCON) enabled. +Supports servers running in [screen](https://en.wikipedia.org/wiki/GNU_Screen), [tmux](https://en.wikipedia.org/wiki/Tmux), or with [RCON](https://wiki.vg/RCON) enabled. ## Features - Create backups of your world folder @@ -40,9 +40,9 @@ Command line options: -o Output directory -p Prefix that shows in Minecraft chat (default: Backup) -q Suppress warnings --s Minecraft server screen name +-s Screen name, tmux session name, or hostname:port:password for RCON -v Verbose mode --w Window manager: screen (default) or tmux +-w Window manager: screen (default), tmux, RCON ``` ### Automate backups with cron diff --git a/backup.sh b/backup.sh index 1b0a487..7ac2fc2 100755 --- a/backup.sh +++ b/backup.sh @@ -47,9 +47,9 @@ while getopts 'a:cd:e:f:hi:l:m:o:p:qs:vw:' FLAG; do echo "-o Output directory" echo "-p Prefix that shows in Minecraft chat (default: Backup)" echo "-q Suppress warnings" - echo "-s Screen name, tmux session name, or hostname:port:password for rcon" + echo "-s Screen name, tmux session name, or hostname:port:password for RCON" echo "-v Verbose mode" - echo "-w Window manager: screen (default), tmux, rcon" + echo "-w Window manager: screen (default), tmux, RCON" exit 0 ;; i) SERVER_WORLD=$OPTARG ;; From ad1dc772b6f82545a38f17b380c02cafb842d635 Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 22:19:00 -0800 Subject: [PATCH 07/21] Allow capital RCON --- backup.sh | 2 +- test/test.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/backup.sh b/backup.sh index 7ac2fc2..5869296 100755 --- a/backup.sh +++ b/backup.sh @@ -223,7 +223,7 @@ execute-command () { ;; "tmux") tmux send-keys -t "$SCREEN_NAME" "$COMMAND" ENTER ;; - "rcon") rcon-command "$SCREEN_NAME" "$COMMAND" + "RCON"|"rcon") rcon-command "$SCREEN_NAME" "$COMMAND" ;; esac fi diff --git a/test/test.sh b/test/test.sh index ae1b0ce..e9e21b1 100755 --- a/test/test.sh +++ b/test/test.sh @@ -21,7 +21,7 @@ setUp () { screen -S "$SCREEN_TMP" -X stuff "cat > $TEST_TMP/screen-output\n" tmux new-session -d -s "$SCREEN_TMP" tmux send-keys -t "$SCREEN_TMP" "cat > $TEST_TMP/tmux-output" ENTER - sleep 0.5 + sleep 1 } tearDown () { @@ -145,7 +145,7 @@ test-rcon-interface () { test-rcon-interface-wrong-password () { TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01")" - OUTPUT="$(./backup.sh -w rcon -i "$TEST_TMP/server/world" -o "$TEST_TMP/backups" -s "localhost:$RCON_PORT:wrong$RCON_PASSWORD" -f "$TIMESTAMP" 2>&1)" + OUTPUT="$(./backup.sh -w RCON -i "$TEST_TMP/server/world" -o "$TEST_TMP/backups" -s "localhost:$RCON_PORT:wrong$RCON_PASSWORD" -f "$TIMESTAMP" 2>&1)" assertContains "$OUTPUT" "Wrong RCON password" } From 71311e23219e62436da5e7a7bd958d3de6e6ee7e Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 22:23:12 -0800 Subject: [PATCH 08/21] Reorder setUp --- test/test.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/test.sh b/test/test.sh index e9e21b1..4bccce7 100755 --- a/test/test.sh +++ b/test/test.sh @@ -15,12 +15,12 @@ setUp () { echo "file2" > "$TEST_TMP/server/world/file2.txt" echo "file3" > "$TEST_TMP/server/world/file3.txt" - python test/mock_rcon.py "$RCON_PORT" "$RCON_PASSWORD" > "$TEST_TMP/rcon-output" & - echo "$!" > "$TEST_TMP/rcon-pid" screen -dmS "$SCREEN_TMP" bash screen -S "$SCREEN_TMP" -X stuff "cat > $TEST_TMP/screen-output\n" tmux new-session -d -s "$SCREEN_TMP" tmux send-keys -t "$SCREEN_TMP" "cat > $TEST_TMP/tmux-output" ENTER + python test/mock_rcon.py "$RCON_PORT" "$RCON_PASSWORD" > "$TEST_TMP/rcon-output" & + echo "$!" > "$TEST_TMP/rcon-pid" sleep 1 } From 80abd684babc68eaa8b209c977585a4b4c3da16e Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 22:59:31 -0800 Subject: [PATCH 09/21] Pass shellcheck --- backup.sh | 85 ++++++++++++++++++++++++++++------------------------ rcon.sh | 6 ++-- test/test.sh | 13 +++++++- 3 files changed, 61 insertions(+), 43 deletions(-) diff --git a/backup.sh b/backup.sh index 5869296..b57a5a3 100755 --- a/backup.sh +++ b/backup.sh @@ -26,6 +26,13 @@ WINDOW_MANAGER="screen" # Choices: screen, tmux DATE_FORMAT="%F_%H-%M-%S" TIMESTAMP=$(date +$DATE_FORMAT) +log-fatal () { + echo -e "\033[0;31mFATAL:\033[0m $*" +} +log-warning () { + echo -e "\033[0;33mWARNING:\033[0m $*" +} + while getopts 'a:cd:e:f:hi:l:m:o:p:qs:vw:' FLAG; do case $FLAG in a) COMPRESSION_ALGORITHM=$OPTARG ;; @@ -61,20 +68,14 @@ while getopts 'a:cd:e:f:hi:l:m:o:p:qs:vw:' FLAG; do s) SCREEN_NAME=$OPTARG ;; v) DEBUG=true ;; w) WINDOW_MANAGER=$OPTARG ;; + *) log-fatal "Invalid option -$FLAG"; exit 1 ;; esac done -log-fatal () { - echo -e "\033[0;31mFATAL:\033[0m $*" -} -log-warning () { - echo -e "\033[0;33mWARNING:\033[0m $*" -} - rcon-command () { - HOST="$(echo $1 | cut -d: -f1)" - PORT="$(echo $1 | cut -d: -f2)" - PASSWORD="$(echo $1 | cut -d: -f3-)" + HOST="$(echo "$1" | cut -d: -f1)" + PORT="$(echo "$1" | cut -d: -f2)" + PASSWORD="$(echo "$1" | cut -d: -f3-)" COMMAND="$2" reverse-hex-endian () { @@ -255,50 +256,54 @@ message-players "Starting backup..." "$ARCHIVE_FILE_NAME" # Parse file timestamp to one readable by "date" parse-file-timestamp () { - local DATE_STRING=$(echo $1 | awk -F_ '{gsub(/-/,":",$2); print $1" "$2}') - echo $DATE_STRING + local DATE_STRING + DATE_STRING="$(echo "$1" | awk -F_ '{gsub(/-/,":",$2); print $1" "$2}')" + echo "$DATE_STRING" } # Delete a backup delete-backup () { local BACKUP=$1 - rm $BACKUP_DIRECTORY/$BACKUP + rm "$BACKUP_DIRECTORY"/"$BACKUP" message-players "Deleted old backup" "$BACKUP" } # Sequential delete method delete-sequentially () { - local BACKUPS=($(ls $BACKUP_DIRECTORY)) + local BACKUPS=("$BACKUP_DIRECTORY"/*) while [[ $MAX_BACKUPS -ge 0 && ${#BACKUPS[@]} -gt $MAX_BACKUPS ]]; do - delete-backup ${BACKUPS[0]} - BACKUPS=($(ls $BACKUP_DIRECTORY)) + delete-backup "$(basename "${BACKUPS[0]}")" + BACKUPS=("$BACKUP_DIRECTORY"/*) done } # Functions to sort backups into correct categories based on timestamps is-hourly-backup () { local TIMESTAMP=$* - local MINUTE=$(date -d "$TIMESTAMP" +%M) - return $MINUTE + local MINUTE + MINUTE=$(date -d "$TIMESTAMP" +%M) + return "$MINUTE" } is-daily-backup () { local TIMESTAMP=$* - local HOUR=$(date -d "$TIMESTAMP" +%H) - return $HOUR + local HOUR + HOUR=$(date -d "$TIMESTAMP" +%H) + return "$HOUR" } is-weekly-backup () { local TIMESTAMP=$* - local DAY=$(date -d "$TIMESTAMP" +%u) - return $((DAY - 1)) + local DAY + DAY=$(date -d "$TIMESTAMP" +%u) + return "$((DAY - 1))" } # Helper function to sum an array array-sum () { SUM=0 - for NUMBER in $*; do + for NUMBER in "$@"; do (( SUM += NUMBER )) done - echo $SUM + echo "$SUM" } # Thinning delete method @@ -310,7 +315,7 @@ delete-thinning () { local BLOCK_FUNCTIONS=("is-hourly-backup" "is-daily-backup" "is-weekly-backup") # Warn if $MAX_BACKUPS does not have enough room for all the blocks - TOTAL_BLOCK_SIZE=$(array-sum ${BLOCK_SIZES[@]}) + TOTAL_BLOCK_SIZE=$(array-sum "${BLOCK_SIZES[@]}") if [[ $MAX_BACKUPS != -1 ]] && [[ $TOTAL_BLOCK_SIZE -gt $MAX_BACKUPS ]]; then if ! $SUPPRESS_WARNINGS; then log-warning "MAX_BACKUPS ($MAX_BACKUPS) is smaller than TOTAL_BLOCK_SIZE ($TOTAL_BLOCK_SIZE)" @@ -318,19 +323,21 @@ delete-thinning () { fi local CURRENT_INDEX=0 - local BACKUPS=($(ls -r $BACKUP_DIRECTORY)) # List newest first + local BACKUPS=("$BACKUP_DIRECTORY"/*) # List newest first - for BLOCK_INDEX in ${!BLOCK_SIZES[@]}; do + for BLOCK_INDEX in "${!BLOCK_SIZES[@]}"; do local BLOCK_SIZE=${BLOCK_SIZES[BLOCK_INDEX]} local BLOCK_FUNCTION=${BLOCK_FUNCTIONS[BLOCK_INDEX]} local OLDEST_BACKUP_IN_BLOCK_INDEX=$((BLOCK_SIZE + CURRENT_INDEX)) # Not an off-by-one error because a new backup was already saved - local OLDEST_BACKUP_IN_BLOCK=${BACKUPS[OLDEST_BACKUP_IN_BLOCK_INDEX]} + local OLDEST_BACKUP_IN_BLOCK + OLDEST_BACKUP_IN_BLOCK="$(basename "${BACKUPS[OLDEST_BACKUP_IN_BLOCK_INDEX]}")" - if [[ $OLDEST_BACKUP_IN_BLOCK == "" ]]; then + if [[ "$OLDEST_BACKUP_IN_BLOCK" == "" ]]; then break fi - local OLDEST_BACKUP_TIMESTAMP=$(parse-file-timestamp ${OLDEST_BACKUP_IN_BLOCK:0:19}) + local OLDEST_BACKUP_TIMESTAMP + OLDEST_BACKUP_TIMESTAMP=$(parse-file-timestamp "${OLDEST_BACKUP_IN_BLOCK:0:19}") local BLOCK_COMMAND="$BLOCK_FUNCTION $OLDEST_BACKUP_TIMESTAMP" if $BLOCK_COMMAND; then @@ -340,7 +347,7 @@ delete-thinning () { fi else # Oldest backup in this block does not satisfy the condition for placement in next block - delete-backup $OLDEST_BACKUP_IN_BLOCK + delete-backup "$OLDEST_BACKUP_IN_BLOCK" break fi @@ -351,7 +358,7 @@ delete-thinning () { } # Ensure directory exists -mkdir -p "$(dirname $ARCHIVE_PATH)" +mkdir -p "$(dirname "$ARCHIVE_PATH")" # Disable world autosaving execute-command "save-off" @@ -360,10 +367,10 @@ execute-command "save-off" START_TIME=$(date +"%s") case $COMPRESSION_ALGORITHM in # No compression - "") tar -cf $ARCHIVE_PATH -C $SERVER_WORLD . + "") tar -cf "$ARCHIVE_PATH" -C "$SERVER_WORLD" . ;; # With compression - *) tar -cf - -C $SERVER_WORLD . | $COMPRESSION_ALGORITHM -cv -$COMPRESSION_LEVEL - > $ARCHIVE_PATH 2>> /dev/null + *) tar -cf - -C "$SERVER_WORLD" . | $COMPRESSION_ALGORITHM -cv -"$COMPRESSION_LEVEL" - > "$ARCHIVE_PATH" 2>> /dev/null ;; esac sync @@ -386,15 +393,15 @@ delete-old-backups () { } # Notify players of completion -WORLD_SIZE_BYTES=$(du -b --max-depth=0 $SERVER_WORLD | awk '{print $1}') -ARCHIVE_SIZE_BYTES=$(du -b $ARCHIVE_PATH | awk '{print $1}') -ARCHIVE_SIZE=$(du -h $ARCHIVE_PATH | awk '{print $1}') -BACKUP_DIRECTORY_SIZE=$(du -h --max-depth=0 $BACKUP_DIRECTORY | awk '{print $1}') +WORLD_SIZE_BYTES=$(du -b --max-depth=0 "$SERVER_WORLD" | awk '{print $1}') +ARCHIVE_SIZE_BYTES=$(du -b "$ARCHIVE_PATH" | awk '{print $1}') +ARCHIVE_SIZE=$(du -h "$ARCHIVE_PATH" | awk '{print $1}') +BACKUP_DIRECTORY_SIZE=$(du -h --max-depth=0 "$BACKUP_DIRECTORY" | awk '{print $1}') TIME_DELTA=$((END_TIME - START_TIME)) # Check that archive size is not null and at least 200 Bytes if [[ "$WORLD_SIZE_BYTES" -gt 0 && "$ARCHIVE_SIZE" != "" && "$ARCHIVE_SIZE_BYTES" -gt 200 ]]; then - COMPRESSION_PERCENT=$(($ARCHIVE_SIZE_BYTES * 100 / $WORLD_SIZE_BYTES)) + COMPRESSION_PERCENT=$((ARCHIVE_SIZE_BYTES * 100 / WORLD_SIZE_BYTES)) message-players-success "Backup complete!" "$TIME_DELTA s, $ARCHIVE_SIZE/$BACKUP_DIRECTORY_SIZE, $COMPRESSION_PERCENT%" delete-old-backups else diff --git a/rcon.sh b/rcon.sh index cc4f679..9bf8e58 100755 --- a/rcon.sh +++ b/rcon.sh @@ -1,9 +1,9 @@ #!/usr/bin/env bash rcon-command () { - HOST="$(echo $1 | cut -d: -f1)" - PORT="$(echo $1 | cut -d: -f2)" - PASSWORD="$(echo $1 | cut -d: -f3-)" + HOST="$(echo "$1" | cut -d: -f1)" + PORT="$(echo "$1" | cut -d: -f2)" + PASSWORD="$(echo "$1" | cut -d: -f3-)" COMMAND="$2" reverse-hex-endian () { diff --git a/test/test.sh b/test/test.sh index 4bccce7..1435ff2 100755 --- a/test/test.sh +++ b/test/test.sh @@ -21,7 +21,10 @@ setUp () { tmux send-keys -t "$SCREEN_TMP" "cat > $TEST_TMP/tmux-output" ENTER python test/mock_rcon.py "$RCON_PORT" "$RCON_PASSWORD" > "$TEST_TMP/rcon-output" & echo "$!" > "$TEST_TMP/rcon-pid" - sleep 1 + + while ! [[ (-f "$TEST_TMP/screen-output") && (-f "$TEST_TMP/tmux-output") && (-f "$TEST_TMP/rcon-output") ]]; do + sleep 0.1 + done } tearDown () { @@ -106,6 +109,13 @@ test-missing-options-suppress-warnings () { assertNotContains "$OUTPUT" "Minecraft screen/tmux/rcon location not specified (use -s)" } +test-invalid-options () { + OUTPUT="$(./backup.sh -z 2>&1)" + EXIT_CODE="$?" + assertEquals 1 "$EXIT_CODE" + assertContains "$OUTPUT" "Invalid option" +} + test-empty-world-warning () { mkdir -p "$TEST_TMP/server/empty-world" OUTPUT="$(./backup.sh -v -i "$TEST_TMP/server/empty-world" -o "$TEST_TMP/backups" -s "$SCREEN_TMP" -f "$TIMESTAMP" 2>&1)" @@ -158,6 +168,7 @@ test-sequential-delete () { TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01 +$i hour")" check-backup "$TIMESTAMP.tar.gz" done + assertEquals 10 "$(find "$TEST_TMP/backups" -type f | wc -l)" } test-thinning-delete () { From 0340f64b238012c07cadf35314897b2d99e8023a Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 23:01:39 -0800 Subject: [PATCH 10/21] Add shellcheck check --- .github/workflows/ci.yml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 73b7e74..443354e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -23,8 +23,8 @@ jobs: with: submodules: 'recursive' - - name: Install kcov - run: sudo apt-get install -y kcov + - name: Install kcov and shellcheck + run: sudo apt-get install -y kcov shellcheck - name: Run tests run: test/test.sh @@ -37,4 +37,7 @@ jobs: with: token: ${{ secrets.CODECOV_TOKEN }} file: ./coverage/test.sh/cov.xml + + - name: shellcheck + run: shellcheck backup.sh From 57fe77cb59de00ca09ebeda7c4d2d43b92dc65ad Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 23:01:58 -0800 Subject: [PATCH 11/21] Rename job to test --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 443354e..e7a6e64 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -12,7 +12,7 @@ on: # A workflow run is made up of one or more jobs that can run sequentially or in parallel jobs: # This workflow contains a single job called "build" - build: + test: # The type of runner that the job will run on runs-on: ubuntu-latest From 1402cfd86a564eb35ad04051bce626c5277e084c Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 23:10:18 -0800 Subject: [PATCH 12/21] Update comments --- backup.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/backup.sh b/backup.sh index b57a5a3..d854d9b 100755 --- a/backup.sh +++ b/backup.sh @@ -8,7 +8,7 @@ # For most convenience, run automatically with cron. # Default Configuration -SCREEN_NAME="" # Name of the GNU Screen or tmux pane your Minecraft server is running in +SCREEN_NAME="" # Name of the GNU Screen, tmux session, or hostname:port:password for RCON SERVER_WORLD="" # Server world directory BACKUP_DIRECTORY="" # Directory to save backups in MAX_BACKUPS=128 # -1 indicates unlimited @@ -20,7 +20,7 @@ ENABLE_CHAT_MESSAGES=false # Tell players in Minecraft chat about backup status PREFIX="Backup" # Shows in the chat message DEBUG=false # Enable debug messages SUPPRESS_WARNINGS=false # Suppress warnings -WINDOW_MANAGER="screen" # Choices: screen, tmux +WINDOW_MANAGER="screen" # Choices: screen, tmux, RCON # Other Variables (do not modify) DATE_FORMAT="%F_%H-%M-%S" From 6495870e1a34950080363f18a44ba89c5f9c7e8d Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Wed, 3 Mar 2021 23:19:55 -0800 Subject: [PATCH 13/21] Improve debug logging --- backup.sh | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/backup.sh b/backup.sh index d854d9b..1b43752 100755 --- a/backup.sh +++ b/backup.sh @@ -32,6 +32,11 @@ log-fatal () { log-warning () { echo -e "\033[0;33mWARNING:\033[0m $*" } +debug-log () { + if "$DEBUG"; then + echo "$1" + fi +} while getopts 'a:cd:e:f:hi:l:m:o:p:qs:vw:' FLAG; do case $FLAG in @@ -175,7 +180,7 @@ rcon-command () { exec 3<>/dev/tcp/"$HOST"/"$PORT" login "$PASSWORD" || return 1 - run-command "$COMMAND" + debug-log "$(run-command "$COMMAND")" # Close the socket exec 3<&- @@ -243,9 +248,7 @@ message-players-color () { local MESSAGE=$1 local HOVER_MESSAGE=$2 local COLOR=$3 - if $DEBUG; then - echo "$MESSAGE ($HOVER_MESSAGE)" - fi + debug-log "$MESSAGE ($HOVER_MESSAGE)" if $ENABLE_CHAT_MESSAGES; then execute-command "tellraw @a [\"\",{\"text\":\"[$PREFIX] \",\"color\":\"gray\",\"italic\":true},{\"text\":\"$MESSAGE\",\"color\":\"$COLOR\",\"italic\":true,\"hoverEvent\":{\"action\":\"show_text\",\"value\":{\"text\":\"\",\"extra\":[{\"text\":\"$HOVER_MESSAGE\"}]}}}]" fi @@ -342,9 +345,7 @@ delete-thinning () { if $BLOCK_COMMAND; then # Oldest backup in this block satisfies the condition for placement in the next block - if $DEBUG; then - echo "$OLDEST_BACKUP_IN_BLOCK promoted to next block" - fi + debug-log "$OLDEST_BACKUP_IN_BLOCK promoted to next block" else # Oldest backup in this block does not satisfy the condition for placement in next block delete-backup "$OLDEST_BACKUP_IN_BLOCK" From b1e744d1dba36f1e9edc91e23124809748d0608d Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Thu, 4 Mar 2021 11:23:20 -0800 Subject: [PATCH 14/21] Add alert for when backup has nonzero exit code --- backup.sh | 15 ++++++++++++--- test/test.sh | 8 ++++++++ 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/backup.sh b/backup.sh index 1b43752..41dd003 100755 --- a/backup.sh +++ b/backup.sh @@ -212,8 +212,8 @@ if $MISSING_CONFIGURATION; then exit 1 fi -ARCHIVE_FILE_NAME=$TIMESTAMP.tar$COMPRESSION_FILE_EXTENSION -ARCHIVE_PATH=$BACKUP_DIRECTORY/$ARCHIVE_FILE_NAME +ARCHIVE_FILE_NAME="$TIMESTAMP.tar$COMPRESSION_FILE_EXTENSION" +ARCHIVE_PATH="$BACKUP_DIRECTORY/$ARCHIVE_FILE_NAME" # Minecraft server screen interface functions message-players () { @@ -374,6 +374,10 @@ case $COMPRESSION_ALGORITHM in *) tar -cf - -C "$SERVER_WORLD" . | $COMPRESSION_ALGORITHM -cv -"$COMPRESSION_LEVEL" - > "$ARCHIVE_PATH" 2>> /dev/null ;; esac +ARCHIVE_EXIT_CODE="$?" +if [ $ARCHIVE_EXIT_CODE -ne 0 ]; then + log-fatal "Archive command exited with nonzero exit code $ARCHIVE_EXIT_CODE" +fi sync END_TIME=$(date +"%s") @@ -401,10 +405,15 @@ BACKUP_DIRECTORY_SIZE=$(du -h --max-depth=0 "$BACKUP_DIRECTORY" | awk '{print $1 TIME_DELTA=$((END_TIME - START_TIME)) # Check that archive size is not null and at least 200 Bytes -if [[ "$WORLD_SIZE_BYTES" -gt 0 && "$ARCHIVE_SIZE" != "" && "$ARCHIVE_SIZE_BYTES" -gt 200 ]]; then +if [[ "$ARCHIVE_EXIT_CODE" -eq 0 && "$WORLD_SIZE_BYTES" -gt 0 && "$ARCHIVE_SIZE" != "" && "$ARCHIVE_SIZE_BYTES" -gt 200 ]]; then COMPRESSION_PERCENT=$((ARCHIVE_SIZE_BYTES * 100 / WORLD_SIZE_BYTES)) message-players-success "Backup complete!" "$TIME_DELTA s, $ARCHIVE_SIZE/$BACKUP_DIRECTORY_SIZE, $COMPRESSION_PERCENT%" delete-old-backups else message-players-error "Backup was not saved!" "Please notify an administrator" + if [ "$ARCHIVE_EXIT_CODE" -ne 0 ]; then + exit "$ARCHIVE_EXIT_CODE" + else + exit 1 + fi fi diff --git a/test/test.sh b/test/test.sh index 1435ff2..2e0d0b7 100755 --- a/test/test.sh +++ b/test/test.sh @@ -129,6 +129,14 @@ test-block-size-warning () { assertContains "$OUTPUT" "is smaller than TOTAL_BLOCK_SIZE" } +test-nonzero-exit-warning () { + TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01")" + OUTPUT="$(./backup.sh -a _BLAH_ -i "$TEST_TMP/server/world" -o "$TEST_TMP/backups" -s "$SCREEN_TMP" -f "$TIMESTAMP" 2>&1)" + EXIT_CODE="$?" + assertNotEquals 0 "$EXIT_CODE" + assertContains "$OUTPUT" "Archive command exited with nonzero exit code" +} + test-screen-interface () { TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01")" ./backup.sh -i "$TEST_TMP/server/world" -o "$TEST_TMP/backups" -s "$SCREEN_TMP" -f "$TIMESTAMP" From 8756a0e6146737b8856b9783f931f00ca87e50ca Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Thu, 4 Mar 2021 11:37:38 -0800 Subject: [PATCH 15/21] Check tar status --- backup.sh | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/backup.sh b/backup.sh index 41dd003..9a78198 100755 --- a/backup.sh +++ b/backup.sh @@ -309,6 +309,19 @@ array-sum () { echo "$SUM" } +# Given two exit codes, print a nonzero one if there is one +exit-code () { + if [ $1 -ne 0 ]; then + echo "$1" + else + if [[ "$2" == "" ]]; then + echo 0 + else + echo "$2" + fi + fi +} + # Thinning delete method delete-thinning () { # sub-hourly, hourly, daily, weekly is everything else @@ -358,7 +371,7 @@ delete-thinning () { delete-sequentially } -# Ensure directory exists +# Ensure backup directory exists mkdir -p "$(dirname "$ARCHIVE_PATH")" # Disable world autosaving @@ -374,7 +387,7 @@ case $COMPRESSION_ALGORITHM in *) tar -cf - -C "$SERVER_WORLD" . | $COMPRESSION_ALGORITHM -cv -"$COMPRESSION_LEVEL" - > "$ARCHIVE_PATH" 2>> /dev/null ;; esac -ARCHIVE_EXIT_CODE="$?" +ARCHIVE_EXIT_CODE="$(exit-code "${PIPESTATUS[0]}" "${PIPESTATUS[1]}")" if [ $ARCHIVE_EXIT_CODE -ne 0 ]; then log-fatal "Archive command exited with nonzero exit code $ARCHIVE_EXIT_CODE" fi From ade562aeeb2ee01487a542bccb415179a01ed40e Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Thu, 4 Mar 2021 11:50:10 -0800 Subject: [PATCH 16/21] Fix shellcheck --- backup.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/backup.sh b/backup.sh index 9a78198..4100f73 100755 --- a/backup.sh +++ b/backup.sh @@ -311,7 +311,7 @@ array-sum () { # Given two exit codes, print a nonzero one if there is one exit-code () { - if [ $1 -ne 0 ]; then + if [ "$1" -ne 0 ]; then echo "$1" else if [[ "$2" == "" ]]; then @@ -388,7 +388,7 @@ case $COMPRESSION_ALGORITHM in ;; esac ARCHIVE_EXIT_CODE="$(exit-code "${PIPESTATUS[0]}" "${PIPESTATUS[1]}")" -if [ $ARCHIVE_EXIT_CODE -ne 0 ]; then +if [ "$ARCHIVE_EXIT_CODE" -ne 0 ]; then log-fatal "Archive command exited with nonzero exit code $ARCHIVE_EXIT_CODE" fi sync From 38251cf0105e3461a486079ae87c77b6ba86ee71 Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Thu, 4 Mar 2021 13:03:40 -0800 Subject: [PATCH 17/21] Update examples --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 694dc5b..bee0efd 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,14 @@ chmod +x backup.sh ## Usage ```bash +# If connecting with RCON: +./backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s localhost:25575:secret -w rcon + +# If running on screen called "minecraft": ./backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s minecraft + +# If running on tmux session 0: +./backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s 0 -w tmux ``` This will show chat messages (`-c`) in the screen called "minecraft" and save a backup of `/home/user/mcserver/world` into `/mnt/storage/backups` using the default thinning deletion policy for old backups. From c079e0f1f5d18c8db443b5d1f19408c40b2fa86a Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Thu, 4 Mar 2021 13:05:51 -0800 Subject: [PATCH 18/21] Touch up readme --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index bee0efd..fc5dcd3 100644 --- a/README.md +++ b/README.md @@ -22,16 +22,16 @@ chmod +x backup.sh ## Usage ```bash # If connecting with RCON: -./backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s localhost:25575:secret -w rcon +./backup.sh -c -i /home/user/server/world -o /mnt/storage/backups -s localhost:25575:secret -w rcon # If running on screen called "minecraft": -./backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s minecraft +./backup.sh -c -i /home/user/server/world -o /mnt/storage/backups -s minecraft # If running on tmux session 0: -./backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s 0 -w tmux +./backup.sh -c -i /home/user/server/world -o /mnt/storage/backups -s 0 -w tmux ``` -This will show chat messages (`-c`) in the screen called "minecraft" and save a backup of `/home/user/mcserver/world` into `/mnt/storage/backups` using the default thinning deletion policy for old backups. +This will show chat messages (`-c`) and save a backup of `/home/user/server/world` into `/mnt/storage/backups` using the default thinning deletion policy for old backups. Command line options: ```text @@ -56,7 +56,7 @@ Command line options: - Edit the crontab with `crontab -e` - Example for hourly backups: ``` -00 * * * * /path/to/backup.sh -c -i /home/user/mcserver/world -o /mnt/storage/backups -s minecraft +00 * * * * /path/to/backup.sh -c -i /home/user/server/world -o /mnt/storage/backups -s minecraft ``` ## Retrieving Backups From 468ef0bd495c741e2006ef8a871e8a2df37f86b8 Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Thu, 4 Mar 2021 13:14:53 -0800 Subject: [PATCH 19/21] Add spaces backup test --- test/test.sh | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/test/test.sh b/test/test.sh index 2e0d0b7..7b35b4f 100755 --- a/test/test.sh +++ b/test/test.sh @@ -44,12 +44,18 @@ assert-equals-directory () { fi } +check-backup-full-paths () { + BACKUP_ARCHIVE="$1" + WORLD_DIR="$2" + mkdir -p "$TEST_TMP/restored" + tar --extract --file "$BACKUP_ARCHIVE" --directory "$TEST_TMP/restored" + assert-equals-directory "$WORLD_DIR" "$TEST_TMP/restored" + rm -rf "$TEST_TMP/restored" +} + check-backup () { BACKUP_ARCHIVE="$1" - mkdir -p "$TEST_TMP/restored" - tar --extract --file "$TEST_TMP/backups/$BACKUP_ARCHIVE" --directory "$TEST_TMP/restored" - assert-equals-directory "$TEST_TMP/server/world" "$TEST_TMP/restored" - rm -rf "$TEST_TMP/restored" + check-backup-full-paths "$TEST_TMP/backups/$BACKUP_ARCHIVE" "$TEST_TMP/server/world" } # Tests @@ -60,6 +66,17 @@ test-backup-defaults () { check-backup "$TIMESTAMP.tar.gz" } +test-backup-spaces-in-directory () { + TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01")" + WORLD_SPACES="$TEST_TMP/minecraft server/the world" + mkdir -p "$(dirname "$WORLD_SPACES")" + BACKUP_SPACES="$TEST_TMP/My Backups" + mkdir -p "$BACKUP_SPACES" + cp -r "$TEST_TMP/server/world" "$WORLD_SPACES" + ./backup.sh -i "$WORLD_SPACES" -o "$BACKUP_SPACES" -s "$SCREEN_TMP" -f "$TIMESTAMP" + check-backup-full-paths "$BACKUP_SPACES/$TIMESTAMP.tar.gz" "$WORLD_SPACES" +} + test-backup-no-compression () { TIMESTAMP="$(date +%F_%H-%M-%S --date="2021-01-01")" ./backup.sh -a "" -e "" -i "$TEST_TMP/server/world" -o "$TEST_TMP/backups" -s "$SCREEN_TMP" -f "$TIMESTAMP" From 858ae2357cff5a027ea33a3f49b94870f05d8eee Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Thu, 4 Mar 2021 13:18:31 -0800 Subject: [PATCH 20/21] Add dependency info to README --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index fc5dcd3..729bf97 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,9 @@ wget https://raw.githubusercontent.com/nicolaschan/minecraft-backup/master/backu chmod +x backup.sh ``` +Make sure your system has `tar` and your chosen compression algorithm (`gzip` by default) installed. +If using RCON, you will also need to have the [`xxd`](https://linux.die.net/man/1/xxd) command. + ## Usage ```bash # If connecting with RCON: From 84df6e7561b3d5ba1a13ad4d169733b62b55f406 Mon Sep 17 00:00:00 2001 From: Nicolas Chan Date: Thu, 4 Mar 2021 13:24:08 -0800 Subject: [PATCH 21/21] Mention bug --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 729bf97..208d3e2 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,9 @@ To fix this problem, the backup script disables autosaving with the `save-off` M - Do not put trailing `/` in the `SERVER_DIRECTORY` or `BACKUP_DIRECTORY` - If "thin" delete method is behaving weirdly, try emptying your backup directory or switch to "sequential" +## Known Issues +There is a Minecraft bug [MC-217729](https://bugs.mojang.com/projects/MC/issues/MC-217729) in recent Minecraft server versions that cause them to automatically save the world even after receiving the `save-off` command. Until this is fixed, there is a chance that the backup will fail because the world files are modified by Minecraft in the process of creating the backup. This script will try to detect and report this problem if it does occur. + ## Disclaimer Backups are essential to the integrity of your Minecraft world. You should automate regular backups and **check that your backups work**. It is up to you to make sure that your backups work and that you have a reliable backup policy.