add PIP_BREAK_SYSTEM_PACKAGES env

* Use --no-tablespaces in mysqldump
* Dump to 0.7.0
* pin docker version
* Include missing packages in setup.py

.dockerignore

@@ -1,9 +0,0 @@
-.venv/
-.vscode/
-.gitignore
-Dockerfile
-tests/
-docker-compose.yaml
-*.env
-*.egg-info
-__pycache__

.gitignore

@@ -19,3 +19,12 @@ venv
 /private/
 restic_data/
 restic_cache/
+alerts.env
+
+# build
+build/
+docs/_build
+dist
+
+# tests
+.tox

.readthedocs.yaml

@@ -0,0 +1,24 @@
+# Read the Docs configuration file for Sphinx projects
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Optionally build your docs in additional formats such as PDF and ePub
+# formats:
+#   - pdf
+#   - epub
+
+python:
+  install:
+    - requirements: docs/requirements.txt

.travis.yml

@@ -0,0 +1,17 @@
+language: python
+
+sudo: false
+
+matrix:
+  include:
+      python: 3.7
+      dist: bionic
+      sudo: true
+
+install: 
+  - pip install -U setuptools pip wheel
+  - pip install -r src/tests/requirements.txt
+  - pip install ./src
+
+script:
+  - tox

Dockerfile

@@ -1,11 +0,0 @@
-FROM restic/restic:0.9.6
-
-RUN apk update && apk add python3 dcron mariadb-client postgresql-client
-
-ADD . /restic-compose-backup
-WORKDIR /restic-compose-backup
-RUN pip3 install -U pip setuptools
-RUN pip3 install -e .
-
-ENTRYPOINT []
-CMD ["./entrypoint.sh"]

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Zetta.IO
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,139 +1,200 @@
 
 # restic-compose-backup
 
-*WORK IN PROGRESS*
+![docs](https://readthedocs.org/projects/restic-compose-backup/badge/?version=latest)
 
-Backup using https://restic.net/ for a docker-compose setup.
+Backup using [restic] for a docker-compose setup.
+Currently tested with docker-ce 17, 18 and 19.
 
-Automatically detects and backs up volumes, mysql, mariadb and postgres databases in a docker-compose setup.
-This includes both host mapped volumes and actual docker volumes.
+* [restic-compose-backup Documentation](https://restic-compose-backup.readthedocs.io)
+* [restic-compose-backup on Github](https://github.com/ZettaIO/restic-compose-backup)
+* [restic-compose-backup on Docker Hub](https://hub.docker.com/r/zettaio/restic-compose-backup)
 
-* Each service in the compose setup is configured with a label
-  to enable backup of volumes or databases
-* When backup starts a new instance of the container is created
-  mapping in all the needed volumes. It will copy networks etc
-  to ensure databases can be reached
-* Volumes are mounted to `/backup/<service_name>/<path>`
-  in the backup process container. `/backup` is pushed into restic
-* Databases are backed up from stdin / dumps
-* Cron triggers backup
+Features:
 
-## Configuration
+* Backs up docker volumes or host binds
+* Backs up postgres, mariadb and mysql databases
+* Notifications over mail/smtp or Discord webhooks
 
-Required env variables for restic:
+Please report issus on [github](https://github.com/ZettaIO/restic-compose-backup/issues).
+
+## Install
+
+```bash
+docker pull zettaio/restic-compose-backup
+```
+
+## Configuration (env vars)
+
+Minimum configuration
 
 ```bash
 RESTIC_REPOSITORY
 RESTIC_PASSWORD
 ```
 
-Backend specific env vars : https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables
+More config options can be found in the [documentation].
 
-### Volumes
+Restic backend specific env vars : https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables
+
+## Compose Example
+
+We simply control what should be backed up by adding
+labels to our containers. More details are covered
+in the [documentation].
+
+restic-backup.env
+
+```bash
+RESTIC_REPOSITORY=<whatever backend restic supports>
+RESTIC_PASSWORD=hopefullyasecturepw
+# snapshot prune rules
+RESTIC_KEEP_DAILY=7
+RESTIC_KEEP_WEEKLY=4
+RESTIC_KEEP_MONTHLY=12
+RESTIC_KEEP_YEARLY=3
+# Cron schedule. Run every day at 1am
+CRON_SCHEDULE="0 1 * * *"
+```
+
+docker-compose.yaml
 
 ```yaml
 version: '3'
 services:
   # The backup service
   backup:
-    build: restic-compose-backup
-    environment:
-      - RESTIC_REPOSITORY=<whatever restic supports>
-      - RESTIC_PASSWORD=hopefullyasecturepw
+    image: zettaio/restic-compose-backup:<version>
     env_file:
-      - some_other_vars.env
+      - restic-backup.env
     volumes:
+      # We need to communicate with docker
       - /var/run/docker.sock:/tmp/docker.sock:ro
-
-  example:
+      # Persistent storage of restic cache (greatly speeds up all restic operations)
+      - cache:/cache
+  web:
     image: some_image
-    # Enable volume backup with label
     labels:
+      # Enables backup of the volumes below
       restic-compose-backup.volumes: true
-    # These volumes will be backed up
     volumes:
-      # Docker volume
       - media:/srv/media
-      # Host map
       - /srv/files:/srv/files
-
-volumes:
-  media:
-```
-
-A simple `include` and `exclude` filter is also available.
-
-```yaml
-  example:
-    image: some_image
-    labels:
-      restic-compose-backup.volumes: true
-      restic-compose-backup.volumes.include: "files,data"
-    volumes:
-      # Source don't match include filter. No backup.
-      - media:/srv/media
-      # Matches include filter
-      - files:/srv/files
-      - /srv/data:/srv/data
-
-volumes:
-  media:
-  files:
-
-```
-
-Exclude
-
-```yaml
-  example:
-    image: some_image
-    labels:
-      restic-compose-backup.volumes: true
-      restic-compose-backup.volumes.exclude: "media"
-    volumes:
-      # Excluded by filter
-      - media:/srv/media
-      # Backed up
-      - files:/srv/files
-      - /srv/data:/srv/data
-
-volumes:
-  media:
-  files:
-```
-
-### Databases
-
-Will dump databases directly into restic through stdin.
-They will appear in restic as a separate snapshot with
-path `/backup/<service_name>/dump.sql` or similar.
-
-```yaml
   mariadb:
     image: mariadb:10
     labels:
+      # Enables backup of this database
       restic-compose-backup.mariadb: true
-```
-
-```yaml
+    env_file:
+      mariadb-credentials.env
+    volumes:
+      - mysqldata:/var/lib/mysql
   mysql:
     image: mysql:5
     labels:
+      # Enables backup of this database
       restic-compose-backup.mysql: true
-```
+    env_file:
+      mysql-credentials.env
+    volumes:
+      - mysqldata:/var/lib/mysql
 
-```yaml
   postgres:
     image: postgres
     labels:
+      # Enables backup of this database
       restic-compose-backup.postgres: true
+    env_file:
+      postgres-credentials.env
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+
+volumes:
+  media:
+  mysqldata:
+  mariadbdata:
+  pgdata:
+  cache:
 ```
 
+## The `rcb` command
+
+Everything is controlled using the `rcb` command.
+After configuring backup with labels and restarted
+the affected services we can quickly view the
+result using the `status` subcommand.
+
+```bash
+$ docker-compose run --rm backup rcb status
+INFO: Status for compose project 'myproject'
+INFO: Repository: '<restic repository>'
+INFO: Backup currently running?: False
+INFO: --------------- Detected Config ---------------
+INFO: service: mysql
+INFO:  - mysql (is_ready=True)
+INFO: service: mariadb
+INFO:  - mariadb (is_ready=True)
+INFO: service: postgres
+INFO:  - postgres (is_ready=True)
+INFO: service: web
+INFO:  - volume: media
+INFO:  - volume: /srv/files
+```
+
+The `status` subcommand lists what will be backed up and
+even pings the database services checking their availability.
+The `restic` command can also be used directly in the container.
+
+More `rcb` commands can be found in the [documentation].
 
 ## Running Tests
 
+```bash
+pip install -e ./src/
+pip install -r src/tests/requirements.txt
+tox
 ```
-python setup.py develop
-pip install -r tests/requirements.txt
-pytest tests
+
+## Building Docs
+
+```bash
+pip install -r docs/requirements.txt
+python src/setup.py build_sphinx
 ```
+
+# Local dev setup
+
+The git repository contains a simple local setup for development
+
+```bash
+# Create an overlay network to link the compose project and stack
+docker network create --driver overlay --attachable global
+# Start the compose project
+docker-compose up -d
+# Deploy the stack
+docker stack deploy -c swarm-stack.yml test
+```
+
+In dev we should ideally start the backup container manually
+
+```bash
+docker-compose run --rm backup sh
+# pip install the package in the container in editable mode to auto sync changes from host source
+pip3 install -e .
+```
+
+Remember to enable swarm mode with `docker swarm init/join` and disable swarm
+mode with `docker swarm leave --force` when needed in development (single node setup).
+
+## Contributing
+
+Contributions are welcome regardless of experience level.
+Don't hesitate submitting issues, opening partial or completed pull requests.
+
+[restic]: https://restic.net/
+[documentation]: https://restic-compose-backup.readthedocs.io
+
+---
+This project is sponsored by [zetta.io](https://www.zetta.io)
+
+[![Zetta.IO](https://raw.githubusercontent.com/ZettaIO/restic-compose-backup/master/.github/logo.png)](https://www.zetta.io)

crontab

@@ -1 +0,0 @@
-1 * * * * source /env.sh && rcb backup > /proc/1/fd/1

docker-compose.yaml

@@ -1,30 +1,38 @@
-version: '3'
+version: '3.7'
 services:
   backup:
-    build: .
+    build: ./src
     env_file:
       - restic_compose_backup.env
+      # - alerts.env
+    labels:
+      restic-compose-backup.volumes: true
+      restic-compose-backup.volumes.include: 'src'
+    networks:
+      - default
+      - global
     volumes:
       # Map in docker socket
       - /var/run/docker.sock:/tmp/docker.sock:ro
-      # Map backup database locally
+      # Map local restic repository for dev
       - ./restic_data:/restic_data
-      - ./restic_cache:/restic_cache
-      # Map in project source
-      - .:/restic-compose-backup
+      # Map restic cache
+      - ./restic_cache:/cache
+      # Map in project source in dev
+      - ./src:/restic-compose-backup
   web:
     image: nginx
     labels:
       restic-compose-backup.volumes: true
       restic-compose-backup.volumes.include: "/tests"
     volumes:
-      - ./tests:/srv/tests
+      - ./src/tests:/srv/tests
       - ./.vscode:/srv/code
     environment:
       - SOME_VALUE=test
       - ANOTHER_VALUE=1
 
-  mysql:
+  mysql5:
     image: mysql:5
     labels:
       restic-compose-backup.mysql: true
@@ -34,7 +42,19 @@ services:
       - MYSQL_USER=myuser
       - MYSQL_PASSWORD=mypassword
     volumes:
-      - mysqldata:/var/lib/mysql
+      - mysqldata5:/var/lib/mysql
+
+  mysql8:
+    image: mysql:8
+    labels:
+      restic-compose-backup.mysql: true
+    environment:
+      - MYSQL_ROOT_PASSWORD=my-secret-pw
+      - MYSQL_DATABASE=mydb
+      - MYSQL_USER=myuser
+      - MYSQL_PASSWORD=mypassword
+    volumes:
+      - mysqldata8:/var/lib/mysql
 
   mariadb:
     image: mariadb:10
@@ -60,6 +80,11 @@ services:
       - pgdata:/var/lib/postgresql/data
 
 volumes:
-  mysqldata:
+  mysqldata5:
+  mysqldata8:
   mariadbdata:
   pgdata:
+
+networks:
+  global:
+    external: true

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/conf.py

@@ -0,0 +1,56 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'restic-compose-backup'
+copyright = '2019, Zetta.IO Technology AS'
+author = 'Zetta.IO Technology AS'
+
+# The full version, including alpha/beta/rc tags
+release = '0.6.0'
+
+# -- General configuration ---------------------------------------------------
+
+master_doc = 'index'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']

docs/guide/advanced.rst

@@ -0,0 +1,30 @@
+Advanced
+--------
+
+Currently work in progress. These are only notes :D
+
+Temp Notes
+~~~~~~~~~~
+
+* Quick setup guide from start to end
+* we group snapshots by path when forgetting
+* explain rcb commands
+* examples of using restic directly
+* Explain what happens during backup process
+* Explain the backup process container
+* cache directory
+* Not displaying passwords in logs
+
+Inner workings
+~~~~~~~~~~~~~~
+
+* Each service in the compose setup is configured with a label
+  to enable backup of volumes or databases
+* When backup starts a new instance of the container is created
+  mapping in all the needed volumes. It will copy networks etc
+  to ensure databases can be reached
+* Volumes are mounted to `/volumes/<service_name>/<path>`
+  in the backup process container. `/volumes` is pushed into restic
+* Databases are backed up from stdin / dumps into restic using path
+  `/databases/<service_name>/dump.sql`
+* Cron triggers backup at 2AM every day

docs/guide/configuration.rst

@@ -0,0 +1,447 @@
+Configuration
+=============
+
+Environment Variables
+---------------------
+
+RESTIC_REPOSITORY
+~~~~~~~~~~~~~~~~~
+
+Sets the restic repository path.
+
+This is a standard environment variable
+the ``restic`` command will read making it simple for
+us to enter the container and use the restic command directly.
+
+More about this value and supported backends:
+https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html
+
+RESTIC_PASSWORD
+~~~~~~~~~~~~~~~
+
+Sets the password is used to encrypt/decrypt data.
+Losing this password will make recovery impossible.
+
+This is a standard environment variable the ``restic``
+command will read making it simple for us to enter the
+container running the command directly.
+
+RESTIC_KEEP_DAILY
+~~~~~~~~~~~~~~~~~
+
+**Default value**: ``7``
+
+How many daily snapshots (grouped by path) back in time we
+want to keep. This is passed to restic in the
+``forget --keep-daily`` option.
+
+RESTIC_KEEP_WEEKLY
+~~~~~~~~~~~~~~~~~~
+
+**Default value**: ``4``
+
+How many weeks back we should keep at least one snapshot
+(grouped by path). This is passed to restic in the
+``forget --keep-weekly`` option.
+
+RESTIC_KEEP_MONTHLY
+~~~~~~~~~~~~~~~~~~~
+
+**Default value**: ``12``
+
+How many months back we should keep at least on snapshot
+(grouped by path). This is passed to restic in the
+``forget --keep-monthly`` option.
+
+The schedule parameters only accepts numeric values
+and is validated when the container starts. Providing
+values cron does not understand will stall all backup.
+
+RESTIC_KEEP_YEARLY
+~~~~~~~~~~~~~~~~~~
+
+**Default value**: ``3``
+
+How many years back we should keep at least one snapshot
+(grouped by path). This is passed to restic in the
+``forget --keep-yearly`` option.
+
+CRON_SCHEDULE
+~~~~~~~~~~~~~
+
+**Default value**: ``0 2 * * *`` (daily at 02:00)
+
+The cron schedule parameters. The crontab is generated when the
+container starts from the ``CRON_SCHEDULE`` and ``CRON_COMMAND``
+env variables.
+
+.. code::
+
+    ┌───────────── minute (0 - 59)
+    │ ┌───────────── hour (0 - 23)
+    │ │ ┌───────────── day of the month (1 - 31)
+    │ │ │ ┌───────────── month (1 - 12)
+    │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
+    │ │ │ │ │
+    │ │ │ │ │
+    │ │ │ │ │
+    * * * * * command to execute
+
+CRON_COMMAND
+~~~~~~~~~~~~
+
+**Default value**: ``source /env.sh && rcb backup > /proc/1/fd/1``
+
+The command executed in the crontab. A single line is generated when
+the container starts from the ``CRON_SCHEDULE`` and ``CRON_COMMAND``
+environment variables.
+
+The default command sources a dump of all env vars, runs the
+backup command and directs output to pid 1 so it appears in
+docker logs.
+
+By default the crontab will look like this::
+
+    0 2 * * * source /env.sh && rcb backup > /proc/1/fd/1
+
+LOG_LEVEL
+~~~~~~~~~
+
+**Default value**: ``info``
+
+Log level for the ``rcb`` command. Valid values are
+``debug``, ``info``, ``warning``, ``error``.
+
+EMAIL_HOST
+~~~~~~~~~~
+
+The email host to use.
+
+Alerts can be tested using the ``rcb alerts`` command.
+This will send a test message to all configured alert
+backends.
+
+EMAIL_PORT
+~~~~~~~~~~
+
+The port to connect to
+
+Alerts can be tested using the ``rcb alerts`` command.
+This will send a test message to all configured alert
+backends.
+
+EMAIL_HOST_USER
+~~~~~~~~~~~~~~~
+
+The user of the sender account
+
+Alerts can be tested using the ``rcb alerts`` command.
+This will send a test message to all configured alert
+backends.
+
+EMAIL_HOST_PASSWORD
+~~~~~~~~~~~~~~~~~~~
+
+The password for the sender account
+
+Alerts can be tested using the ``rcb alerts`` command.
+This will send a test message to all configured alert
+backends.
+
+EMAIL_SEND_TO
+~~~~~~~~~~~~~
+
+The email address to send alerts
+
+Alerts can be tested using the ``rcb alerts`` command.
+This will send a test message to all configured alert
+backends.
+
+DISCORD_WEBHOOK
+~~~~~~~~~~~~~~~
+
+The discord webhook url. And administrator can quickly set this up
+by going to server settings in the discord client and create
+a webhook that will post embedded messages to a specific channel.
+
+The url usually looks like this: ``https://discordapp.com/api/webhooks/...```
+
+DOCKER_HOST
+~~~~~~~~~~~
+
+**Default value**: ``unix://tmp/docker.sock``
+
+The socket or host of the docker service.
+
+DOCKER_TLS_VERIFY
+~~~~~~~~~~~~~~~~~
+
+If defined verify the host against a CA certificate.
+Path to certs is defined in ``DOCKER_CERT_PATH``
+and can be copied or mapped into this backup container.
+
+DOCKER_CERT_PATH
+~~~~~~~~~~~~~~~~
+
+A path to a directory containing TLS certificates to use when
+connecting to the Docker host. Combined with ``DOCKER_TLS_VERIFY``
+this can be used to talk to docker through TLS in cases
+were we cannot map in the docker socket.
+
+INCLUDE_PROJECT_NAME
+~~~~~~~~~~~~~~~~~~~~
+
+Define this environment variable if your backup destination
+paths needs project name as a prefix. This is useful
+when running multiple projects.
+
+EXCLUDE_BIND_MOUNTS
+~~~~~~~~~~~~~~~~~~~
+
+Docker has to volumes types. Binds and volumes.
+Volumes are docker volumes (``docker`volume list``).
+Binds are paths mapped into the container from
+the host for example in the ``volumes`` section
+of a service.
+
+If defined all host binds will be ignored globally.
+This is useful when you only care about actual
+docker volumes. Often host binds are only used
+for mapping in configuration. This saves the user
+from manually excluding these bind volumes.
+
+SWARM_MODE
+~~~~~~~~~~
+
+If defined containers in swarm stacks are also evaluated.
+
+Compose Labels
+--------------
+
+What is backed up is controlled by simple labels in the compose
+yaml file. At any point we can verify this configuration
+by running the ``rcb status`` command.
+
+.. code:
+
+    $ docker-compose run --rm backup rcb status
+    INFO: Status for compose project 'myproject'
+    INFO: Repository: '<restic repository>'
+    INFO: Backup currently running?: False
+    INFO: --------------- Detected Config ---------------
+    INFO: service: mysql
+    INFO:  - mysql (is_ready=True)
+    INFO: service: mariadb
+    INFO:  - mariadb (is_ready=True)
+    INFO: service: postgres
+    INFO:  - postgres (is_ready=True)
+    INFO: service: web
+    INFO:  - volume: media
+    INFO:  - volume: /srv/files
+
+Here we can see what volumes and databases are detected for backup.
+
+Volumes
+~~~~~~~
+
+To enable volume backup for a service we simply add the
+`restic-compose-backup.volumes: true` label. The value
+must be ``true``.
+
+Example:
+
+.. code:: yaml
+
+    myservice:
+      image: some_image
+      labels:
+        restic-compose-backup.volumes: true
+      volumes:
+        - uploaded_media:/srv/media
+        - uploaded_files:/srv/files
+        - /srv/data:/srv/data
+
+    volumes:
+      media:
+      files:
+
+This will back up the three volumes mounted to this service.
+Their path in restic will be:
+
+- /volumes/myservice/srv/media
+- /volumes/myservice/srv/files
+- /volumes/myservice/srv/data
+
+A simple `include` and `exclude` filter for what volumes
+should be backed up is also available. Note that this
+includes or excludes entire volumes and are not include/exclude
+patterns for files in the volumes.
+
+.. note:: The ``exclude`` and ``include`` filtering is applied on
+          the source path, not the destination.
+
+Include example including two volumes only:
+
+.. code:: yaml
+
+    myservice:
+      image: some_image
+      labels:
+        restic-compose-backup.volumes: true
+        restic-compose-backup.volumes.include: "uploaded_media,uploaded_files"
+      volumes:
+        - uploaded_media:/srv/media
+        - uploaded_files:/srv/files
+        - /srv/data:/srv/data
+
+    volumes:
+      media:
+      files:
+
+Exclude example achieving the same result as the example above.
+
+.. code:: yaml
+
+    example:
+      image: some_image
+      labels:
+        restic-compose-backup.volumes: true
+        restic-compose-backup.volumes.exclude: "data"
+      volumes:
+        # Excluded by filter
+        - media:/srv/media
+        # Backed up
+        - files:/srv/files
+        - /srv/data:/srv/data
+
+    volumes:
+      media:
+      files:
+
+The ``exclude`` and ``include`` tag can be used together
+in more complex situations.
+
+mariadb
+~~~~~~~
+
+To enable backup of mariadb simply add the
+``restic-compose-backup.mariadb: true`` label.
+
+Credentials are fetched from the following environment
+variables in the mariadb service. This is the standard
+when using the official mariadb_ image.
+
+.. code::
+
+    MYSQL_USER
+    MYSQL_PASSWORD
+
+Backups are done by dumping all databases directly into
+restic through stdin using ``mysqldump``. It will appear
+in restic as a separate snapshot with path
+``/databases/<service_name>/all_databases.sql``.
+
+.. warning: This will only back up the databases the
+            ``MYSQL_USER` has access to. If you have multiple
+            databases this must be taken into consideration.
+
+Example:
+
+.. code:: yaml
+
+    mariadb:
+      image: mariadb:10
+      labels:
+        restic-compose-backup.mariadb: true
+      env_file:
+        mariadb-credentials.env
+      volumes:
+        - mariadb:/var/lib/mysql
+
+    volumes:
+      mariadb:
+
+mysql
+~~~~~
+
+To enable backup of mysql simply add the
+``restic-compose-backup.mysql: true`` label.
+
+Credentials are fetched from the following environment
+variables in the mysql service. This is the standard
+when using the official mysql_ image.
+
+.. code::
+
+    MYSQL_USER
+    MYSQL_PASSWORD
+
+Backups are done by dumping all databases directly into
+restic through stdin using ``mysqldump``. It will appear
+in restic as a separate snapshot with path
+``/databases/<service_name>/all_databases.sql``.
+
+.. warning: This will only back up the databases the
+            ``MYSQL_USER` has access to. If you have multiple
+            databases this must be taken into consideration.
+
+Example:
+
+.. code:: yaml
+
+    mysql:
+      image: mysql:5
+      labels:
+        restic-compose-backup.mysql: true
+      env_file:
+        mysql-credentials.env
+      volumes:
+        - mysql:/var/lib/mysql
+
+volumes:
+  mysql:
+
+postgres
+~~~~~~~~
+
+To enable backup of mysql simply add the
+``restic-compose-backup.postgres: true`` label.
+
+Credentials are fetched from the following environment
+variables in the postgres service. This is the standard
+when using the official postgres_ image.
+
+.. code::
+
+    POSTGRES_USER
+    POSTGRES_PASSWORD
+    POSTGRES_DB
+
+Backups are done by dumping the ``POSTGRES_DB`` directly into
+restic through stdin using ``pg_dump``. It will appear
+in restic as a separate snapshot with path
+``/databases/<service_name>/<POSTGRES_DB>.sql``.
+
+.. warning:: Currently only the ``POSTGRES_DB`` database
+             is dumped.
+
+Example:
+
+.. code:: yaml
+
+    postgres:
+      image: postgres:11
+      labels:
+        # Enables backup of this database
+        restic-compose-backup.postgres: true
+      env_file:
+        postgres-credentials.env
+      volumes:
+        - pgdata:/var/lib/postgresql/data
+
+    volumes:
+      pgdata:
+
+.. _mariadb: https://hub.docker.com/_/mariadb
+.. _mysql: https://hub.docker.com/_/mysql
+.. _postgres: https://hub.docker.com/_/postgres

docs/guide/install.rst

@@ -0,0 +1,53 @@
+Introduction
+============
+
+
+Install
+-------
+
+restic-compose-backup is available at docker `docker hub`_.
+
+.. code::
+
+    docker pull restic-compose-backup
+
+Optionally it can be built from source using the github_ repository.
+
+.. code:: bash
+
+    git clone https://github.com/ZettaIO/restic-compose-backup.git
+    cd restic-compose-backup
+    # Build and tag the image locally
+    docker build src/ --tag restic-compose-backup
+
+Bug reports and issues
+----------------------
+
+Please report bugs an issues on github_
+
+Development setup
+-----------------
+
+Getting started with local development is fairly simple.
+The github_ repository contains a simple ``docker-compose.yaml``
+
+.. code:: bash
+
+    docker-compose up -d
+    # Enter the container in sh
+    docker-compose run --rm backup sh
+
+The dev compose setup maps in the source from the host
+and the spawned backup container will inherit all
+the volumes from the backup service ensuring code changes
+propagates during development.
+
+Set up a local venv and install the package in development mode::
+
+    python -m venv .venv
+    . .venv/bin/activate
+    pip install -e ./src
+
+
+.. _docker hub: https://hub.docker.com/r/zettaio/restic-compose-backup
+.. _github: https://github.com/ZettaIO/restic-compose-backup

docs/guide/rcb.rst

@@ -0,0 +1,199 @@
+
+The `rcb` command
+-----------------
+
+The ``rcb`` command is is basically what this entire project is.
+It provides useful commands interacting with the compose setup
+and restic.
+
+The command can be executed inside the container or through ``run``.
+
+.. code:: bash
+
+    # Get the current status using run
+    $ docker-compose run --rm backup rcb status
+
+    # by entering the container
+    $ docker-compose exec backup sh
+    /restic-compose-backup # rcb status
+
+Log level can be overridden by using the ``--log-level``
+flag. This can help you better understand what is going on
+for example by using ``--log-level debug``.
+
+version
+~~~~~~~
+
+Displays the version.
+
+Example output::
+
+    /restic-compose-backup # rcb version
+    0.4.0
+
+status
+~~~~~~
+
+Shows the general status of our setup. The command is doing
+the following operations
+
+- Displays the name of the compose setup
+- Displays the repository path
+- Tells us if a backup is currently running
+- Removes stale backup process containers if the exist
+- Checks is the repository is initialized
+- Initializes the repository if this is not already done
+- Displays what volumes and databases are flagged for backup
+
+Example output::
+
+    INFO: Status for compose project 'myproject'
+    INFO: Repository: '<restic repository>'
+    INFO: Backup currently running?: False
+    INFO: --------------- Detected Config ---------------
+    INFO: service: mysql
+    INFO:  - mysql (is_ready=True)
+    INFO: service: mariadb
+    INFO:  - mariadb (is_ready=True)
+    INFO: service: postgres
+    INFO:  - postgres (is_ready=True)
+    INFO: service: web
+    INFO:  - volume: media
+    INFO:  - volume: /srv/files
+
+alert
+~~~~~
+
+Sends a test message to all configured alert backends
+and is there for you to verify that alerts are in
+fact working and configured correctly.
+
+The format of this message::
+
+    subject: myproject: Test Alert
+    body: Test message
+
+snapshots
+~~~~~~~~~
+
+Displays the latest snapshots in restic. This can also
+be done with ``restic snapshots``.
+
+Example output::
+
+    /restic-compose-backup # rcb snapshots
+    repository f325264e opened successfully, password is correct
+    ID        Time                 Host          Tags        Paths
+    ---------------------------------------------------------------------------------------------
+    19928e1c  2019-12-09 02:07:44  b3038db04ec1              /volumes
+    7a642f37  2019-12-09 02:07:45  b3038db04ec1              /databases/mysql/all_databases.sql
+    883dada4  2019-12-09 02:07:46  b3038db04ec1              /databases/mariadb/all_databases.sql
+    76ef2457  2019-12-09 02:07:47  b3038db04ec1              /databases/postgres/test.sql
+    ---------------------------------------------------------------------------------------------
+    4 snapshots
+
+backup
+~~~~~~
+
+Starts a backup process by spawning a new docker container.
+The network stack, mounted volumes, env vars etc. from the
+backup service are copied to this container.
+
+We attach to this container and stream the logs and delete
+the container with the backup process is completed. If the
+container for any reason should not be deleted, it will
+be in next backup run as these containers are tagged with
+a unique label and detected.
+
+If anything goes wrong the exist status of the container
+is non-zero and the logs from this backup run will be sent
+to the user through the configure alerts.
+
+This command is by default called by cron every
+day at 02:00 unless configured otherwise. We can also run this
+manually is needed.
+
+Running this command will do the following:
+
+* Checks if a backup process is already running.
+  If so, we alert the user and abort
+* Gathers all the volumes configured for backup and starts
+  the backup process with these volumes mounted into ``/volumes``
+* Checks the status of the process and reports to the user
+  if anything failed
+
+The backup process does the following:
+
+* ``status`` is first called to ensure everything is ok
+* Backs up ``/volumes`` if any volumes were mounted
+* Backs up each configured database
+* Runs ``cleanup`` purging snapshots based on the configured policy
+* Checks the health of the repository
+
+Example::
+
+        $ docker-compose exec backup sh
+        /restic-compose-backup # rcb backup
+        INFO: Starting backup container
+        INFO: Backup process container: loving_jepsen
+        INFO: 2019-12-09 04:50:22,817 - INFO: Status for compose project 'restic-compose-backup'
+        INFO: 2019-12-09 04:50:22,817 - INFO: Repository: '/restic_data'
+        INFO: 2019-12-09 04:50:22,817 - INFO: Backup currently running?: True
+        INFO: 2019-12-09 04:50:23,701 - INFO: ------------------------- Detected Config -------------------------
+        INFO: 2019-12-09 04:50:23,701 - INFO: service: mysql
+        INFO: 2019-12-09 04:50:23,718 - INFO:  - mysql (is_ready=True)
+        INFO: 2019-12-09 04:50:23,718 - INFO: service: mariadb
+        INFO: 2019-12-09 04:50:23,726 - INFO:  - mariadb (is_ready=True)
+        INFO: 2019-12-09 04:50:23,727 - INFO: service: postgres
+        INFO: 2019-12-09 04:50:23,734 - INFO:  - postgres (is_ready=True)
+        INFO: 2019-12-09 04:50:23,735 - INFO: service: web
+        INFO: 2019-12-09 04:50:23,736 - INFO:  - volume: /some/volume
+        INFO: 2019-12-09 04:50:23,736 - INFO: -------------------------------------------------------------------
+        INFO: 2019-12-09 04:50:23,736 - INFO: Backing up volumes
+        INFO: 2019-12-09 04:50:24,661 - INFO: Backing up databases
+        INFO: 2019-12-09 04:50:24,661 - INFO: Backing up mysql in service mysql
+        INFO: 2019-12-09 04:50:25,643 - INFO: Backing up mariadb in service mariadb
+        INFO: 2019-12-09 04:50:26,580 - INFO: Backing up postgres in service postgres
+        INFO: 2019-12-09 04:50:27,555 - INFO: Forget outdated snapshots
+        INFO: 2019-12-09 04:50:28,457 - INFO: Prune stale data freeing storage space
+        INFO: 2019-12-09 04:50:31,547 - INFO: Checking the repository for errors
+        INFO: 2019-12-09 04:50:32,869 - INFO: Backup completed
+        INFO: Backup container exit code: 0
+
+crontab
+~~~~~~~
+
+Generates and verifies the crontab. This is done automatically when
+the container starts. It can be user to verify the configuration.
+
+Example output::
+
+    /restic-compose-backup # rcb crontab
+    10 2 * * * source /env.sh && rcb backup > /proc/1/fd/1
+
+cleanup
+~~~~~~~
+
+Purges all snapshots based on the configured policy. (``RESTIC_KEEP_*``
+env variables). It runs ``restic forget`` and ``restic purge``.
+
+Example output::
+
+    /restic-compose-backup # rcb cleanup
+    2019-12-09 05:09:52,892 - INFO: Forget outdated snapshots
+    2019-12-09 05:09:53,776 - INFO: Prune stale data freeing storage space
+
+start-backup-process
+~~~~~~~~~~~~~~~~~~~~
+
+This can only be executed by the backup process container.
+Attempting to run this command in the backup service
+will simply tell you it's not possible.
+
+The backup process is doing the following:
+
+* ``status`` is first called to ensure everything is ok
+* Backs up ``/volumes`` if any volumes were mounted
+* Backs up each configured database
+* Runs ``cleanup`` purging snapshots based on the configured policy
+* Checks the health of the repository

docs/index.rst

@@ -0,0 +1,27 @@
+.. restic-compose-backup documentation master file, created by
+   sphinx-quickstart on Thu Dec  5 01:34:58 2019.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to restic-compose-backup's documentation!
+=================================================
+
+Simple backup with restic_ for docker-compose setups.
+
+.. toctree::
+   :maxdepth: 3
+   :caption: Contents:
+
+   guide/install
+   guide/configuration
+   guide/rcb
+   guide/advanced
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
+.. _restic: https://restic.net/

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/requirements.txt

@@ -0,0 +1,2 @@
+sphinx
+sphinx-rtd-theme

extras/example-conpose-container.json

@@ -0,0 +1,196 @@
+{
+    "Id": "efa5196b4959648e3efcf5ae9f24bc4032849c2665805a5b405216f343b4decd",
+    "Created": "2020-03-05T21:07:34.88927951Z",
+    "Path": "docker-entrypoint.sh",
+    "Args": ["mysqld"],
+    "State": {
+        "Status": "running",
+        "Running": true,
+        "Paused": false,
+        "Restarting": false,
+        "OOMKilled": false,
+        "Dead": false,
+        "Pid": 4887,
+        "ExitCode": 0,
+        "Error": "",
+        "StartedAt": "2020-03-06T01:31:39.728842925Z",
+        "FinishedAt": "2020-03-06T01:31:33.847583199Z"
+    },
+    "Image": "sha256:1fd0e719c4952e22a99e30662fdd7daad53e7e53fbe135d543cc6b82be213951",
+    "ResolvConfPath": "/var/lib/docker/containers/efa5196b4959648e3efcf5ae9f24bc4032849c2665805a5b405216f343b4decd/resolv.conf",
+    "HostnamePath": "/var/lib/docker/containers/efa5196b4959648e3efcf5ae9f24bc4032849c2665805a5b405216f343b4decd/hostname",
+    "HostsPath": "/var/lib/docker/containers/efa5196b4959648e3efcf5ae9f24bc4032849c2665805a5b405216f343b4decd/hosts",
+    "LogPath": "/var/lib/docker/containers/efa5196b4959648e3efcf5ae9f24bc4032849c2665805a5b405216f343b4decd/efa5196b4959648e3efcf5ae9f24bc4032849c2665805a5b405216f343b4decd-json.log",
+    "Name": "/restic-compose-backup_mariadb_1",
+    "RestartCount": 0,
+    "Driver": "overlay2",
+    "Platform": "linux",
+    "MountLabel": "",
+    "ProcessLabel": "",
+    "AppArmorProfile": "",
+    "ExecIDs": null,
+    "HostConfig": {
+        "Binds": ["restic-compose-backup_mariadbdata:/var/lib/mysql:rw"],
+        "ContainerIDFile": "",
+        "LogConfig": {
+            "Type": "json-file",
+            "Config": {}
+        },
+        "NetworkMode": "restic-compose-backup_default",
+        "PortBindings": {},
+        "RestartPolicy": {
+            "Name": "",
+            "MaximumRetryCount": 0
+        },
+        "AutoRemove": false,
+        "VolumeDriver": "",
+        "VolumesFrom": [],
+        "CapAdd": null,
+        "CapDrop": null,
+        "Capabilities": null,
+        "Dns": null,
+        "DnsOptions": null,
+        "DnsSearch": null,
+        "ExtraHosts": null,
+        "GroupAdd": null,
+        "IpcMode": "shareable",
+        "Cgroup": "",
+        "Links": null,
+        "OomScoreAdj": 0,
+        "PidMode": "",
+        "Privileged": false,
+        "PublishAllPorts": false,
+        "ReadonlyRootfs": false,
+        "SecurityOpt": null,
+        "UTSMode": "",
+        "UsernsMode": "",
+        "ShmSize": 67108864,
+        "Runtime": "runc",
+        "ConsoleSize": [0, 0],
+        "Isolation": "",
+        "CpuShares": 0,
+        "Memory": 0,
+        "NanoCpus": 0,
+        "CgroupParent": "",
+        "BlkioWeight": 0,
+        "BlkioWeightDevice": null,
+        "BlkioDeviceReadBps": null,
+        "BlkioDeviceWriteBps": null,
+        "BlkioDeviceReadIOps": null,
+        "BlkioDeviceWriteIOps": null,
+        "CpuPeriod": 0,
+        "CpuQuota": 0,
+        "CpuRealtimePeriod": 0,
+        "CpuRealtimeRuntime": 0,
+        "CpusetCpus": "",
+        "CpusetMems": "",
+        "Devices": null,
+        "DeviceCgroupRules": null,
+        "DeviceRequests": null,
+        "KernelMemory": 0,
+        "KernelMemoryTCP": 0,
+        "MemoryReservation": 0,
+        "MemorySwap": 0,
+        "MemorySwappiness": null,
+        "OomKillDisable": false,
+        "PidsLimit": null,
+        "Ulimits": null,
+        "CpuCount": 0,
+        "CpuPercent": 0,
+        "IOMaximumIOps": 0,
+        "IOMaximumBandwidth": 0,
+        "MaskedPaths": ["/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware"],
+        "ReadonlyPaths": ["/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger"]
+    },
+    "GraphDriver": {
+        "Data": {
+            "LowerDir": "/var/lib/docker/overlay2/96e51e6162c0cb4385248375192ec777dd42b3ae7973e402de351f3932c502d0-init/diff:/var/lib/docker/overlay2/38780a41f93b7a20de03f1d76febb885f9213906fb30bad17cb3ad231fb7ce43/diff:/var/lib/docker/overlay2/a2abce521690b1baf6aa61e109a4659cb4272936871bc1afa73271eb8e453449/diff:/var/lib/docker/overlay2/a696286588d1d33b994b7f6e31c176c5f7e67c4f757d730323a7b6591d55f786/diff:/var/lib/docker/overlay2/c4bd8133c0d9547945d38a9998439082ce7b53df7e64737add5a5c824e6f67f2/diff:/var/lib/docker/overlay2/110e275ef21b8c9cc2cd0cce312fed5aabceb056460f637b958dfee56b7b3be8/diff:/var/lib/docker/overlay2/831c8a624e424f298766028e76a8ac08df0c5cf4564f63cae61330a8bce0cf63/diff:/var/lib/docker/overlay2/7ad8ae774951ec40c68b0993ef07ef3d70aa8aed44ea9f1e4d943ca5404cc717/diff:/var/lib/docker/overlay2/19bca9fb61ef1156f8a97313c126a6c06d7fe44a6c49e3affe16f50f2d5e56ff/diff:/var/lib/docker/overlay2/dcd4dda04d06b0a0c7e78517c6209fd67735b3027afda2c85a92de37ff7297d1/diff:/var/lib/docker/overlay2/babf41f5fe1f7b88c17cfce27214a4ad9473b0f8e0f118db948d2acddf4d4798/diff:/var/lib/docker/overlay2/b5f97865010acd5b04b4031d6223cd0b34fab89267891d61256ea16936be52f8/diff:/var/lib/docker/overlay2/6aba0159141ebb6d6783181d154c65046447b7d2bebce65d44c4939ba7943cca/diff:/var/lib/docker/overlay2/c71c34fe0e7e95409a9fc18698f0aee505940fd96aa3718836e2d89f3cfb2d49/diff:/var/lib/docker/overlay2/3be993436e2a6764a6c3c57a2e948f7a57e45ed0ec26cdd3366f4c1106c69869/diff",
+            "MergedDir": "/var/lib/docker/overlay2/96e51e6162c0cb4385248375192ec777dd42b3ae7973e402de351f3932c502d0/merged",
+            "UpperDir": "/var/lib/docker/overlay2/96e51e6162c0cb4385248375192ec777dd42b3ae7973e402de351f3932c502d0/diff",
+            "WorkDir": "/var/lib/docker/overlay2/96e51e6162c0cb4385248375192ec777dd42b3ae7973e402de351f3932c502d0/work"
+        },
+        "Name": "overlay2"
+    },
+    "Mounts": [{
+        "Type": "volume",
+        "Name": "restic-compose-backup_mariadbdata",
+        "Source": "/var/lib/docker/volumes/restic-compose-backup_mariadbdata/_data",
+        "Destination": "/var/lib/mysql",
+        "Driver": "local",
+        "Mode": "rw",
+        "RW": true,
+        "Propagation": ""
+    }],
+    "Config": {
+        "Hostname": "efa5196b4959",
+        "Domainname": "",
+        "User": "",
+        "AttachStdin": false,
+        "AttachStdout": false,
+        "AttachStderr": false,
+        "ExposedPorts": {
+            "3306/tcp": {}
+        },
+        "Tty": false,
+        "OpenStdin": false,
+        "StdinOnce": false,
+        "Env": ["MYSQL_ROOT_PASSWORD=my-secret-pw", "MYSQL_DATABASE=mydb", "MYSQL_USER=myuser", "MYSQL_PASSWORD=mypassword", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GOSU_VERSION=1.10", "GPG_KEYS=177F4010FE56CA3336300305F1656F24C74CD1D8", "MARIADB_MAJOR=10.4", "MARIADB_VERSION=1:10.4.12+maria~bionic"],
+        "Cmd": ["mysqld"],
+        "Image": "mariadb:10",
+        "Volumes": {
+            "/var/lib/mysql": {}
+        },
+        "WorkingDir": "",
+        "Entrypoint": ["docker-entrypoint.sh"],
+        "OnBuild": null,
+        "Labels": {
+            "com.docker.compose.config-hash": "c6ecde85ad111d324a4c97cde3a03898074b026c68ecffc0f7020e5eca9a71d7",
+            "com.docker.compose.container-number": "1",
+            "com.docker.compose.oneoff": "False",
+            "com.docker.compose.project": "restic-compose-backup",
+            "com.docker.compose.project.config_files": "docker-compose.yaml",
+            "com.docker.compose.project.working_dir": "C:\\Users\\efors\\projects\\zetta.io\\projects\\restic-compose-backup",
+            "com.docker.compose.service": "mariadb",
+            "com.docker.compose.version": "1.25.4",
+            "restic-compose-backup.mariadb": "True"
+        }
+    },
+    "NetworkSettings": {
+        "Bridge": "",
+        "SandboxID": "d462bb5dfdd26aba12b8a395ac90262ab00d65408bf60dfa1ade0ab6a1851c70",
+        "HairpinMode": false,
+        "LinkLocalIPv6Address": "",
+        "LinkLocalIPv6PrefixLen": 0,
+        "Ports": {
+            "3306/tcp": null
+        },
+        "SandboxKey": "/var/run/docker/netns/d462bb5dfdd2",
+        "SecondaryIPAddresses": null,
+        "SecondaryIPv6Addresses": null,
+        "EndpointID": "",
+        "Gateway": "",
+        "GlobalIPv6Address": "",
+        "GlobalIPv6PrefixLen": 0,
+        "IPAddress": "",
+        "IPPrefixLen": 0,
+        "IPv6Gateway": "",
+        "MacAddress": "",
+        "Networks": {
+            "restic-compose-backup_default": {
+                "IPAMConfig": null,
+                "Links": null,
+                "Aliases": ["efa5196b4959", "mariadb"],
+                "NetworkID": "8f3349b0debec88f9f48fff02d84cda3feae0a0e8c516e8b42e5777bb03db1cb",
+                "EndpointID": "0b75d3f00aa077fe95156bc80463d33fb21d241a287b33c06769047855c38400",
+                "Gateway": "172.19.0.1",
+                "IPAddress": "172.19.0.3",
+                "IPPrefixLen": 16,
+                "IPv6Gateway": "",
+                "GlobalIPv6Address": "",
+                "GlobalIPv6PrefixLen": 0,
+                "MacAddress": "02:42:ac:13:00:03",
+                "DriverOpts": null
+            }
+        }
+    }
+}

extras/example-stack-container.json

@@ -0,0 +1,207 @@
+{
+    "Id": "56c57903b6da3afd331312b244ddd0324f5b21cbbe5fc30072edf24781d80f76",
+    "Created": "2020-03-06T22:36:17.266061631Z",
+    "Path": "docker-entrypoint.sh",
+    "Args": ["mysqld"],
+    "State": {
+        "Status": "running",
+        "Running": true,
+        "Paused": false,
+        "Restarting": false,
+        "OOMKilled": false,
+        "Dead": false,
+        "Pid": 35967,
+        "ExitCode": 0,
+        "Error": "",
+        "StartedAt": "2020-03-06T22:36:17.636265528Z",
+        "FinishedAt": "0001-01-01T00:00:00Z"
+    },
+    "Image": "sha256:1fd0e719c4952e22a99e30662fdd7daad53e7e53fbe135d543cc6b82be213951",
+    "ResolvConfPath": "/var/lib/docker/containers/56c57903b6da3afd331312b244ddd0324f5b21cbbe5fc30072edf24781d80f76/resolv.conf",
+    "HostnamePath": "/var/lib/docker/containers/56c57903b6da3afd331312b244ddd0324f5b21cbbe5fc30072edf24781d80f76/hostname",
+    "HostsPath": "/var/lib/docker/containers/56c57903b6da3afd331312b244ddd0324f5b21cbbe5fc30072edf24781d80f76/hosts",
+    "LogPath": "/var/lib/docker/containers/56c57903b6da3afd331312b244ddd0324f5b21cbbe5fc30072edf24781d80f76/56c57903b6da3afd331312b244ddd0324f5b21cbbe5fc30072edf24781d80f76-json.log",
+    "Name": "/test_mariadb.1.q4uji32qvw4tkuvwx3pbnbgqq",
+    "RestartCount": 0,
+    "Driver": "overlay2",
+    "Platform": "linux",
+    "MountLabel": "",
+    "ProcessLabel": "",
+    "AppArmorProfile": "",
+    "ExecIDs": null,
+    "HostConfig": {
+        "Binds": null,
+        "ContainerIDFile": "",
+        "LogConfig": {
+            "Type": "json-file",
+            "Config": {}
+        },
+        "NetworkMode": "default",
+        "PortBindings": {},
+        "RestartPolicy": {
+            "Name": "",
+            "MaximumRetryCount": 0
+        },
+        "AutoRemove": false,
+        "VolumeDriver": "",
+        "VolumesFrom": null,
+        "CapAdd": null,
+        "CapDrop": null,
+        "Capabilities": null,
+        "Dns": null,
+        "DnsOptions": null,
+        "DnsSearch": null,
+        "ExtraHosts": null,
+        "GroupAdd": null,
+        "IpcMode": "private",
+        "Cgroup": "",
+        "Links": null,
+        "OomScoreAdj": 0,
+        "PidMode": "",
+        "Privileged": false,
+        "PublishAllPorts": false,
+        "ReadonlyRootfs": false,
+        "SecurityOpt": null,
+        "UTSMode": "",
+        "UsernsMode": "",
+        "ShmSize": 67108864,
+        "Runtime": "runc",
+        "ConsoleSize": [0, 0],
+        "Isolation": "default",
+        "CpuShares": 0,
+        "Memory": 0,
+        "NanoCpus": 0,
+        "CgroupParent": "",
+        "BlkioWeight": 0,
+        "BlkioWeightDevice": null,
+        "BlkioDeviceReadBps": null,
+        "BlkioDeviceWriteBps": null,
+        "BlkioDeviceReadIOps": null,
+        "BlkioDeviceWriteIOps": null,
+        "CpuPeriod": 0,
+        "CpuQuota": 0,
+        "CpuRealtimePeriod": 0,
+        "CpuRealtimeRuntime": 0,
+        "CpusetCpus": "",
+        "CpusetMems": "",
+        "Devices": null,
+        "DeviceCgroupRules": null,
+        "DeviceRequests": null,
+        "KernelMemory": 0,
+        "KernelMemoryTCP": 0,
+        "MemoryReservation": 0,
+        "MemorySwap": 0,
+        "MemorySwappiness": null,
+        "OomKillDisable": false,
+        "PidsLimit": null,
+        "Ulimits": null,
+        "CpuCount": 0,
+        "CpuPercent": 0,
+        "IOMaximumIOps": 0,
+        "IOMaximumBandwidth": 0,
+        "Mounts": [{
+            "Type": "volume",
+            "Source": "test_mariadbdata",
+            "Target": "/var/lib/mysql",
+            "VolumeOptions": {
+                "Labels": {
+                    "com.docker.stack.namespace": "test"
+                }
+            }
+        }],
+        "MaskedPaths": ["/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware"],
+        "ReadonlyPaths": ["/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger"]
+    },
+    "GraphDriver": {
+        "Data": {
+            "LowerDir": "/var/lib/docker/overlay2/ba8a39bdb1d2e25d373b6b00c764be3d37353e57cf03981c4c3e5a20ae6a602b-init/diff:/var/lib/docker/overlay2/38780a41f93b7a20de03f1d76febb885f9213906fb30bad17cb3ad231fb7ce43/diff:/var/lib/docker/overlay2/a2abce521690b1baf6aa61e109a4659cb4272936871bc1afa73271eb8e453449/diff:/var/lib/docker/overlay2/a696286588d1d33b994b7f6e31c176c5f7e67c4f757d730323a7b6591d55f786/diff:/var/lib/docker/overlay2/c4bd8133c0d9547945d38a9998439082ce7b53df7e64737add5a5c824e6f67f2/diff:/var/lib/docker/overlay2/110e275ef21b8c9cc2cd0cce312fed5aabceb056460f637b958dfee56b7b3be8/diff:/var/lib/docker/overlay2/831c8a624e424f298766028e76a8ac08df0c5cf4564f63cae61330a8bce0cf63/diff:/var/lib/docker/overlay2/7ad8ae774951ec40c68b0993ef07ef3d70aa8aed44ea9f1e4d943ca5404cc717/diff:/var/lib/docker/overlay2/19bca9fb61ef1156f8a97313c126a6c06d7fe44a6c49e3affe16f50f2d5e56ff/diff:/var/lib/docker/overlay2/dcd4dda04d06b0a0c7e78517c6209fd67735b3027afda2c85a92de37ff7297d1/diff:/var/lib/docker/overlay2/babf41f5fe1f7b88c17cfce27214a4ad9473b0f8e0f118db948d2acddf4d4798/diff:/var/lib/docker/overlay2/b5f97865010acd5b04b4031d6223cd0b34fab89267891d61256ea16936be52f8/diff:/var/lib/docker/overlay2/6aba0159141ebb6d6783181d154c65046447b7d2bebce65d44c4939ba7943cca/diff:/var/lib/docker/overlay2/c71c34fe0e7e95409a9fc18698f0aee505940fd96aa3718836e2d89f3cfb2d49/diff:/var/lib/docker/overlay2/3be993436e2a6764a6c3c57a2e948f7a57e45ed0ec26cdd3366f4c1106c69869/diff",
+            "MergedDir": "/var/lib/docker/overlay2/ba8a39bdb1d2e25d373b6b00c764be3d37353e57cf03981c4c3e5a20ae6a602b/merged",
+            "UpperDir": "/var/lib/docker/overlay2/ba8a39bdb1d2e25d373b6b00c764be3d37353e57cf03981c4c3e5a20ae6a602b/diff",
+            "WorkDir": "/var/lib/docker/overlay2/ba8a39bdb1d2e25d373b6b00c764be3d37353e57cf03981c4c3e5a20ae6a602b/work"
+        },
+        "Name": "overlay2"
+    },
+    "Mounts": [{
+        "Type": "volume",
+        "Name": "test_mariadbdata",
+        "Source": "/var/lib/docker/volumes/test_mariadbdata/_data",
+        "Destination": "/var/lib/mysql",
+        "Driver": "local",
+        "Mode": "z",
+        "RW": true,
+        "Propagation": ""
+    }],
+    "Config": {
+        "Hostname": "56c57903b6da",
+        "Domainname": "",
+        "User": "",
+        "AttachStdin": false,
+        "AttachStdout": false,
+        "AttachStderr": false,
+        "ExposedPorts": {
+            "3306/tcp": {}
+        },
+        "Tty": false,
+        "OpenStdin": false,
+        "StdinOnce": false,
+        "Env": ["MYSQL_DATABASE=mydb", "MYSQL_PASSWORD=mypassword", "MYSQL_ROOT_PASSWORD=my-secret-pw", "MYSQL_USER=myuser", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "GOSU_VERSION=1.10", "GPG_KEYS=177F4010FE56CA3336300305F1656F24C74CD1D8", "MARIADB_MAJOR=10.4", "MARIADB_VERSION=1:10.4.12+maria~bionic"],
+        "Cmd": ["mysqld"],
+        "Image": "mariadb:10@sha256:d1ceee944c90ee3b596266de1b0ac25d2f34adbe9c35156b75bcb9a7047c7545",
+        "Volumes": {
+            "/var/lib/mysql": {}
+        },
+        "WorkingDir": "",
+        "Entrypoint": ["docker-entrypoint.sh"],
+        "OnBuild": null,
+        "Labels": {
+            "com.docker.stack.namespace": "test",
+            "com.docker.swarm.node.id": "gj73oe0vgmldlv2pdcj243231",
+            "com.docker.swarm.service.id": "jewh88xvythjkga24wy1thxc2",
+            "com.docker.swarm.service.name": "test_mariadb",
+            "com.docker.swarm.task": "",
+            "com.docker.swarm.task.id": "q4uji32qvw4tkuvwx3pbnbgqq",
+            "com.docker.swarm.task.name": "test_mariadb.1.q4uji32qvw4tkuvwx3pbnbgqq",
+            "restic-compose-backup.mariadb": "true"
+        }
+    },
+    "NetworkSettings": {
+        "Bridge": "",
+        "SandboxID": "5aa81b0859dfd6f6be629eb966ce365f22dc86620359cce3e3d25d5291b539db",
+        "HairpinMode": false,
+        "LinkLocalIPv6Address": "",
+        "LinkLocalIPv6PrefixLen": 0,
+        "Ports": {
+            "3306/tcp": null
+        },
+        "SandboxKey": "/var/run/docker/netns/5aa81b0859df",
+        "SecondaryIPAddresses": null,
+        "SecondaryIPv6Addresses": null,
+        "EndpointID": "",
+        "Gateway": "",
+        "GlobalIPv6Address": "",
+        "GlobalIPv6PrefixLen": 0,
+        "IPAddress": "",
+        "IPPrefixLen": 0,
+        "IPv6Gateway": "",
+        "MacAddress": "",
+        "Networks": {
+            "test_default": {
+                "IPAMConfig": {
+                    "IPv4Address": "10.0.1.3"
+                },
+                "Links": null,
+                "Aliases": ["56c57903b6da"],
+                "NetworkID": "8aweh54u31eq3i47vqdr2aonc",
+                "EndpointID": "5369b4c82a479a3e9dfb3547cb7ac3a0fab888e38ad5c1d0ad02b0e9a9523a64",
+                "Gateway": "",
+                "IPAddress": "10.0.1.3",
+                "IPPrefixLen": 24,
+                "IPv6Gateway": "",
+                "GlobalIPv6Address": "",
+                "GlobalIPv6PrefixLen": 0,
+                "MacAddress": "02:42:0a:00:01:03",
+                "DriverOpts": null
+            }
+        }
+    }
+}

extras/example_swarm_nodes.json

@@ -0,0 +1,93 @@
+{
+    "ID": "k5427pk4t7ss4d7ylacumeavz",
+    "Version": {
+        "Index": 30
+    },
+    "CreatedAt": "2020-03-08T17:25:59.451947759Z",
+    "UpdatedAt": "2020-03-08T17:26:38.552002711Z",
+    "Spec": {
+        "Labels": {},
+        "Role": "manager",
+        "Availability": "active"
+    },
+    "Description": {
+        "Hostname": "docker-desktop",
+        "Platform": {
+            "Architecture": "x86_64",
+            "OS": "linux"
+        },
+        "Resources": {
+            "NanoCPUs": 4000000000,
+            "MemoryBytes": 2085535744
+        },
+        "Engine": {
+            "EngineVersion": "19.03.5",
+            "Plugins": [{
+                "Type": "Log",
+                "Name": "awslogs"
+            }, {
+                "Type": "Log",
+                "Name": "fluentd"
+            }, {
+                "Type": "Log",
+                "Name": "gcplogs"
+            }, {
+                "Type": "Log",
+                "Name": "gelf"
+            }, {
+                "Type": "Log",
+                "Name": "journald"
+            }, {
+                "Type": "Log",
+                "Name": "json-file"
+            }, {
+                "Type": "Log",
+                "Name": "local"
+            }, {
+                "Type": "Log",
+                "Name": "logentries"
+            }, {
+                "Type": "Log",
+                "Name": "splunk"
+            }, {
+                "Type": "Log",
+                "Name": "syslog"
+            }, {
+                "Type": "Network",
+                "Name": "bridge"
+            }, {
+                "Type": "Network",
+                "Name": "host"
+            }, {
+                "Type": "Network",
+                "Name": "ipvlan"
+            }, {
+                "Type": "Network",
+                "Name": "macvlan"
+            }, {
+                "Type": "Network",
+                "Name": "null"
+            }, {
+                "Type": "Network",
+                "Name": "overlay"
+            }, {
+                "Type": "Volume",
+                "Name": "local"
+            }]
+        },
+        "TLSInfo": {
+            "TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBazCCARCgAwIBAgIUfx7TP8c4SHCrwPPxjSFJQcfTP5QwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMjAwMzA4MTcyMTAwWhcNNDAwMzAzMTcy\nMTAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABGOa/9Rdd6qNc24wvuL/I9t5Vt3MJzlwC+WN0R6HrA4Ik1h2dmSRZTQqnCI7\nWh16y+PLaFwIfN0JkN4FrpnUBsyjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBTAPDjHRwjQhNGUxqE3COHCOQrOkjAKBggqhkjO\nPQQDAgNJADBGAiEAxd/lPEKy3gt3nfZ8DX7kDaaNH8jSPgCBx3ejUs3SoaUCIQD3\nZ8dVxNvG4+Gvn28mDjWhTNLCn0BYW6JFE8eTI0xv4A==\n-----END CERTIFICATE-----\n",
+            "CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
+            "CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEY5r/1F13qo1zbjC+4v8j23lW3cwnOXAL5Y3RHoesDgiTWHZ2ZJFlNCqcIjtaHXrL48toXAh83QmQ3gWumdQGzA=="
+        }
+    },
+    "Status": {
+        "State": "ready",
+        "Addr": "192.168.65.3"
+    },
+    "ManagerStatus": {
+        "Leader": true,
+        "Reachability": "reachable",
+        "Addr": "192.168.65.3:2377"
+    }
+}

extras/release.md

@@ -0,0 +1,21 @@
+# Making a release
+
+- Update version in `setup.py`
+- Update version in `docs/conf.py`
+- Update version in `restic_compose_backup/__init__.py`
+- Build and tag image
+- push: `docker push zettaio/restic-compose-backup:<version>`
+- Ensure RTD has new docs published
+
+## Example
+
+When releasing a bugfix version we need to update the
+main image as well.
+
+```bash
+docker build src --tag zettaio/restic-compose-backup:0.6
+docker build src --tag zettaio/restic-compose-backup:0.6.0
+
+docker push zettaio/restic-compose-backup:0.5
+docker push zettaio/restic-compose-backup:0.5.0
+```

pytest.ini

@@ -1,4 +1,4 @@
 [pytest]
-testpaths = tests
+testpaths = src/tests
 python_files=test*.py
 addopts = -v --verbose

restic_compose_backup.env

@@ -1,9 +1,32 @@
 # DON'T COMMIT THIS FILE IF YOU MODIFY IN DEV
 
-DOCKER_BASE_URL=unix://tmp/docker.sock
+# DOCKER_HOST=unix://tmp/docker.sock
+# DOCKER_TLS_VERIFY=1
+# DOCKER_CERT_PATH=''
+
+SWARM_MODE=true
+INCLUDE_PROJECT_NAME=false
+EXCLUDE_BIND_MOUNTS=false
+
 RESTIC_REPOSITORY=/restic_data
 RESTIC_PASSWORD=password
 
+RESTIC_KEEP_DAILY=7
+RESTIC_KEEP_WEEKLY=4
+RESTIC_KEEP_MONTHLY=12
+RESTIC_KEEP_YEARLY=3
+
+LOG_LEVEL=info
+CRON_SCHEDULE=10 2 * * *
+
+# EMAIL_HOST=
+# EMAIL_PORT=
+# EMAIL_HOST_USER=
+# EMAIL_HOST_PASSWORD=
+# EMAIL_SEND_TO=
+
+# DISCORD_WEBHOOK=u
+
 # Various env vars for restic : https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables
 # RESTIC_REPOSITORY                   Location of repository (replaces -r)
 # RESTIC_PASSWORD_FILE                Location of password file (replaces --password-file)

restic_compose_backup/cli.py

@@ -1,139 +0,0 @@
-import argparse
-import pprint
-import logging
-
-from restic_compose_backup import (
-    backup_runner,
-    log,
-    restic,
-)
-from restic_compose_backup.config import Config
-from restic_compose_backup.containers import RunningContainers
-
-logger = logging.getLogger(__name__)
-
-
-def main():
-    """CLI entrypoint"""
-    args = parse_args()
-    config = Config()
-    containers = RunningContainers()
-
-    if args.action == 'status':
-        status(config, containers)
-
-    elif args.action == 'backup':
-        backup(config, containers)
-
-    elif args.action == 'start-backup-process':
-        start_backup_process(config, containers)
-
-
-def status(config, containers):
-    """Outputs the backup config for the compose setup"""
-
-    logger.info("Backup config for compose project '%s'", containers.this_container.project_name)
-    logger.info("Current service: %s", containers.this_container.name)
-    # logger.info("Backup process: %s", containers.backup_process_container.name
-    #             if containers.backup_process_container else 'Not Running')
-    logger.info("Backup running: %s", containers.backup_process_running)
-
-    backup_containers = containers.containers_for_backup()
-    for container in backup_containers:
-        logger.info('service: %s', container.service_name)
-
-        if container.volume_backup_enabled:
-            for mount in container.filter_mounts():
-                logger.info(' - volume: %s', mount.source)
-
-        if container.database_backup_enabled:
-            instance = container.instance
-            ping = instance.ping()
-            logger.info(' - %s (is_ready=%s)', instance.container_type, ping == 0)
-
-    if len(backup_containers) == 0:
-        logger.info("No containers in the project has 'restic-compose-backup.enabled' label")
-
-
-def backup(config, containers):
-    """Request a backup to start"""
-    # Make sure we don't spawn multiple backup processes
-    if containers.backup_process_running:
-        raise ValueError("Backup process already running")
-
-    logger.info("Initializing repository")
-
-    # TODO: Errors when repo already exists
-    restic.init_repo(config.repository)
-
-    logger.info("Starting backup container..")
-
-    # Map all volumes from the backup container into the backup process container
-    volumes = containers.this_container.volumes
-
-    # Map volumes from other containers we are backing up
-    mounts = containers.generate_backup_mounts('/backup')
-    volumes.update(mounts)
-
-    result = backup_runner.run(
-        image=containers.this_container.image,
-        command='restic-compose-backup start-backup-process',
-        volumes=volumes,
-        environment=containers.this_container.environment,
-        source_container_id=containers.this_container.id,
-        labels={
-            "restic-compose-backup.backup_process": 'True',
-            "com.docker.compose.project": containers.this_container.project_name,
-        },
-    )
-    logger.info('Backup container exit code: %s', result)
-    # TODO: Alert
-
-
-def start_backup_process(config, containers):
-    """The actual backup process running inside the spawned container"""
-    if (not containers.backup_process_container
-       or containers.this_container == containers.backup_process_container is False):
-        logger.error(
-            "Cannot run backup process in this container. Use backup command instead. "
-            "This will spawn a new container with the necessary mounts."
-        )
-        return
-
-    status(config, containers)
-    logger.info("start-backup-process")
-
-    # Back up volumes
-    try:
-        vol_result = restic.backup_files(config.repository, source='/backup')
-        logger.info('Volume backup exit code: %s', vol_result)
-        # TODO: Alert
-    except Exception as ex:
-        logger.error(ex)
-        # TODO: Alert
-
-    # back up databases
-    for container in containers.containers_for_backup():
-        if container.database_backup_enabled:
-            try:
-                instance = container.instance
-                logger.info('Backing up %s in service %s', instance.container_type, instance.service_name)
-                result = instance.backup()
-                logger.info('Exit code: %s', result)
-                # TODO: Alert
-            except Exception as ex:
-                logger.error(ex)
-                # TODO: Alert
-
-
-def parse_args():
-    parser = argparse.ArgumentParser(prog='restic_compose_backup')
-    parser.add_argument(
-        'action',
-        choices=['status', 'backup', 'start-backup-process'],
-    )
-    return parser.parse_args()
-
-
-if __name__ == '__main__':
-    main()

restic_compose_backup/commands.py

@@ -1,68 +0,0 @@
-import logging
-from typing import List
-from subprocess import Popen, PIPE
-
-logger = logging.getLogger(__name__)
-
-
-def test():
-    return run_command(['ls', '/backup'])
-
-
-def ping_mysql(host, port, username, password) -> int:
-    """Check if the mysql is up and can be reached"""
-    return run([
-        'mysqladmin',
-        'ping',
-        '--host',
-        host,
-        '--port',
-        port,
-        '--user',
-        username,
-        f'--password={password}',
-    ])
-
-
-def ping_mariadb(host, port, username, password) -> int:
-    """Check if the mariadb is up and can be reached"""
-    return run([
-        'mysqladmin',
-        'ping',
-        '--host',
-        host,
-        '--port',
-        port,
-        '--user',
-        username,
-        f'--password={password}',
-    ])
-
-
-def ping_postgres(host, port, username, password) -> int:
-    """Check if postgres can be reached"""
-    return run([
-        "pg_isready",
-        f"--host={host}",
-        f"--port={port}",
-        f"--username={username}", 
-    ])
-
-
-def run(cmd: List[str]) -> int:
-    """Run a command with parameters"""
-    logger.info('cmd: %s', ' '.join(cmd))
-    child = Popen(cmd, stdout=PIPE, stderr=PIPE)
-    stdoutdata, stderrdata = child.communicate()
-
-    if stdoutdata:
-        logger.info(stdoutdata.decode().strip())
-        logger.info('-' * 28)
-
-    if stderrdata:
-        logger.info('%s STDERR %s', '-' * 10, '-' * 10)
-        logger.info(stderrdata.decode().strip())
-        logger.info('-' * 28)
-
-    logger.info("returncode %s", child.returncode)
-    return child.returncode

restic_compose_backup/config.py

@@ -1,19 +0,0 @@
-import os
-
-
-class Config:
-    """Bag for config values"""
-    def __init__(self, check=True):
-        self.repository = os.environ['RESTIC_REPOSITORY']
-        self.password = os.environ['RESTIC_PASSWORD']
-        self.docker_base_url = os.environ.get('DOCKER_BASE_URL') or "unix://tmp/docker.sock"
-
-        if check:
-            self.check()
-
-    def check(self):
-        if not self.repository:
-            raise ValueError("CONTAINER env var not set")
-
-        if not self.password:
-            raise ValueError("PASSWORD env var not set")

restic_compose_backup/log.py

@@ -1,13 +0,0 @@
-import logging
-import os
-import sys
-
-logger = logging.getLogger('restic_compose_backup')
-HOSTNAME = os.environ['HOSTNAME']
-
-level = logging.INFO
-logger.setLevel(level)
-ch = logging.StreamHandler(stream=sys.stdout)
-ch.setLevel(level)
-ch.setFormatter(logging.Formatter(f'%(asctime)s - {HOSTNAME} - %(name)s - %(levelname)s - %(message)s'))
-logger.addHandler(ch)

restic_compose_backup/mail.py

@@ -1,37 +0,0 @@
-"""
-"""
-import smtplib
-from email.mime.text import MIMEText
-
-EMAIL_HOST = "smtp.gmail.com"
-EMAIL_PORT = 465
-EMAIL_HOST_USER = ""
-EMAIL_HOST_PASSWORD = ""
-EMAIL_SEND_TO = ['']
-
-
-def main():
-    send_mail("Hello world!")
-
-
-def send_mail(text):
-    msg = MIMEText(text)
-    msg['Subject'] = "Message from restic-compose-backup"
-    msg['From'] = EMAIL_HOST_USER
-    msg['To'] = ', '.join(EMAIL_SEND_TO)
-
-    try:
-        print("Connecting to {} port {}".format(EMAIL_HOST, EMAIL_PORT))
-        server = smtplib.SMTP_SSL(EMAIL_HOST, EMAIL_PORT)
-        server.ehlo()
-        server.login(EMAIL_HOST_USER, EMAIL_HOST_PASSWORD)
-        server.sendmail(EMAIL_HOST_USER, EMAIL_SEND_TO, msg.as_string())
-        print('Email Sent')
-    except Exception as e:
-        print(e)
-    finally:
-        server.close()
-
-
-if __name__ == '__main__':
-    main()

restic_compose_backup/restic.py

@@ -1,72 +0,0 @@
-"""
-Restic commands
-"""
-import logging
-from typing import List
-from subprocess import Popen, PIPE
-from restic_compose_backup import commands
-
-logger = logging.getLogger(__name__)
-
-
-def init_repo(repository: str):
-    """
-    Attempt to initialize the repository.
-    Doing this after the repository is initialized
-    """
-    return commands.run(restic(repository, [
-        "init",
-    ]))
-
-
-def backup_files(repository: str, source='/backup'):
-    return commands.run(restic(repository, [
-        "--verbose",
-        "backup",
-        source,
-    ]))
-
-
-def backup_from_stdin(repository: str, filename: str, source_command: List[str]):
-    """
-    Backs up from stdin running the source_command passed in.
-    It will appear in restic with the filename (including path) passed in.
-    """
-    dest_command = restic(repository, [
-        'backup',
-        '--stdin',
-        '--stdin-filename',
-        filename,
-    ])
-
-    # pipe source command into dest command
-    source_process = Popen(source_command, stdout=PIPE)
-    dest_process = Popen(dest_command, stdin=source_process.stdout)
-    dest_process.communicate()
-
-    # Ensure both processes exited with code 0
-    source_exit, dest_exit = source_process.poll(), dest_process.poll()
-    return 0 if (source_exit == 0 and dest_exit == 0) else 1
-
-
-def snapshots(repository: str):
-    return commands.run(restic(repository, [
-        "snapshots",
-    ]))
-
-
-def check(repository: str):
-    return commands.run(restic(repository, [
-        "check",
-    ]))
-
-
-def restic(repository: str, args: List[str]):
-    """Generate restic command"""
-    return [
-        "restic",
-        "--cache-dir",
-        "/restic_cache",
-        "-r",
-        repository,
-    ] + args

restic_compose_backup/utils.py

@@ -1,54 +0,0 @@
-import os
-from contextlib import contextmanager
-import docker
-
-from restic_compose_backup.config import Config
-
-TRUE_VALUES = ['1', 'true', 'True', True, 1]
-
-
-def list_containers():
-    """
-    List all containers.
-
-    Returns:
-        List of raw container json data from the api
-    """
-    config = Config()
-    client = docker.DockerClient(base_url=config.docker_base_url)
-    all_containers = client.containers.list()
-    client.close()
-    return [c.attrs for c in all_containers]
-
-
-def is_true(value):
-    """
-    Evaluates the truthfullness of a bool value in container labels
-    """
-    return value in TRUE_VALUES
-
-
-def strip_root(path):
-    """
-    Removes the root slash in a path.
-    Example: /srv/data becomes srv/data
-    """
-    path = path.strip()
-    if path.startswith('/'):
-        return path[1:]
-
-    return path
-
-
-@contextmanager
-def environment(name, value):
-    """Tempset env var"""
-    old_val = os.environ.get(name)
-    os.environ[name] = value
-    try:
-        yield
-    finally:
-        if old_val is None:
-            del os.environ[name]
-        else:
-            os.environ[name] = old_val

src/.dockerignore

@@ -0,0 +1,3 @@
+tests/
+__pycache__
+.DS_Store

src/Dockerfile

@@ -0,0 +1,18 @@
+FROM restic/restic:0.17.3
+
+ENV PIP_BREAK_SYSTEM_PACKAGES 1
+
+RUN apk update && apk add python3 \ 
+    py3-pip \
+    dcron \
+    mariadb-client \
+    postgresql-client \
+    mariadb-connector-c-dev 
+
+ADD . /restic-compose-backup
+WORKDIR /restic-compose-backup
+RUN pip3 install -U pip setuptools wheel && pip3 install -e .
+ENV XDG_CACHE_HOME=/cache
+
+ENTRYPOINT []
+CMD ["./entrypoint.sh"]

src/crontab

@@ -0,0 +1,2 @@
+10 2 * * * source /env.sh && rcb backup > /proc/1/fd/1
+

src/entrypoint.sh

@@ -1,7 +1,10 @@
 #!/bin/sh
 
 # Dump all env vars so we can source them in cron jobs
-printenv | sed 's/^\(.*\)$/export \1/g' > /env.sh
+rcb dump-env > /env.sh
+
+# Write crontab
+rcb crontab > crontab
 
 # start cron in the foreground
 crontab crontab

src/restic_compose_backup/__init__.py

@@ -0,0 +1 @@
+__version__ = '0.7.1'

src/restic_compose_backup/alerts/__init__.py

@@ -0,0 +1,43 @@
+import logging
+
+from restic_compose_backup.alerts.smtp import SMTPAlert
+from restic_compose_backup.alerts.discord import DiscordWebhookAlert
+
+logger = logging.getLogger(__name__)
+
+ALERT_INFO = 'INFO',
+ALERT_ERROR = 'ERROR'
+ALERT_TYPES = [ALERT_INFO, ALERT_ERROR]
+BACKENDS = [SMTPAlert, DiscordWebhookAlert]
+
+
+def send(subject: str = None, body: str = None, alert_type: str = 'INFO'):
+    """Send alert to all configured backends"""
+    alert_classes = configured_alert_types()
+    for instance in alert_classes:
+        logger.info('Configured: %s', instance.name)
+        try:
+            instance.send(
+                subject=f'[{alert_type}] {subject}',
+                body=body,
+            )
+        except Exception as ex:
+            logger.error("Exception raised when sending alert [%s]: %s", instance.name, ex)
+            logger.exception(ex)
+
+    if len(alert_classes) == 0:
+        logger.info("No alerts configured")
+
+
+def configured_alert_types():
+    """Returns a list of configured alert class instances"""
+    logger.debug('Getting alert backends')
+    entires = []
+
+    for cls in BACKENDS:
+        instance = cls.create_from_env()
+        logger.debug("Alert backend '%s' configured: %s", cls.name, instance is not None)
+        if instance:
+            entires.append(instance)
+
+    return entires

src/restic_compose_backup/alerts/base.py

@@ -0,0 +1,14 @@
+
+
+class BaseAlert:
+    name = None
+
+    def create_from_env(self):
+        return None
+
+    @property
+    def properly_configured(self) -> bool:
+        return False
+
+    def send(self, subject: str = None, body: str = None, alert_type: str = None):
+        pass

src/restic_compose_backup/alerts/discord.py

@@ -0,0 +1,48 @@
+import os
+import logging
+
+import requests
+from restic_compose_backup.alerts.base import BaseAlert
+
+logger = logging.getLogger(__name__)
+
+
+class DiscordWebhookAlert(BaseAlert):
+    name = 'discord_webhook'
+    success_codes = [200]
+
+    def __init__(self, webhook_url):
+        self.url = webhook_url
+
+    @classmethod
+    def create_from_env(cls):
+        instance = cls(os.environ.get('DISCORD_WEBHOOK'))
+
+        if instance.properly_configured:
+            return instance
+
+        return None
+
+    @property
+    def properly_configured(self) -> bool:
+        return isinstance(self.url, str) and self.url.startswith("https://")
+
+    def send(self, subject: str = None, body: str = None, alert_type: str = None):
+        """Send basic webhook request. Max embed size is 6000"""
+        logger.info("Triggering discord webhook")
+        # NOTE: The title size is 2048
+        #       The max description size is 2048
+        #       Total embed size limit is 6000 characters (per embed)
+        data = {
+            'embeds': [
+                {
+                    'title': subject[-256:],
+                    'description': body[-2048:] if body else "",
+                },
+            ]
+        }
+        response = requests.post(self.url, params={'wait': True}, json=data)
+        if response.status_code not in self.success_codes:
+            logger.error("Discord webhook failed: %s: %s", response.status_code, response.content)
+        else:
+            logger.info('Discord webhook successful')

src/restic_compose_backup/alerts/smtp.py

@@ -0,0 +1,56 @@
+import os
+import smtplib
+import logging
+from email.mime.text import MIMEText
+
+from restic_compose_backup.alerts.base import BaseAlert
+
+logger = logging.getLogger(__name__)
+
+
+class SMTPAlert(BaseAlert):
+    name = 'smtp'
+
+    def __init__(self, host, port, user, password, to):
+        self.host = host
+        self.port = port
+        self.user = user
+        self.password = password or ""
+        self.to = to
+
+    @classmethod
+    def create_from_env(cls):
+        instance = cls(
+            os.environ.get('EMAIL_HOST'),
+            os.environ.get('EMAIL_PORT'),
+            os.environ.get('EMAIL_HOST_USER'),
+            os.environ.get('EMAIL_HOST_PASSWORD'),
+            (os.environ.get('EMAIL_SEND_TO') or "").split(','),
+        )
+        if instance.properly_configured:
+            return instance
+
+        return None
+
+    @property
+    def properly_configured(self) -> bool:
+        return self.host and self.port and self.user and len(self.to) > 0
+
+    def send(self, subject: str = None, body: str = None, alert_type: str = 'INFO'):
+        # send_mail("Hello world!")
+        msg = MIMEText(body)
+        msg['Subject'] = f"[{alert_type}] {subject}"
+        msg['From'] = self.user
+        msg['To'] = ', '.join(self.to)
+
+        try:
+            logger.info("Connecting to %s port %s", self.host, self.port)
+            server = smtplib.SMTP_SSL(self.host, self.port)
+            server.ehlo()
+            server.login(self.user, self.password)
+            server.sendmail(self.user, self.to, msg.as_string())
+            logger.info('Email sent')
+        except Exception as ex:
+            logger.exception(ex)
+        finally:
+            server.close()

src/restic_compose_backup/backup_runner.py

@@ -1,8 +1,7 @@
 import logging
 import os
-import docker
 
-from restic_compose_backup.config import Config
+from restic_compose_backup import utils
 
 logger = logging.getLogger(__name__)
 
@@ -10,8 +9,7 @@ logger = logging.getLogger(__name__)
 def run(image: str = None, command: str = None, volumes: dict = None,
         environment: dict = None, labels: dict = None, source_container_id: str = None):
     logger.info("Starting backup container")
-    config = Config()
-    client = docker.DockerClient(base_url=config.docker_base_url)
+    client = utils.docker_client()
 
     container = client.containers.run(
         image,
@@ -19,7 +17,7 @@ def run(image: str = None, command: str = None, volumes: dict = None,
         labels=labels,
         # auto_remove=True,  # We remove the container further down
         detach=True,
-        environment=environment,
+        environment=environment + ['BACKUP_PROCESS_CONTAINER=true'],
         volumes=volumes,
         network_mode=f'container:{source_container_id}',  # Reuse original container's network stack.
         working_dir=os.getcwd(),
@@ -35,7 +33,13 @@ def run(image: str = None, command: str = None, volumes: dict = None,
             line = ""
             while True:
                 try:
-                    line += next(stream).decode()
+                    # Make log streaming work for docker ce 17 and 18.
+                    # For some reason strings are returned instead if bytes.
+                    data = next(stream)
+                    if isinstance(data, bytes):
+                        line += data.decode()
+                    elif isinstance(data, str):
+                        line += data
                     if line.endswith('\n'):
                         break
                 except StopIteration:
@@ -51,7 +55,7 @@ def run(image: str = None, command: str = None, volumes: dict = None,
             fd.write('\n')
             logger.info(line)
 
-
+    container.wait()
     container.reload()
     logger.debug("Container ExitCode %s", container.attrs['State']['ExitCode'])
     container.remove()

src/restic_compose_backup/cli.py

@@ -0,0 +1,331 @@
+import argparse
+import os
+import logging
+
+from restic_compose_backup import (
+    alerts,
+    backup_runner,
+    log,
+    restic,
+)
+from restic_compose_backup.config import Config
+from restic_compose_backup.containers import RunningContainers
+from restic_compose_backup import cron, utils
+
+logger = logging.getLogger(__name__)
+
+
+def main():
+    """CLI entrypoint"""
+    args = parse_args()
+    config = Config()
+    log.setup(level=args.log_level or config.log_level)
+    containers = RunningContainers()
+
+    # Ensure log level is propagated to parent container if overridden
+    if args.log_level:
+        containers.this_container.set_config_env('LOG_LEVEL', args.log_level)
+
+    if args.action == 'status':
+        status(config, containers)
+
+    elif args.action == 'snapshots':
+        snapshots(config, containers)
+
+    elif args.action == 'backup':
+        backup(config, containers)
+
+    elif args.action == 'start-backup-process':
+        start_backup_process(config, containers)
+
+    elif args.action == 'cleanup':
+        cleanup(config, containers)
+
+    elif args.action == 'alert':
+        alert(config, containers)
+
+    elif args.action == 'version':
+        import restic_compose_backup
+        print(restic_compose_backup.__version__)
+
+    elif args.action == "crontab":
+        crontab(config)
+
+    elif args.action == "dump-env":
+        dump_env()
+
+    # Random test stuff here
+    elif args.action == "test":
+        nodes = utils.get_swarm_nodes()
+        print("Swarm nodes:")
+        for node in nodes:
+            addr = node.attrs['Status']['Addr']
+            state = node.attrs['Status']['State']
+            print(' - {} {} {}'.format(node.id, addr, state))
+
+
+def status(config, containers):
+    """Outputs the backup config for the compose setup"""
+    logger.info("Status for compose project '%s'", containers.project_name)
+    logger.info("Repository: '%s'", config.repository)
+    logger.info("Backup currently running?: %s", containers.backup_process_running)
+    logger.info("Include project name in backup path?: %s", utils.is_true(config.include_project_name))
+    logger.debug("Exclude bind mounts from backups?: %s", utils.is_true(config.exclude_bind_mounts))
+    logger.info("Checking docker availability")
+
+    utils.list_containers()
+
+    if containers.stale_backup_process_containers:
+        utils.remove_containers(containers.stale_backup_process_containers)
+
+    # Check if repository is initialized with restic snapshots
+    if not restic.is_initialized(config.repository):
+        logger.info("Could not get repository info. Attempting to initialize it.")
+        result = restic.init_repo(config.repository)
+        if result == 0:
+            logger.info("Successfully initialized repository: %s", config.repository)
+        else:
+            logger.error("Failed to initialize repository")
+
+    logger.info("%s Detected Config %s", "-" * 25, "-" * 25)
+
+    # Start making snapshots
+    backup_containers = containers.containers_for_backup()
+    for container in backup_containers:
+        logger.info('service: %s', container.service_name)
+
+        if container.volume_backup_enabled:
+            for mount in container.filter_mounts():
+                logger.info(
+                    ' - volume: %s -> %s',
+                    mount.source,
+                    container.get_volume_backup_destination(mount, '/volumes'),
+                )
+
+        if container.database_backup_enabled:
+            instance = container.instance
+            ping = instance.ping()
+            logger.info(
+                ' - %s (is_ready=%s) -> %s',
+                instance.container_type,
+                ping,
+                instance.backup_destination_path(),
+            )
+            if not ping:
+                logger.error("Database '%s' in service %s cannot be reached",
+                             instance.container_type, container.service_name)
+
+    if len(backup_containers) == 0:
+        logger.info("No containers in the project has 'restic-compose-backup.*' label")
+
+    logger.info("-" * 67)
+
+
+def backup(config, containers):
+    """Request a backup to start"""
+    # Make sure we don't spawn multiple backup processes
+    if containers.backup_process_running:
+        alerts.send(
+            subject="Backup process container already running",
+            body=(
+                "A backup process container is already running. \n"
+                f"Id: {containers.backup_process_container.id}\n"
+                f"Name: {containers.backup_process_container.name}\n"
+            ),
+            alert_type='ERROR',
+        )
+        raise RuntimeError("Backup process already running")
+
+    # Map all volumes from the backup container into the backup process container
+    volumes = containers.this_container.volumes
+
+    # Map volumes from other containers we are backing up
+    mounts = containers.generate_backup_mounts('/volumes')
+    volumes.update(mounts)
+
+    logger.debug('Starting backup container with image %s', containers.this_container.image)
+    try:
+        result = backup_runner.run(
+            image=containers.this_container.image,
+            command='restic-compose-backup start-backup-process',
+            volumes=volumes,
+            environment=containers.this_container.environment,
+            source_container_id=containers.this_container.id,
+            labels={
+                containers.backup_process_label: 'True',
+                "com.docker.compose.project": containers.project_name,
+            },
+        )
+    except Exception as ex:
+        logger.exception(ex)
+        alerts.send(
+            subject="Exception during backup",
+            body=str(ex),
+            alert_type='ERROR',
+        )
+        return
+
+    logger.info('Backup container exit code: %s', result)
+
+    # Alert the user if something went wrong
+    if result != 0:
+        alerts.send(
+            subject="Backup process exited with non-zero code",
+            body=open('backup.log').read(),
+            alert_type='ERROR',
+        )
+
+
+def start_backup_process(config, containers):
+    """The actual backup process running inside the spawned container"""
+    if not utils.is_true(os.environ.get('BACKUP_PROCESS_CONTAINER')):
+        logger.error(
+            "Cannot run backup process in this container. Use backup command instead. "
+            "This will spawn a new container with the necessary mounts."
+        )
+        alerts.send(
+            subject="Cannot run backup process in this container",
+            body=(
+                "Cannot run backup process in this container. Use backup command instead. "
+                "This will spawn a new container with the necessary mounts."
+            )
+        )
+        exit(1)
+
+    status(config, containers)
+    errors = False
+
+    # Did we actually get any volumes mounted?
+    try:
+        has_volumes = os.stat('/volumes') is not None
+    except FileNotFoundError:
+        logger.warning("Found no volumes to back up")
+        has_volumes = False
+
+    # Warn if there is nothing to do
+    if len(containers.containers_for_backup()) == 0 and not has_volumes:
+        logger.error("No containers for backup found")
+        exit(1)
+
+    if has_volumes:
+        try:
+            logger.info('Backing up volumes')
+            vol_result = restic.backup_files(config.repository, source='/volumes')
+            logger.debug('Volume backup exit code: %s', vol_result)
+            if vol_result != 0:
+                logger.error('Volume backup exited with non-zero code: %s', vol_result)
+                errors = True
+        except Exception as ex:
+            logger.error('Exception raised during volume backup')
+            logger.exception(ex)
+            errors = True
+
+    # back up databases
+    logger.info('Backing up databases')
+    for container in containers.containers_for_backup():
+        if container.database_backup_enabled:
+            try:
+                instance = container.instance
+                logger.info('Backing up %s in service %s', instance.container_type, instance.service_name)
+                result = instance.backup()
+                logger.debug('Exit code: %s', result)
+                if result != 0:
+                    logger.error('Backup command exited with non-zero code: %s', result)
+                    errors = True
+            except Exception as ex:
+                logger.exception(ex)
+                errors = True
+
+    if errors:
+        logger.error('Exit code: %s', errors)
+        exit(1)
+
+    # Only run cleanup if backup was successful
+    result = cleanup(config, container)
+    logger.debug('cleanup exit code: %s', result)
+    if result != 0:
+        logger.error('cleanup exit code: %s', result)
+        exit(1)
+
+    # Test the repository for errors
+    logger.info("Checking the repository for errors")
+    result = restic.check(config.repository)
+    if result != 0:
+        logger.error('Check exit code: %s', result)
+        exit(1)
+
+    logger.info('Backup completed')
+
+
+def cleanup(config, containers):
+    """Run forget / prune to minimize storage space"""
+    logger.info('Forget outdated snapshots')
+    forget_result = restic.forget(
+        config.repository,
+        config.keep_daily,
+        config.keep_weekly,
+        config.keep_monthly,
+        config.keep_yearly,
+    )
+    logger.info('Prune stale data freeing storage space')
+    prune_result = restic.prune(config.repository)
+    return forget_result and prune_result
+
+
+def snapshots(config, containers):
+    """Display restic snapshots"""
+    stdout, stderr = restic.snapshots(config.repository, last=True)
+    for line in stdout.decode().split('\n'):
+        print(line)
+
+
+def alert(config, containers):
+    """Test alerts"""
+    logger.info("Testing alerts")
+    alerts.send(
+        subject="{}: Test Alert".format(containers.project_name),
+        body="Test message",
+    )
+
+
+def crontab(config):
+    """Generate the crontab"""
+    print(cron.generate_crontab(config))
+
+
+def dump_env():
+    """Dump all environment variables to a script that can be sourced from cron"""
+    print("#!/bin/bash")
+    print("# This file was generated by restic-compose-backup")
+    for key, value in os.environ.items():
+        print("export {}='{}'".format(key, value))
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(prog='restic_compose_backup')
+    parser.add_argument(
+        'action',
+        choices=[
+            'status',
+            'snapshots',
+            'backup',
+            'start-backup-process',
+            'alert',
+            'cleanup',
+            'version',
+            'crontab',
+            'dump-env',
+            'test',
+        ],
+    )
+    parser.add_argument(
+        '--log-level',
+        default=None,
+        choices=list(log.LOG_LEVELS.keys()),
+        help="Log level"
+    )
+    return parser.parse_args()
+
+
+if __name__ == '__main__':
+    main()

src/restic_compose_backup/commands.py

@@ -0,0 +1,104 @@
+import logging
+from typing import List, Tuple, Union
+from subprocess import Popen, PIPE
+
+from restic_compose_backup import utils
+
+logger = logging.getLogger(__name__)
+
+
+def test():
+    return run(['ls', '/volumes'])
+
+
+def ping_mysql(container_id, host, port, username, password) -> int:
+    """Check if the mysql is up and can be reached"""
+    return docker_exec(container_id, [
+        'mysqladmin',
+        'ping',
+        '--user',
+        username,
+    ], environment={
+        'MYSQL_PWD': password
+    })
+
+
+def ping_mariadb(container_id, host, port, username, password) -> int:
+    """Check if the mariadb is up and can be reached"""
+    return docker_exec(container_id, [
+        'mysqladmin',
+        'ping',
+        '--user',
+        username,
+    ], environment={
+        'MYSQL_PWD': password
+    })
+
+
+def ping_postgres(container_id, host, port, username, password) -> int:
+    """Check if postgres can be reached"""
+    return docker_exec(container_id, [
+        "pg_isready",
+        f"--host={host}",
+        f"--port={port}",
+        f"--username={username}",
+    ])
+
+def docker_exec(container_id: str, cmd: List[str], environment: Union[dict, list] = []) -> int:
+    """Execute a command within the given container"""
+    client = utils.docker_client()
+    logger.debug('docker exec inside %s: %s', container_id, ' '.join(cmd))
+    exit_code, (stdout, stderr) = client.containers.get(container_id).exec_run(cmd, demux=True, environment=environment)
+
+    if stdout:
+        log_std('stdout', stdout.decode(),
+                logging.DEBUG if exit_code == 0 else logging.ERROR)
+
+    if stderr:
+        log_std('stderr', stderr.decode(), logging.ERROR)
+
+    return exit_code
+
+
+def run(cmd: List[str]) -> int:
+    """Run a command with parameters"""
+    logger.debug('cmd: %s', ' '.join(cmd))
+    child = Popen(cmd, stdout=PIPE, stderr=PIPE)
+    stdoutdata, stderrdata = child.communicate()
+
+    if stdoutdata.strip():
+        log_std('stdout', stdoutdata.decode(),
+                logging.DEBUG if child.returncode == 0 else logging.ERROR)
+
+    if stderrdata.strip():
+        log_std('stderr', stderrdata.decode(), logging.ERROR)
+
+    logger.debug("returncode %s", child.returncode)
+    return child.returncode
+
+
+def run_capture_std(cmd: List[str]) -> Tuple[str, str]:
+    """Run a command with parameters and return stdout, stderr"""
+    logger.debug('cmd: %s', ' '.join(cmd))
+    child = Popen(cmd, stdout=PIPE, stderr=PIPE)
+    return child.communicate()
+
+
+def log_std(source: str, data: str, level: int):
+    if isinstance(data, bytes):
+        data = data.decode()
+
+    if not data.strip():
+        return
+
+    log_func = logger.debug if level == logging.DEBUG else logger.error
+    log_func('%s %s %s', '-' * 10, source, '-' * 10)
+
+    lines = data.split('\n')
+    if lines[-1] == '':
+        lines.pop()
+
+    for line in lines:
+        log_func(line)
+
+    log_func('-' * 28)

src/restic_compose_backup/config.py

@@ -0,0 +1,39 @@
+import os
+
+
+class Config:
+    default_backup_command = "source /env.sh && rcb backup > /proc/1/fd/1"
+    default_crontab_schedule = "0 2 * * *"
+
+    """Bag for config values"""
+    def __init__(self, check=True):
+        # Mandatory values
+        self.repository = os.environ.get('RESTIC_REPOSITORY')
+        self.password = os.environ.get('RESTIC_REPOSITORY')
+        self.cron_schedule = os.environ.get('CRON_SCHEDULE') or self.default_crontab_schedule
+        self.cron_command = os.environ.get('CRON_COMMAND') or self.default_backup_command
+        self.swarm_mode = os.environ.get('SWARM_MODE') or False
+        self.include_project_name = os.environ.get('INCLUDE_PROJECT_NAME') or False
+        self.exclude_bind_mounts = os.environ.get('EXCLUDE_BIND_MOUNTS') or False
+
+        # Log
+        self.log_level = os.environ.get('LOG_LEVEL')
+
+        # forget / keep
+        self.keep_daily = os.environ.get('KEEP_DAILY') or "7"
+        self.keep_weekly = os.environ.get('KEEP_WEEKLY') or "4"
+        self.keep_monthly = os.environ.get('KEEP_MONTHLY') or "12"
+        self.keep_yearly = os.environ.get('KEEP_YEARLY') or "3"
+
+        if check:
+            self.check()
+
+    def check(self):
+        if not self.repository:
+            raise ValueError("RESTIC_REPOSITORY env var not set")
+
+        if not self.password:
+            raise ValueError("RESTIC_REPOSITORY env var not set")
+
+
+config = Config()

src/restic_compose_backup/containers.py

@@ -1,8 +1,12 @@
 import os
+import logging
 from pathlib import Path
 from typing import List
 
-from restic_compose_backup import utils
+from restic_compose_backup import enums, utils
+from restic_compose_backup.config import config
+
+logger = logging.getLogger(__name__)
 
 VOLUME_TYPE_BIND = "bind"
 VOLUME_TYPE_VOLUME = "volume"
@@ -27,8 +31,8 @@ class Container:
         if self._labels is None:
             raise ValueError('Container meta missing Config->Labels')
 
-        self._include = self._parse_pattern(self.get_label('restic-compose-backup.volumes.include'))
-        self._exclude = self._parse_pattern(self.get_label('restic-compose-backup.volumes.exclude'))
+        self._include = self._parse_pattern(self.get_label(enums.LABEL_VOLUMES_INCLUDE))
+        self._exclude = self._parse_pattern(self.get_label(enums.LABEL_VOLUMES_EXCLUDE))
 
     @property
     def instance(self) -> 'Container':
@@ -60,17 +64,62 @@ class Container:
         """Image name"""
         return self.get_config('Image')
 
+    @property
+    def name(self) -> str:
+        """Container name"""
+        return self._data['Name'].replace('/', '')
+
+    @property
+    def service_name(self) -> str:
+        """Name of the container/service"""
+        return self.get_label('com.docker.compose.service', default='') or \
+            self.get_label('com.docker.swarm.service.name', default='')
+
+    @property
+    def backup_process_label(self) -> str:
+        """str: The unique backup process label for this project"""
+        return f"{enums.LABEL_BACKUP_PROCESS}-{self.project_name}"
+
+    @property
+    def project_name(self) -> str:
+        """str: Name of the compose setup"""
+        return self.get_label('com.docker.compose.project', default='')
+
+    @property
+    def stack_name(self) -> str:
+        """str: Name of the stack is present"""
+        return self.get_label("com.docker.stack.namespace")
+
+    @property
+    def is_oneoff(self) -> bool:
+        """Was this container started with run command?"""
+        return self.get_label('com.docker.compose.oneoff', default='False') == 'True'
+
     @property
     def environment(self) -> list:
         """All configured env vars for the container as a list"""
-        return self.get_config('Env', default=[])
+        return self.get_config('Env')
+
+    def remove(self):
+        self._data.remove()
 
     def get_config_env(self, name) -> str:
         """Get a config environment variable by name"""
         # convert to dict and fetch env var by name
-        data = {i[0:i.find('=')]: i[i.find('=')+1:] for i in self.environment}
+        data = {i[0:i.find('=')]: i[i.find('=') + 1:] for i in self.environment}
         return data.get(name)
 
+    def set_config_env(self, name, value):
+        """Set an environment variable"""
+        env = self.environment
+        new_value = f'{name}={value}'
+        for i, entry in enumerate(env):
+            if f'{name}=' in entry:
+                env[i] = new_value
+                break
+        else:
+            env.append(new_value)
+
     @property
     def volumes(self) -> dict:
         """
@@ -96,7 +145,8 @@ class Container:
 
     @property
     def volume_backup_enabled(self) -> bool:
-        return utils.is_true(self.get_label('restic-compose-backup.volumes'))
+        """bool: If the ``restic-compose-backup.volumes`` label is set"""
+        return utils.is_true(self.get_label(enums.LABEL_VOLUMES_ENABLED))
 
     @property
     def database_backup_enabled(self) -> bool:
@@ -109,46 +159,29 @@ class Container:
 
     @property
     def mysql_backup_enabled(self) -> bool:
-        return utils.is_true(self.get_label('restic-compose-backup.mysql'))
+        """bool: If the ``restic-compose-backup.mysql`` label is set"""
+        return utils.is_true(self.get_label(enums.LABEL_MYSQL_ENABLED))
 
     @property
     def mariadb_backup_enabled(self) -> bool:
-        return utils.is_true(self.get_label('restic-compose-backup.mariadb'))
+        """bool: If the ``restic-compose-backup.mariadb`` label is set"""
+        return utils.is_true(self.get_label(enums.LABEL_MARIADB_ENABLED))
 
     @property
     def postgresql_backup_enabled(self) -> bool:
-        return utils.is_true(self.get_label('restic-compose-backup.postgres'))
+        """bool: If the ``restic-compose-backup.postgres`` label is set"""
+        return utils.is_true(self.get_label(enums.LABEL_POSTGRES_ENABLED))
 
     @property
     def is_backup_process_container(self) -> bool:
         """Is this container the running backup process?"""
-        return self.get_label('restic-compose-backup.backup_process') == 'True'
+        return self.get_label(self.backup_process_label) == 'True'
 
     @property
     def is_running(self) -> bool:
-        """Is the container running?"""
+        """bool: Is the container running?"""
         return self._state.get('Running', False)
 
-    @property
-    def name(self) -> str:
-        """Container name"""
-        return self._data['Name'].replace('/', '')
-
-    @property
-    def service_name(self) -> str:
-        """Name of the container/service"""
-        return self.get_label('com.docker.compose.service', default='')
-
-    @property
-    def project_name(self) -> str:
-        """Name of the compose setup"""
-        return self.get_label('com.docker.compose.project', default='')
-
-    @property
-    def is_oneoff(self) -> bool:
-        """Was this container started with run command?"""
-        return self.get_label('com.docker.compose.oneoff', default='False') == 'True'
-
     def get_config(self, name, default=None):
         """Get value from config dict"""
         return self._config.get(name, default)
@@ -160,8 +193,16 @@ class Container:
     def filter_mounts(self):
         """Get all mounts for this container matching include/exclude filters"""
         filtered = []
+
+        # If exclude_bind_mounts is true, only volume mounts are kept in the list of mounts
+        exclude_bind_mounts = utils.is_true(config.exclude_bind_mounts)
+        mounts = list(filter(lambda m: not exclude_bind_mounts or m.type == "volume", self._mounts))
+
+        if not self.volume_backup_enabled:
+            return filtered
+
         if self._include:
-            for mount in self._mounts:
+            for mount in mounts:
                 for pattern in self._include:
                     if pattern in mount.source:
                         break
@@ -171,29 +212,43 @@ class Container:
                 filtered.append(mount)
 
         elif self._exclude:
-            for mount in self._mounts:
+            for mount in mounts:
                 for pattern in self._exclude:
                     if pattern in mount.source:
                         break
                 else:
                     filtered.append(mount)
         else:
-            return self._mounts
+            return mounts
 
         return filtered
 
-    def volumes_for_backup(self, source_prefix='/backup', mode='ro'):
+    def volumes_for_backup(self, source_prefix='/volumes', mode='ro'):
         """Get volumes configured for backup"""
         mounts = self.filter_mounts()
         volumes = {}
         for mount in mounts:
             volumes[mount.source] = {
-                'bind': str(Path(source_prefix) / self.service_name / Path(utils.strip_root(mount.destination))),
+                'bind': self.get_volume_backup_destination(mount, source_prefix),
                 'mode': mode,
             }
 
         return volumes
 
+    def get_volume_backup_destination(self, mount, source_prefix) -> str:
+        """Get the destination path for backups of the given mount"""
+        destination = Path(source_prefix)
+
+        if utils.is_true(config.include_project_name):
+            project_name = self.project_name
+            if project_name != '':
+                destination /= project_name
+
+        destination /= self.service_name
+        destination /= Path(utils.strip_root(mount.destination))
+
+        return str(destination)
+
     def get_credentials(self) -> dict:
         """dict: get credentials for the service"""
         raise NotImplementedError("Base container class don't implement this")
@@ -206,6 +261,10 @@ class Container:
         """Back up this service"""
         raise NotImplementedError("Base container class don't implement this")
 
+    def backup_destination_path(self) -> str:
+        """Return the path backups will be saved at"""
+        raise NotImplementedError("Base container class don't implement this")
+
     def dump_command(self) -> list:
         """list: create a dump command restic and use to send data through stdin"""
         raise NotImplementedError("Base container class don't implement this")
@@ -295,6 +354,7 @@ class RunningContainers:
         self.containers = []
         self.this_container = None
         self.backup_process_container = None
+        self.stale_backup_process_containers = []
 
         # Find the container we are running in.
         # If we don't have this information we cannot continue
@@ -305,20 +365,51 @@ class RunningContainers:
         if not self.this_container:
             raise ValueError("Cannot find metadata for backup container")
 
-        # Gather all containers in the current compose setup
+        # Gather all running containers in the current compose setup
         for container_data in all_containers:
             container = Container(container_data)
 
+            # Gather stale backup process containers
+            if (self.this_container.image == container.image
+                    and not container.is_running
+                    and container.is_backup_process_container):
+                self.stale_backup_process_containers.append(container)
+
+            # We only care about running containers after this point
+            if not container.is_running:
+                continue
+
             # Detect running backup process container
             if container.is_backup_process_container:
                 self.backup_process_container = container
 
-            # Detect containers belonging to the current compose setup
-            if (container.project_name == self.this_container.project_name
-               and not container.is_oneoff):
-                if container.id != self.this_container.id:
+            # --- Determine what containers should be evaludated
+
+            # If not swarm mode we need to filter in compose project
+            if not config.swarm_mode:
+                if container.project_name != self.this_container.project_name:
+                    continue
+
+            # Containers started manually are not included
+            if container.is_oneoff:
+                continue
+
+            # Do not include the backup process container
+            if container == self.backup_process_container:
+                continue
+
             self.containers.append(container)
 
+    @property
+    def project_name(self) -> str:
+        """str: Name of the compose project"""
+        return self.this_container.project_name
+
+    @property
+    def backup_process_label(self) -> str:
+        """str: The backup process label for this project"""
+        return self.this_container.backup_process_label
+
     @property
     def backup_process_running(self) -> bool:
         """Is the backup process container running?"""
@@ -328,7 +419,7 @@ class RunningContainers:
         """Obtain all containers with backup enabled"""
         return [container for container in self.containers if container.backup_enabled]
 
-    def generate_backup_mounts(self, dest_prefix='/backup') -> dict:
+    def generate_backup_mounts(self, dest_prefix='/volumes') -> dict:
         """Generate mounts for backup for the entire compose setup"""
         mounts = {}
         for container in self.containers_for_backup():
@@ -338,6 +429,7 @@ class RunningContainers:
         return mounts
 
     def get_service(self, name) -> Container:
+        """Container: Get a service by name"""
         for container in self.containers:
             if container.service_name == name:
                 return container

src/restic_compose_backup/containers_db.py

@@ -1,5 +1,7 @@
+from pathlib import Path
+
 from restic_compose_backup.containers import Container
-from restic_compose_backup.config import Config
+from restic_compose_backup.config import config, Config
 from restic_compose_backup import (
     commands,
     restic,
@@ -22,33 +24,52 @@ class MariadbContainer(Container):
     def ping(self) -> bool:
         """Check the availability of the service"""
         creds = self.get_credentials()
-        return commands.ping_mysql(
+
+        return commands.ping_mariadb(
+            self.id,
             creds['host'],
             creds['port'],
             creds['username'],
-            creds['password'],
-        )
+            creds['password']
+        ) == 0
 
     def dump_command(self) -> list:
         """list: create a dump command restic and use to send data through stdin"""
         creds = self.get_credentials()
         return [
             "mysqldump",
-            f"--host={creds['host']}",
-            f"--port={creds['port']}",
             f"--user={creds['username']}",
-            f"--password={creds['password']}",
             "--all-databases",
+            "--no-tablespaces",
         ]
 
     def backup(self):
         config = Config()
+        creds = self.get_credentials()
+
         return restic.backup_from_stdin(
             config.repository,
-            f'/backup/{self.service_name}/all_databases.sql',
+            self.backup_destination_path(),
+            self.id,
             self.dump_command(),
+            environment={
+                'MYSQL_PWD': creds['password']
+            }
         )
 
+    def backup_destination_path(self) -> str:
+        destination = Path("/databases")
+
+        if utils.is_true(config.include_project_name):
+            project_name = self.project_name
+            if project_name != "":
+                destination /= project_name
+
+        destination /= self.service_name
+        destination /= "all_databases.sql"
+
+        return destination
+
 
 class MysqlContainer(Container):
     container_type = 'mysql'
@@ -65,33 +86,52 @@ class MysqlContainer(Container):
     def ping(self) -> bool:
         """Check the availability of the service"""
         creds = self.get_credentials()
+
         return commands.ping_mysql(
+            self.id,
             creds['host'],
             creds['port'],
             creds['username'],
-            creds['password'],
-        )
+            creds['password']
+        ) == 0
 
     def dump_command(self) -> list:
         """list: create a dump command restic and use to send data through stdin"""
         creds = self.get_credentials()
         return [
             "mysqldump",
-            f"--host={creds['host']}",
-            f"--port={creds['port']}",
             f"--user={creds['username']}",
-            f"--password={creds['password']}",
             "--all-databases",
+            "--no-tablespaces",
         ]
 
     def backup(self):
         config = Config()
+        creds = self.get_credentials()
+
         return restic.backup_from_stdin(
             config.repository,
-            f'/backup/{self.service_name}/all_databases.sql',
+            self.backup_destination_path(),
+            self.id,
             self.dump_command(),
+            environment={
+                "MYSQL_PWD": creds['password']
+            }
         )
 
+    def backup_destination_path(self) -> str:
+        destination = Path("/databases")
+
+        if utils.is_true(config.include_project_name):
+            project_name = self.project_name
+            if project_name != "":
+                destination /= project_name
+
+        destination /= self.service_name
+        destination /= "all_databases.sql"
+
+        return destination
+
 
 class PostgresContainer(Container):
     container_type = 'postgres'
@@ -110,11 +150,12 @@ class PostgresContainer(Container):
         """Check the availability of the service"""
         creds = self.get_credentials()
         return commands.ping_postgres(
+            self.id,
             creds['host'],
             creds['port'],
             creds['username'],
             creds['password'],
-        )
+        ) == 0
 
     def dump_command(self) -> list:
         """list: create a dump command restic and use to send data through stdin"""
@@ -122,19 +163,29 @@ class PostgresContainer(Container):
         creds = self.get_credentials()
         return [
             "pg_dump",
-            f"--host={creds['host']}",
-            f"--port={creds['port']}",
             f"--username={creds['username']}",
             creds['database'],
         ]
 
     def backup(self):
         config = Config()
-        creds = self.get_credentials()
 
-        with utils.environment('PGPASSWORD', creds['password']):
         return restic.backup_from_stdin(
             config.repository,
-                f"/backup/{self.service_name}/{creds['database']}.sql",
+            self.backup_destination_path(),
+            self.id,
             self.dump_command(),
         )
+
+    def backup_destination_path(self) -> str:
+        destination = Path("/databases")
+
+        if utils.is_true(config.include_project_name):
+            project_name = self.project_name
+            if project_name != "":
+                destination /= project_name
+
+        destination /= self.service_name
+        destination /= f"{self.get_credentials()['database']}.sql"
+
+        return destination

src/restic_compose_backup/cron.py

@@ -0,0 +1,69 @@
+"""
+# ┌───────────── minute (0 - 59)
+# │ ┌───────────── hour (0 - 23)
+# │ │ ┌───────────── day of the month (1 - 31)
+# │ │ │ ┌───────────── month (1 - 12)
+# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
+# │ │ │ │ │                                   7 is also Sunday on some systems)
+# │ │ │ │ │
+# │ │ │ │ │
+# * * * * * command to execute
+"""
+QUOTE_CHARS = ['"', "'"]
+
+
+def generate_crontab(config):
+    """Generate a crontab entry for running backup job"""
+    command = config.cron_command.strip()
+    schedule = config.cron_schedule
+
+    if schedule:
+        schedule = schedule.strip()
+        schedule = strip_quotes(schedule)
+        if not validate_schedule(schedule):
+            schedule = config.default_crontab_schedule
+    else:
+        schedule = config.default_crontab_schedule
+
+    return f'{schedule} {command}\n'
+
+
+def validate_schedule(schedule: str):
+    """Validate crontab format"""
+    parts = schedule.split()
+    if len(parts) != 5:
+        return False
+
+    for p in parts:
+        if p != '*' and not p.isdigit():
+            return False
+
+    minute, hour, day, month, weekday = parts
+    try:
+        validate_field(minute, 0, 59)
+        validate_field(hour, 0, 23)
+        validate_field(day, 1, 31)
+        validate_field(month, 1, 12)
+        validate_field(weekday, 0, 6)
+    except ValueError:
+        return False
+
+    return True
+
+
+def validate_field(value, min, max):
+    if value == '*':
+        return
+
+    i = int(value)
+    return min <= i <= max
+
+
+def strip_quotes(value: str):
+    """Strip enclosing single or double quotes if present"""
+    if value[0] in QUOTE_CHARS:
+        value = value[1:]
+    if value[-1] in QUOTE_CHARS:
+        value = value[:-1]
+
+    return value

src/restic_compose_backup/enums.py

@@ -0,0 +1,11 @@
+
+# Labels
+LABEL_VOLUMES_ENABLED = 'restic-compose-backup.volumes'
+LABEL_VOLUMES_INCLUDE = 'restic-compose-backup.volumes.include'
+LABEL_VOLUMES_EXCLUDE = 'restic-compose-backup.volumes.exclude'
+
+LABEL_MYSQL_ENABLED = 'restic-compose-backup.mysql'
+LABEL_POSTGRES_ENABLED = 'restic-compose-backup.postgres'
+LABEL_MARIADB_ENABLED = 'restic-compose-backup.mariadb'
+
+LABEL_BACKUP_PROCESS = 'restic-compose-backup.process'

src/restic_compose_backup/log.py

@@ -0,0 +1,28 @@
+import logging
+import os
+import sys
+
+logger = logging.getLogger('restic_compose_backup')
+HOSTNAME = os.environ['HOSTNAME']
+
+DEFAULT_LOG_LEVEL = logging.INFO
+LOG_LEVELS = {
+    'debug': logging.DEBUG,
+    'info': logging.INFO,
+    'warning': logging.WARNING,
+    'error': logging.ERROR,
+}
+
+
+def setup(level: str = 'warning'):
+    """Set up logging"""
+    level = level or ""
+    level = LOG_LEVELS.get(level.lower(), DEFAULT_LOG_LEVEL)
+    logger.setLevel(level)
+
+    ch = logging.StreamHandler(stream=sys.stdout)
+    ch.setLevel(level)
+    # ch.setFormatter(logging.Formatter('%(asctime)s - {HOSTNAME} - %(name)s - %(levelname)s - %(message)s'))
+    # ch.setFormatter(logging.Formatter('%(asctime)s - {HOSTNAME} - %(levelname)s - %(message)s'))
+    ch.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s: %(message)s'))
+    logger.addHandler(ch)

src/restic_compose_backup/restic.py

@@ -0,0 +1,138 @@
+"""
+Restic commands
+"""
+import logging
+from typing import List, Tuple, Union
+from subprocess import Popen, PIPE
+from restic_compose_backup import commands
+from restic_compose_backup import utils
+
+logger = logging.getLogger(__name__)
+
+
+def init_repo(repository: str):
+    """
+    Attempt to initialize the repository.
+    Doing this after the repository is initialized
+    """
+    return commands.run(restic(repository, [
+        "init",
+    ]))
+
+
+def backup_files(repository: str, source='/volumes'):
+    return commands.run(restic(repository, [
+        "--verbose",
+        "backup",
+        source,
+    ]))
+
+
+def backup_from_stdin(repository: str, filename: str, container_id: str,
+                      source_command: List[str], environment: Union[dict, list] = None):
+    """
+    Backs up from stdin running the source_command passed in within the given container.
+    It will appear in restic with the filename (including path) passed in.
+    """
+    dest_command = restic(repository, [
+        'backup',
+        '--stdin',
+        '--stdin-filename',
+        filename,
+    ])
+
+    client = utils.docker_client()
+
+    logger.debug('docker exec inside %s: %s', container_id, ' '.join(source_command))
+
+    # Create and start source command inside the given container
+    handle = client.api.exec_create(container_id, source_command, environment=environment)
+    exec_id = handle["Id"]
+    stream = client.api.exec_start(exec_id, stream=True, demux=True)
+    source_stderr = ""
+
+    # Create the restic process to receive the output of the source command
+    dest_process = Popen(dest_command, stdin=PIPE, stdout=PIPE, stderr=PIPE, bufsize=65536)
+
+    # Send the ouptut of the source command over to restic in the chunks received
+    for stdout_chunk, stderr_chunk in stream:
+        if stdout_chunk:
+            dest_process.stdin.write(stdout_chunk)
+
+        if stderr_chunk:
+            source_stderr += stderr_chunk.decode()
+
+    # Wait for restic to finish
+    stdout, stderr = dest_process.communicate()
+
+    # Ensure both processes exited with code 0
+    source_exit = client.api.exec_inspect(exec_id).get("ExitCode")
+    dest_exit = dest_process.poll()
+    exit_code = source_exit or dest_exit
+
+    if stdout:
+        commands.log_std('stdout', stdout, logging.DEBUG if exit_code == 0 else logging.ERROR)
+
+    if source_stderr:
+        commands.log_std(f'stderr ({source_command[0]})', source_stderr, logging.ERROR)
+
+    if stderr:
+        commands.log_std('stderr (restic)', stderr, logging.ERROR)
+
+    return exit_code
+
+
+def snapshots(repository: str, last=True) -> Tuple[str, str]:
+    """Returns the stdout and stderr info"""
+    args = ["snapshots"]
+    if last:
+        args.append('--last')
+    return commands.run_capture_std(restic(repository, args))
+
+
+def is_initialized(repository: str) -> bool:
+    """
+    Checks if a repository is initialized using snapshots command.
+    Note that this cannot separate between uninitalized repo
+    and other errors, but this method is reccomended by the restic
+    community.
+    """
+    return commands.run(restic(repository, ["snapshots", '--last'])) == 0
+
+
+def forget(repository: str, daily: str, weekly: str, monthly: str, yearly: str):
+    return commands.run(restic(repository, [
+        'forget',
+        '--group-by',
+        'paths',
+        '--keep-daily',
+        daily,
+        '--keep-weekly',
+        weekly,
+        '--keep-monthly',
+        monthly,
+        '--keep-yearly',
+        yearly,
+    ]))
+
+
+def prune(repository: str):
+    return commands.run(restic(repository, [
+        'prune',
+    ]))
+
+
+def check(repository: str):
+    return commands.run(restic(repository, [
+        "check",
+        # "--with-cache",
+    ]))
+
+
+def restic(repository: str, args: List[str]):
+    """Generate restic command"""
+    return [
+        "restic",
+        "-r",
+        repository,
+    ] + args

src/restic_compose_backup/utils.py

@@ -0,0 +1,98 @@
+import os
+import logging
+from typing import List, TYPE_CHECKING
+from contextlib import contextmanager
+import docker
+from docker import DockerClient
+
+if TYPE_CHECKING:
+    from restic_compose_backup.containers import Container
+
+logger = logging.getLogger(__name__)
+
+TRUE_VALUES = ['1', 'true', 'True', True, 1]
+
+
+def docker_client() -> DockerClient:
+    """
+    Create a docker client from the following environment variables::
+
+        DOCKER_HOST=unix://tmp/docker.sock
+        DOCKER_TLS_VERIFY=1
+        DOCKER_CERT_PATH=''
+    """
+    # NOTE: Remove this fallback in 1.0
+    if not os.environ.get('DOCKER_HOST'):
+        os.environ['DOCKER_HOST'] = 'unix://tmp/docker.sock'
+
+    return docker.from_env()
+
+
+def list_containers() -> List[dict]:
+    """
+    List all containers.
+
+    Returns:
+        List of raw container json data from the api
+    """
+    client = docker_client()
+    all_containers = client.containers.list(all=True)
+    client.close()
+    return [c.attrs for c in all_containers]
+
+
+def get_swarm_nodes():
+    client = docker_client()
+    # NOTE: If not a swarm node docker.errors.APIError is raised
+    #       503 Server Error: Service Unavailable
+    #       ("This node is not a swarm manager. Use "docker swarm init" or
+    #       "docker swarm join" to connect this node to swarm and try again.")
+    try:
+        return client.nodes.list()
+    except docker.errors.APIError:
+        return []
+
+
+def remove_containers(containers: List['Container']):
+    client = docker_client()
+    logger.info('Attempting to delete stale backup process containers')
+    for container in containers:
+        logger.info(' -> deleting %s', container.name)
+        try:
+            c = client.containers.get(container.name)
+            c.remove()
+        except Exception as ex:
+            logger.exception(ex)
+
+
+def is_true(value):
+    """
+    Evaluates the truthfullness of a bool value in container labels
+    """
+    return value in TRUE_VALUES
+
+
+def strip_root(path):
+    """
+    Removes the root slash in a path.
+    Example: /srv/data becomes srv/data
+    """
+    path = path.strip()
+    if path.startswith('/'):
+        return path[1:]
+
+    return path
+
+
+@contextmanager
+def environment(name, value):
+    """Tempset env var"""
+    old_val = os.environ.get(name)
+    os.environ[name] = value
+    try:
+        yield
+    finally:
+        if old_val is None:
+            del os.environ[name]
+        else:
+            os.environ[name] = old_val

src/setup.py

@@ -3,12 +3,15 @@ from setuptools import setup, find_namespace_packages
 setup(
     name="restic-compose-backup",
     url="https://github.com/ZettaIO/restic-compose-backup",
-    version="0.2.0",
+    version="0.7.1",
     author="Einar Forselv",
     author_email="eforselv@gmail.com",
-    packages=find_namespace_packages(include=['restic_compose_backup']),
+    packages=find_namespace_packages(include=[
+        'restic_compose_backup',
+        'restic_compose_backup.*',
+    ]),
     install_requires=[
-        'docker==3.7.2',
+        'docker~=6.1.3',
     ],
     entry_points={'console_scripts': [
         'restic-compose-backup = restic_compose_backup.cli:main',

src/tests/container.json

@@ -160,7 +160,7 @@
     "OpenStdin": true,
     "StdinOnce": true,
     "Env": [
-      "DOCKER_BASE_URL=unix://tmp/docker.sock",
+      "DOCKER_HOST=unix://tmp/docker.sock",
       "RESTIC_REPOSITORY=/tmp/backup",
       "RESTIC_PASSWORD=password",
       "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

src/tests/requirements.txt

@@ -1 +1,2 @@
 pytest==4.3.1
+tox

src/tests/tests.py

@@ -3,6 +3,9 @@ import os
 import unittest
 from unittest import mock
 
+os.environ['RESTIC_REPOSITORY'] = "test"
+os.environ['RESTIC_PASSWORD'] = "password"
+
 from restic_compose_backup import utils
 from restic_compose_backup.containers import RunningContainers
 import fixtures
@@ -15,8 +18,8 @@ class ResticBackupTests(unittest.TestCase):
     @classmethod
     def setUpClass(cls):
         """Set up basic environment variables"""
-        os.environ['RESTIC_REPOSITORY'] = "test"
-        os.environ['RESTIC_PASSWORD'] = "password"
+        # os.environ['RESTIC_REPOSITORY'] = "test"
+        # os.environ['RESTIC_PASSWORD'] = "password"
 
     def createContainers(self):
         backup_hash = fixtures.generate_sha256()
@@ -59,6 +62,7 @@ class ResticBackupTests(unittest.TestCase):
             {
                 'service': 'web',
                 'labels': {
+                    'restic-compose-backup.volumes': True,
                     'test': 'test',
                 },
                 'mounts': [{
@@ -76,7 +80,7 @@ class ResticBackupTests(unittest.TestCase):
         ]
         with mock.patch(list_containers_func, fixtures.containers(containers=containers)):
             result = RunningContainers()
-            self.assertEqual(len(result.containers), 3, msg="Three containers expected")
+            self.assertEqual(len(result.containers), 4, msg="Three containers expected")
             self.assertNotEqual(result.this_container, None, msg="No backup container found")
             web_service = result.get_service('web')
             self.assertNotEqual(web_service, None)
@@ -111,7 +115,7 @@ class ResticBackupTests(unittest.TestCase):
         with mock.patch(list_containers_func, fixtures.containers(containers=containers)):
             cnt = RunningContainers()
             self.assertTrue(len(cnt.containers_for_backup()) == 2)
-            self.assertEqual(cnt.generate_backup_mounts(), {'test': {'bind': '/backup/web/test', 'mode': 'ro'}})
+            self.assertEqual(cnt.generate_backup_mounts(), {'test': {'bind': '/volumes/web/test', 'mode': 'ro'}})
 
     def test_include(self):
         containers = self.createContainers()
@@ -119,7 +123,8 @@ class ResticBackupTests(unittest.TestCase):
             {
                 'service': 'web',
                 'labels': {
-                    'restic-compose-backup.include': 'media',
+                    'restic-compose-backup.volumes': True,
+                    'restic-compose-backup.volumes.include': 'media',
                 },
                 'mounts': [
                     {
@@ -142,6 +147,7 @@ class ResticBackupTests(unittest.TestCase):
         self.assertNotEqual(web_service, None, msg="Web service not found")
 
         mounts = web_service.filter_mounts()
+        print(mounts)
         self.assertEqual(len(mounts), 1)
         self.assertEqual(mounts[0].source, '/srv/files/media')
 
@@ -151,7 +157,8 @@ class ResticBackupTests(unittest.TestCase):
             {
                 'service': 'web',
                 'labels': {
-                    'restic-compose-backup.exclude': 'stuff',
+                    'restic-compose-backup.volumes': True,
+                    'restic-compose-backup.volumes.exclude': 'stuff',
                 },
                 'mounts': [
                     {
@@ -187,7 +194,7 @@ class ResticBackupTests(unittest.TestCase):
             {
                 'service': 'backup_runner',
                 'labels': {
-                    'restic-compose-backup.backup_process': 'True',
+                    'restic-compose-backup.process-default': 'True',
                 },
             },
         ]

swarm-stack.yml

@@ -0,0 +1,30 @@
+version: '3.7'
+
+services:
+  mariadb:
+    image: mariadb:10
+    labels:
+      restic-compose-backup.mariadb: "true"
+    environment:
+      - MYSQL_ROOT_PASSWORD=my-secret-pw
+      - MYSQL_DATABASE=mydb
+      - MYSQL_USER=myuser
+      - MYSQL_PASSWORD=mypassword
+    networks:
+      - global
+    volumes:
+      - mariadbdata:/var/lib/mysql
+  files:
+    image: nginx:1.17-alpine
+    labels:
+      restic-compose-backup.volumes: "true"
+    volumes:
+      - files:/srv/files
+
+volumes:
+  mariadbdata:
+  files:
+
+networks:
+  global:
+    external: true

tox.ini

@@ -0,0 +1,56 @@
+# Ensure that this file do not contain non-ascii characters
+# as flake8 can fail to parse the file on OS X and Windows
+
+[tox]
+skipsdist = True
+setupdir={toxinidir}/src
+envlist =
+    py38
+    pep8
+
+[testenv]
+usedevelop = True
+basepython =
+    py38: python3.8
+
+deps = 
+    -r{toxinidir}/src//tests/requirements.txt
+commands =
+    ; coverage run --source=restic_compose_backup -m pytest tests/
+    ; coverage report
+    pytest
+
+[testenv:pep8]
+usedevelop = false
+deps = flake8
+basepython = python3.8
+commands = flake8
+
+[pytest]
+norecursedirs = tests/* .venv/* .tox/* build/ docs/
+
+[flake8]
+# H405: multi line docstring summary not separated with an empty line
+# D100: Missing docstring in public module
+# D101: Missing docstring in public class
+# D102: Missing docstring in public method
+# D103: Missing docstring in public function
+# D104: Missing docstring in public package
+# D105: Missing docstring in magic method
+# D200: One-line docstring should fit on one line with quotes
+# D202: No blank lines allowed after function docstring
+# D203: 1 blank required before class docstring.
+# D204: 1 blank required after class docstring
+# D205: Blank line required between one-line summary and description.
+# D207: Docstring is under-indented
+# D208: Docstring is over-indented
+# D211: No blank lines allowed before class docstring
+# D301: Use r""" if any backslashes in a docstring
+# D400: First line should end with a period.
+# D401: First line should be in imperative mood.
+# *** E302 expected 2 blank lines, found 1
+# *** W503 line break before binary operator
+ignore = H405,D100,D101,D102,D103,D104,D105,D200,D202,D203,D204,D205,D211,D301,D400,D401,W503
+show-source = True
+max-line-length = 120
+exclude = .tox,env,tests,build,conf.py