Open source data backup tool Duplicati

/ 掘墓人的小铲子

Original link: https://juemuren4449.com/archives/duplicati

Using Duplicati + Tencent Cloud COS, my server data is perfectly backed up.

file backup needs

I have two Tencent Cloud servers, one deploys a blog, and the other deploys some personal projects. Although the cloud server is very stable and there is basically no problem of data loss, I have personally experienced data loss before. As the saying goes, I have been bitten by a snake for ten years and I am afraid of well ropes, so I have been trying to back up the blog and project data to other places.

I have written a Java program for backing up files myself. Through crontab timing tasks and shell scripts, files can be regularly backed up to Qiniu Cloud. But because there is no visual interface, it is more troublesome to configure, and the program itself does not support data recovery, so you need to go to Qiniu Cloud to download the data yourself, so I still want to find a relatively complete backup tool.

The choice of backup tools must be open source first. In addition to supporting graphical interface configuration, it must also support domestic object storage, and support periodic backup and data recovery.

After looking around, I finally chose the tool ” Duplicati “, which fully meets the above needs. Here is a little more about the storage type. Although the domestic object storage supported by Duplicati is only “Tencent COS”, which is the COS of Tencent Cloud, it can support the object storage of most domestic cloud vendors by using the “S3 compatible” protocol.

Duplicati use

Duplicati open source address: https://github.com/duplicati/duplicati

deploy

First, deploy Duplicati. I chose the Docker Compose method for deployment, and the image used is linuxserver/duplicati .

For the installation of docker and docker compose, please refer to ” Docker environment construction (CentOS chapter) “.

The content of the docker-compose.yml file is as follows:

 version: "2.1" services: duplicati: image: lscr.io/linuxserver/duplicati:latest container_name: duplicati environment: - PUID=1000 - PGID=1000 - TZ=Asia/Shanghai - CLI_ARGS= #optional volumes: - ./appdata/config:/config - ./backups:/backups - ./source:/source - /opt:/opt ports: - 8200:8200 restart: unless-stopped

Here is an additional explanation of the configuration file. Deploying in docker mode may cause some file directories to be unreadable, so you need to add mappings under the volumes node, such as opt mappings. If you also encounter the problem that file directories cannot be read, Refer to the above configuration to add volumes mapping. If you need to read the root directory, you need to change the PUID and PGID to 0, and set the root volumes mapping.

Execute docker compose up -d in the same directory as the docker-compose.yml file to start the project, and access http://{ip}:8200 with a browser or configure domain name access by yourself. You will be prompted to set a password when entering for the first time, or you can set it later in the settings.

first visit

backup file

After Duplicati is deployed, let’s see how to back up files.

Click the “New Backup” menu, and the default selection is “Configure New File”. If you need to import tasks exported from other places, you can choose “Import from File”.

add new backup

Click Next to set the name, description and encryption method of the backup. I chose no encryption here.

General backup settings

Click Next to configure the save location of the backup. The default is to back up to the local disk. You can click the drop-down menu to select other options.

backup save location

Here I choose “S3 compatible”, and then fill in the following configuration:

  • Server: Select a custom server address. My bucket belongs to Beijing, so the address is cos.ap-beijing.myqcloud.com , and other regions can replace ap-beijing in the link
  • Bucket name: Visit ” Tencent Cloud Storage Bucket List ” to view the name of the bucket
  • Folder path: duplicati/photo-stream , fill in according to personal needs
  • AWS Access ID: Visit ” API Key Management ” to view SecretId
  • AWS Access Key: Visit ” API Key Management ” to view the SecretKey

S3 Compatible

Keep other options as default, click Test, a pop-up box prompts to adjust the bucket name, select “No”, and the pop-up box shows that the test is successful again, indicating that the configuration is correct.

Click Next to configure the source data, that is, the files or directories to be backed up. Here I choose to back up my album directory.

source data

Click Next to set the backup plan. I am modifying the time here to 23:00 in the evening.

plan

Click Next to configure the remote volume size and backup retention policy. The size of the remote volume can be kept as default. I have set it to be much larger than the file size to be backed up. The backup retention policy is permanent. You can also choose an appropriate backup retention policy according to your needs.

general options

Click Save, and a pop-up box will prompt whether to continue to disable encryption. Since I chose not to encrypt at the beginning, I choose to continue not to enable encryption here.

don't enable encryption

After the save is successful, it will jump to the home page, and the backup task list will be displayed, where you can execute a task immediately, or click the drop-down menu to make more choices and settings, such as editing, exporting, deleting, viewing logs, etc.

front page

Here I ran a backup task, and as you can see from the log in the figure below, the backup of a 500M file was completed in 40 seconds, which is quite fast.

log

recover files

After talking about backup, let’s talk about how to restore files. Generally, there are two scenarios. One is that the backed up files are accidentally deleted and need to be restored to the original location; the other is that the backed up files need to be downloaded to other devices. Duplicati can be satisfied.

Click the “Recover Files” menu, there are three recovery methods:

  • Restore directly from the backup file: In my case, it is to directly connect to Tencent Cloud COS through the S3 compatible protocol for file recovery
  • Restore from configuration: restore from backup job files exported from other devices
  • Existing backup: restore from the existing backup task of this machine

recover files

The specific operation and configuration options can be checked by yourself, which is skipped here.

experience feelings

storage type selection

The storage type I used at the beginning was Qiniu Cloud, but I found that every time I backed up Duplicati, the backed up files would be downloaded to the server, and Qiniu Cloud’s S3 compatibility method is paid, which means frequent periodic backups will result in higher costs.

After searching, I found that Tencent Cloud will give permanent free quota to old COS users, so I changed to use Tencent Cloud COS. If you are also a Tencent Cloud customer, you can visit the ” Free Quota Resource Package ” to check whether there is a free quota.

Free Quota Resource Pack

At first I chose “Tencent COS” for configuration, but found that the memory usage of Duplicati remained high, and even caused the service to crash. Since I did not find this problem when using Qiniu Cloud’s S3 compatibility protocol, I tried to use S3 Compatible protocol configuration Tencent Cloud COS, sure enough, the problem of excessive memory usage does not exist.

Since my server is also Tencent Cloud, and it is in the same region as COS, this makes the backup go to the intranet, which means that unless I restore the file to my computer or other places, it will not consume the external network downlink traffic quota . The only limitation is the storage capacity, but the 50G free quota is enough.

You can choose the corresponding object storage according to your own server manufacturer. In theory, the same server and object storage should support intranet transmission, and the object storage of most domestic cloud server manufacturers also supports the S3 compatible protocol. It can save a lot of external network downlink traffic quota or fees.

backup process

Duplicati backup will pack and compress the files and upload them, and generate corresponding configuration files in the same directory.

file structure

Each subsequent backup will download the file to the local, and then compare it. As mentioned earlier, this part may incur costs, and the choice of object storage and backup frequency needs to be considered.

advantage

  • The project is open source and free
  • Can be deployed by yourself
  • Take up less resources
  • simple interface
  • easy to use

Precautions

  • Deployment in docker mode may not see some folders, just configure volumes mapping for the corresponding directory
  • Selecting Tencent COS as the storage type will lead to high memory usage, it is recommended to change it to an S3 compatible protocol
  • Backup will cause object storage to generate external network downlink traffic or fees, and you need to consider choosing an appropriate object storage and backup frequency

client

In addition to being used on the server, Duplicati can also be used on the computer. Visit ” Duplicati Download ” to view more system client downloads.

Duplicati cooperates with Tencent Cloud COS to perfectly solve my server file backup needs. If you also have file backup needs, you can try Duplicati.

This article is transferred from: https://juemuren4449.com/archives/duplicati
This site is only for collection, and the copyright belongs to the original author.