Original link: https://qq52o.me/2809.html
In other places such as GitHub or Gitee, “white prostitution” storage will inevitably encounter inaccessible times. For example, some time ago, jsDelivr was inaccessible due to DNS pollution, and Gitee’s public warehouse was private…
In these two cases, it is okay. You can use git to get data and pull it locally. For students who have a server, you can put it back locally and replace the access link to continue accessing. Without a server or without any backup program, it is very unfriendly, and there may be problems of data loss.
At this time, it is recommended to upload data to the cloud. Take Tencent Cloud as an example: you can migrate local data to COS.
Object Storage (Cloud Object Storage, COS) is a distributed storage service provided by Tencent Cloud for storing massive files. It has the advantages of high scalability, low cost, reliability and security. For different storage types, Tencent Cloud promises that the availability of standard storage services is no less than 99.95%, and the availability of low-frequency storage is no less than 99.9% .
When it comes to migration, the first thing that comes to mind may be the COS Migration tool:
COS Migration is an all-in-one tool that integrates COS data migration functions. Through simple configuration operations, users can migrate local data to COS, which has the following features:
- Resume upload from a breakpoint: The tool supports resume upload from a breakpoint when uploading. For some large files, if you quit midway or because of a service failure, you can re-run the tool to resume uploading the files that have not been uploaded.
- Block upload: Upload objects to COS in blocks.
- Parallel upload: Support multiple objects to upload at the same time.
- Migration check: check after object migration.
However, this article recommends using the COSCLI tool. COS Migration is developed in Java language. It needs to rely on JDK when using it. The Linux environment needs IFUNC support. Make sure that the binutils version of the environment is greater than 2.20, which is not very friendly to novice users. COSCLI is developed in Go language, which is easy to deploy and supports cross-bucket operations.
In addition, the usage of COSCMD has been introduced before, so what is the difference between the COSCLI tool and the COSCMD tool?
- The COSCLI tool is built with golang and directly publishes the compiled binary package. Users do not need to pre-install any dependencies during installation and deployment, and it can be used out of the box; the COSCMD tool is built with Python, and users need to install the Python environment and dependency packages before installation.
- The COSCLI tool supports setting a bucket alias, and a short string can be used to replace
<BucketName-APPID>, which is convenient for users to use; COSCMD tool does not support bucket aliases, users need to input<BucketName-APPID>to specify a bucket, the command is cumbersome and difficult to read. - The COSCLI tool supports configuring multiple buckets in the configuration file and supports cross-bucket operations; the COSCMD tool can only configure one bucket in the configuration file, and the cross-bucket operation commands are too verbose.
Download and install configuration
COSCLI tools provide binary packages for Windows, Mac, and Linux operating systems, which can be used after simple installation and configuration.
| GitHub address | Domestic site |
|---|---|
| Windows | Windows |
| Mac | Mac |
| Linux | Linux |
The current version number is v0.11.1-beta. For the latest version, historical version and change log of the tool, please go to GitHub to check.
Download it according to what environment you need to use. Here, take macOS as an example. For other environments, you can refer to the documentation .
- Download COSCLI
wget https://cosbrowser.cloud.tencent.com/software/coscli/coscli-mac
- Rename and modify file permissions
mv coscli-mac /usr/local/bin/coscli chmod 755 /usr/local/bin/coscli
- Check the version number to see if the installation was successful
$ coscli -v coscli version v0.11.1-beta
Seeing the output of version indicates that the installation was successful.
When using COSCLI under the macOS system, if there is a popup
无法打开“coscli”，因为无法验证开发者, you can go to设置> 安全性与隐私> 通用and choose to仍要打开coscli, and then you can use COSCLI normally.
- Configuration parameters
You can use the coscli --help command to quickly see how to use COSCLI. When using the coscli command for the first time, the initial configuration will be performed. You need to enter information such as Secret ID . After filling in the steps, COSCLI will generate a configuration file at the location of ~/.cos.yaml by default.
$ coscli 2022/08/06 17:11:46 Welcome to coscli! When you use coscli for the first time, you need to input some necessary information to generate the default configuration file of coscli. The path of the configuration file: /Users/lufei/.cos.yaml Input Your Secret ID:
Later, you can also use the coscli config init command to interactively generate configuration files for COSCLI in other locations, or you can directly write the configuration files of COSCLI manually.
The description of each configuration item in the configuration file is as follows:
| configuration item | illustrate |
|---|---|
| Secret ID | Key ID, which can be created and obtained from the Access Management Console . |
| Secret Key | The key Key, which can be created and obtained from the Access Management Console . |
| Session Token | Temporary key token, which needs to be configured when using a temporary key. If you do not use it, you can simply press Enter to skip it. |
| APP ID | The APP ID is the account you get after successfully applying for a Tencent Cloud account. It is automatically assigned by the system and can be obtained from account information . The full name of a bucket consists of two elements, Bucket Name and APP ID , in the format <BucketName-APPID> . For details, see Bucket Naming Specification . |
| Bucket Name | The bucket name, together with the APP ID, form the full name of the bucket, in the format <BucketName-APPID> . For details, see Bucket Naming Specifications . |
| Bucket Region | The region where the bucket is located. For details, see Region and Access Domain Name . |
| Bucket Alias | Bucket alias. After configuration, BucketAlias can be used instead of BucketName-APPID to reduce the length of the command to be entered. If this item is not configured, the value of BucketAlias is the value of BucketName-APPID . |
After the configuration is complete, you can view the ~/.cos.yaml file, and you can see something similar to the following:
$ cat ~/.cos.yaml cos: base: secretid: xxxxx secretkey: xxxxx sessiontoken: xxxxx protocol: https buckets: - name: sy-records-85464277 alias: sy-records-85464277 region: "" endpoint: cos.ap-shanghai.myqcloud.com
secretid/secretkey/sessiontoken are encrypted strings, not the values configured in plain text.
If you need to support multiple buckets , you can use the coscli config add command to add a bucket configuration.
use
COSCLI supports the following commands:
$ coscli --help Welcome to use coscli! Usage: coscli [flags] coscli [command] Available Commands: abort Abort parts bucket-tagging Modify bucket tagging config Init or edit config file cp Upload, download or copy objects du Displays the size of a bucket or objects hash Calculate local file's hash-code or show cos file's hash-code help Help about any command ls List buckets or objects lsparts List multipart uploads mb Create bucket rb Remove bucket restore Restore objects rm Remove objects signurl Gets the signed download URL sync Synchronize objects Flags: -c, --config-path string config file path(default is $HOME/.cos.yaml) -e, --endpoint string config endpoint -h, --help help for coscli -i, --secret-id string config secretId -k, --secret-key string config secretKey -t, --session-token string config sessionToken -v, --version version for coscli Use "coscli [command] --help" for more information about a command.
Here is an introduction to the cp and sync commands
-
The cp command is used to upload, download or copy files.
-
The sync command is used to upload, download or copy files synchronously.
The difference from the cp command is that the sync command first compares the crc64 of the file with the same name, and does not transfer if the crc64 value is the same.
These two commands automatically enable concurrent upload/download when uploading and downloading large files. When uploading/downloading files in chunks, resuming the upload will be enabled by default.
Taking WordPress migration as an example, you can use the following command to upload the WordPress media library to COS, where /yourpath/wp-content/uploads is the local media library storage path of your WordPress site directory, and wp-content/uploads is the storage path Path in COS.
first upload
- Upload all files in the local wp-content/uploads folder to the wp-content/uploads folder in the bucket1 bucket
coscli cp /yourpath/wp-content/uploads/ cos://bucket1/wp-content/uploads/ -r
- Upload all .mp4 files in the local wp-content/uploads folder to the wp-content/uploads folder in bucket1
coscli cp /yourpath/wp-content/uploads/ cos://bucket1/wp-content/uploads/ -r --include .*.mp4
- Upload all non-.md files in the local wp-content/uploads folder to the wp-content/uploads folder in bucket1
coscli cp /yourpath/wp-content/uploads/ cos://bucket1/wp-content/uploads/ -r --exclude .*.md
secondary upload
If there is some abnormality, or if you want to re-upload after manually stopping, you can use sync . This command will compare the crc64 of the file with the same name, and if the crc64 value is the same, it will not be transferred.
coscli sync /yourpath/wp-content/uploads/ cos://bucket1/wp-content/uploads/ -r
For more information on how to use COSCLI, please refer to the official website documentation
This article is reprinted from: https://qq52o.me/2809.html
This site is for inclusion only, and the copyright belongs to the original author.