Image processing and upload command line tool PICTL

Original link: https://lisz.me/tech/project/pictl.html

foreword

Since the blog changed from WordPress to a static blog (first Hexo and then Jekyll), the image processing and uploading of articles has become a difficult thing. The first is to use the free image bed provided by vgy.me , which supports uploading directly from the clipboard. The operation is relatively simple and does not require any local storage. However, after vgy.me was upgraded and revised, the original clipboard upload function is no longer supported, and occasionally the image pixels are reduced or lost. At the same time, consider that the WebP format may be suitable for blog use, and vgy.me does not support this format yet. So began to switch to the “object storage + CDN” solution. The locally prepared PNG format image is first converted into a WebP format image through the cwebp command line, and then uploaded to the object storage after modifying the file name through the uPic tool. Since Tencent Cloud’s COS object storage and CDN were used before, they often faced trivial matters such as SSL certificate renewal. In this way, the overall efficiency is actually not high, and it can only be said to be barely acceptable.

Existing tools and storage inspection

For this purpose, other tools and object stores are also examined.

PicGo is a picture uploading tool that integrates client UI and command line. Although you can use the method of command series to easily convert the two steps of picture format conversion and upload into one step, there are still some uncomfortable places, such as PicGo Filename modification (as a fixed-length random string) feature is not provided. And uPic itself focuses on client desktop interaction and does not provide a command line interface.

As for object storage, all domestic cloud service providers need to manually update the SSL certificate by themselves, and charges are charged. Although old users of Tencent Cloud CDN can use coupons for free every month, they will be charged once they forget. In fact, there is also a gradual shift to charging. All foreign cloud service providers basically need to use object storage or CDN for a fee, which is not bad.

Of course, there has always been a “Backblaze B2 + Cloudflare” solution on the Internet. Since B2 itself has free quotas for storage and read and write operations, traffic needs to be charged, and domain names cannot be customized, while B2 and Cloudflare belong to the Broadband Alliance (internal traffic is free), and Cloudflare also provides URL rewriting functions. This solution is temporary became a trend. However, the access link of B2 has a string of parameters in the middle, and there are still some parameters after rewriting, and the final URL is not so simple. Overall, it’s still kind of awkward.

The emergence of R2

So wait and see for a long time, until Cloudflare launched R2. R2 is a product that benchmarks against AWS S3, built on Cloudflare’s commitment to the Broadband Alliance, and provides zero-cost egress for storage objects, which is actually a free CDN. Since Cloudflare itself is a CDN service provider, it is easy to customize the object storage access domain name and automatically generate and deploy SSL certificates. R2 provides 10GB of free storage, 10 million free read operations per month, and 1 million free write, update, and delete operations per month. This should be sufficient for general static blogs. Even if the free quota is exceeded, the excess charges will be cheaper than other cloud service providers.

For the author, the R2 product organically combines the two products of object storage and CDN, and solves the “last mile” before the external access of static blog pictures. So I wanted to migrate from COS to R2, but uPic sold a “flaw” at this time. uPic seems to be starting to turn to commercial charging, no new version will be released on Github, and only support for R2 will continue to be updated on the Appstore. Of course, there is a fee for uPic on the Appstore ($4.99, which is not expensive).

open a new project

After fully sorting out my personal needs, I decided to start a new project – to develop a simple and easy-to-use command line tool PICTL (full name Picture C on t ro l ). The tool must have the following properties:

• Considering the storage space of pictures and the loading time of web pages, all pictures should be compressed and converted into WebP format;
• The names of all pictures should be modified into a fixed-length random string, and support uploading to S3-like object storage;
• Easy to use, command line priority, no web page or desktop interactive interface;
• (Optional) Add watermark automatically;
• (Optional) Automatically adjust the picture size according to the settings;
• (Optional) Images can be read directly from the clipboard.

architecture design

According to the listed features, the main modules of this command-line tool are: image processing module and upload module . The architecture is shown in the figure below.

The image processing module includes functions such as image compression, image format conversion, image watermark, size adjustment, etc., and can be seamlessly extended horizontally. The upload module mainly includes support for three storage methods: the first type is the most extensive S3-like object storage, such as R2, AWS S3 and various object storages of domestic and foreign cloud service manufacturers, etc. The second type is still in the blog Widely popular third-party image beds, such as SM.MS , vgy.me , chevereto type image beds, etc. The third category is the self-hosted Git platform and FTP platform.

Since this tool only supports the command line, it is planned to use Python and Click for development. Although command-line tools can be quickly developed with the help of Google Fire , its usage is different from native *nix command-line tools. In contrast, developing with Click may be a little troublesome, but it can develop a more native Python command line tool. Of course, current development still prefers to use Rust or Go language to develop high-performance command-line tools. However, this tool only does some very simple work, there is no performance bottleneck, and it is enough to develop with Python.

Development Plan

Please move to Github Project to learn more.

Instructions

Currently, PICTL has released the v0.1.0 version simultaneously on Github and PyPI . This version implements the basic functions in the above architecture diagram, that is, the first three necessary items in the features. Next, introduce how to install and use this tool.

Install

Two installation methods are currently supported: source code installation and PIP installation , and support for brew will be added in the future.

Source installation

 git clone https://github.com/zhonger/pictl cd pictl pip3 install .

pip install

 pip3 install pictl

use

show version

 ╰─ $ pictl -V PICTL 0.1.0

List all supported commands

The current v0.1.0 version contains four subcommands: config (configuration management), compress (image processing), upload (upload files) and cup (one-click compression, conversion, upload).

 ╰─ $ pictl Usage: pictl [ OPTIONS] COMMAND [ ARGS]... A command line tool for image processing and uploading ( ex. S3-type ) . Now it supports: - transformation from other image types to ` webp ` image as well as image compression. - image file uploading to AWS S3 or Cloudflare R2. Options: -V , --version Show the pictl version. -h , --help Show this message and exit. Commands: compress Compress any image into ` webp ` image. config Operations for the config file ` ~/.pictlrc ` . cup Compress image and upload to remote storage ( compress and... upload Upload the file to remote storage

config subcommand

The config subcommand includes four operations: add (add configuration), delete (delete configuration), info (view all configurations) and init (initialize configuration). To use this tool, you need to perform initial row configuration operations before performing other configuration operations.

 ╰─ $ pictl config Usage: pictl config [ OPTIONS] COMMAND [ ARGS]... Operations for the config file ` ~/.pictlrc ` . Options: -h , --help Show this message and exit. Commands: add Add configs to the config file. delete Delete config group from the config file. info Check the configs. init Initialize config file with default configs.

Initialize the configuration using the pictl config init command as shown below. When executing for the second time, because the configuration file already exists, it will prompt that it has been initialized.

 ╰─ $ pictl config init The settings has been initilized in /home/ubuntu/.pictlrc. ╰─ $ pictl config init /home/ubuntu/.pictlrc already exists. Please add settings or change it manaully.

Use pictl config info to print all configuration information in JSON format. Configuration files are in TOML format by default.

 ╰─ $ pictl config info { 'basic' : { 'length' : 6, 'ntype' : 'random' , 'algorithm' : 'sha1' } , } ╰─ $ cat ~/.pictlrc [ basic] length = 6 ntype = "random" algorithm = "sha1"

Add configuration interactively from the command line using pictl config add (example for R2 below). After the addition is complete, you can view all the newly added configurations again.

 ╰─ $ pictl config add ****** Please input these information ****** Group Name ( default 'blog' ) : test [ ?] Type: : R2 S3 > R2 Account ID: testid Bucket Name: test Key: testkey Secret: testsecret Prefix ( Default is None ) : Access Url ( like ` https://i.example.com ` ) : https://i.example.com New group 'test' has been saved in /Users/zhonger/.pictlrc. ╰─ $ pictl config info { 'basic' : { 'length' : 6, 'ntype' : 'random' , 'algorithm' : 'sha1' } , 'test' : { 'endpoint' : 'https://testid.r2.cloudflarestorage.com/test' , 'bucket' : 'test' , 'prefix' : '' , 'key' : 'testkey' , 'secret' : 'testsecret' , 'url' : 'https://i.example.com' , 'type' : 'R2' } }

Use the pictl config delete command to choose to delete the remote configuration group (here, in addition to test, other configuration groups are added for demonstration).

 ╰─ $ pictl config delete [ ?] Please select one group: : test blog cover > test test has been deleted.

compress subcommand

The input parameter of this subcommand is the image file name , which can include a path. The output image will be saved in the location where the command is executed, not the directory where the original image is located . After performing this operation, the picture will not be uploaded yet, and you need to continue to use the upload command to complete the upload.

 ╰─ $ pictl compress -h Usage: pictl compress [ OPTIONS] FILENAME Compress any image into ` webp ` image. FILENAME is the name of the file to compress. Options: -h , --help Show this message and exit. ╰─ $ pictl compress ../../pictl/fig01.png The output file is EVyP2J.webp

upload subcommand

As shown below, use pictl upload command to specify the upload file name and remote configuration group to complete the upload.

 ╰─ $ pictl upload -h Usage: pictl upload [ OPTIONS] FILENAME GROUP Upload the file to remote storage. FILENAME is the name of the file to upload. GROUP is the group in the config file you want to use. Options: -h , --help Show this message and exit. ╰─ $ pictl upload EVyP2J.webp test Direct URL: https://i.example.com/EVyP2J.webp Markdown: ![ EVyP2J.webp] ( https://i.example.com/EVyP2J.webp ) HTML Code: <img src = "https://i.example.com/EVyP2J.webp" alt = "EVyP2J.webp" />

cup subcommand

cup subcommand = compress subcommand + upload subcommand, only need to specify the initial image file and remote configuration group, and then complete image compression, format conversion, name modification, and upload in one step.

 ╰─ $ pictl cup -h Usage: pictl cup [ OPTIONS] FILENAME GROUP Compress image and upload to remote storage ( compress and upload ) . FILENAME is the name of the file to compress. GROUP is the group in the config file you want to use. Options: -h , --help Show this message and exit. ╰─ $ pictl cup ../../pictl/fig01.png test The output file is Gl8qhI.webp Direct URL: https://i.example.com/Gl8qhI.webp Markdown: ![ Gl8qhI.webp] ( https://i.example.com/Gl8qhI.webp ) HTML Code: <img src = "https://i.example.com/Gl8qhI.webp" alt = "Gl8qhI.webp" />

TODO

The current v0.1.0 version is relatively crude, but it can meet the author’s necessary needs and effectively improve efficiency. The follow-up will continue to improve and add new functions:

• Auto-completion (shell completion) enhances the experience of using the command line.
• Added support for object storage provided by various cloud service providers.
• Added support for various popular image beds.
• Added support for self-hosted Git.
• Added support for FTP.
• Added support for image manipulation including watermarking, resizing, etc.
• Add links to public galleries such as Unplash, making it easier to operate blog covers.

References

• Introducing Cloudflare R2 Storage – Fast, Reliable Object Storage with Reduced Egress Fees
• R2 Documentation

This article is transferred from: https://lisz.me/tech/project/pictl.html
This site is only for collection, and the copyright belongs to the original author.