Embed binaries in shell scripts

Original link: https://articles.singee.me/embed-bin-in-shell-script

foreword

When building a Linux/Unix system installation package, in addition to being packaged into a standard software package suitable for various distributions, we may also hope to provide a shell script for program installation, and the installation steps are simply converged into two steps : Download script + run script.

Usually, most of these installation scripts download the required resources from the Internet again, which can minimize the size of the script and ensure that the latest version is always installed, but this also leads to the nature of the downloaded “installation package” The above is an “installer”, which cannot be installed offline.

This article will introduce a solution that has been verified in the production environment to dynamically embed URLs in the installation package.

Due to some reasons, this article explains more from the principle level, and cannot provide a complete code solution for the time being, please understand

In addition, the following codes are written for this article based on the principles. Although the principles have been verified in production, the code used has not been strictly verified in production. Please let us know if there are any bugs

script composition

The whole script consists of two parts: head + embed-bin; embed-bin embeds our program without modification, and head is a dynamically generated script for extracting embed-bin from the current script and executing it .

The head script is dynamically generated, but for the sake of easy maintenance, it is in the form of a template

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 twenty one
 twenty two
 twenty three
 twenty four
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73

 #!/bin/sh
 #
 # NAME:
 # PLATFORM: { .Platform }}
 # DIGEST: , @LINES@

 THIS_DIR=$(DIRNAME=$( dirname " $0 " ); cd " $DIRNAME " ; pwd )
 THIS_FILE=$( basename " $0 " )
 THIS_PATH = " $THIS_DIR / $THIS_FILE "
 EXTRACT=10
 FORCE_EXTRACT=0
 PREFIX=
 EXECUTE=10


 USAGE= ""

 while getopts ":h" flag; do
 case " $flag " in
 h)
 printf "%s" " $USAGE "
 exit 2
 ;;
    
)
        
;;
 *)
 printf "ERROR: did not recognize option '%s', please try -h\\n" " $1 "
 exit 1
 ;;
 esac
 done

 # Verify MD5
 printf "%s\\n" "Verifying file..."
 MD5=$( tail -n +@LINES@ " $THIS_PATH " | md5sum )
 if ! echo " $MD5 " | grep >/dev/null; then
 printf "ERROR: md5sum mismatch of tar archive\\n" >&2
 printf "expected: \\n" >&2
 printf "got: %s\\n" " $MD5 " >&2
 exit 3
 the fi


 if [ -z " $PREFIX " ]; then
 PREFIX=$( mktemp -d -p $( pwd ))
 the fi

 if [ " $EXTRACT " = "1" ]; then
 if [ " $FORCE_EXTRACT " = "1" ] || [ ! -f " $PREFIX /.extract-done" ] || [ " $(cat " $PREFIX /.extract-done" ) " != "" ] ; then
 printf "Extracting archive to %s ...\\n" " $PREFIX "

 {
 dd if = " $THIS_PATH " bs=1 skip=@ARCHIVE_FIRST_OFFSET@ count=@ARCHIVE_FIRST_BYTES@ 2>/dev/null
 dd if = " $THIS_PATH " bs=@BLOCK_SIZE@ skip=@ARCHIVE_BLOCK_OFFSET@ count=@ARCHIVE_BLOCKS_COUNT@ 2>/dev/null
 dd if = " $THIS_PATH " bs=1 skip=@ARCHIVE_LAST_OFFSET@ count=@ARCHIVE_LAST_BYTES@ 2>/dev/null
 } | tar zxf - -C " $PREFIX "

 echo -n > " $PREFIX /.extract-done"
 else
 printf "Archive has already been extracted to %s\\n" " $PREFIX "
 the fi
 the fi

 if [ " $EXECUTE " = "1" ]; then
 echo "Run Command:"
 cd " $PREFIX " &&
 the fi


 exit 0
 ## --- DATA --- ##

There are two types of variables in this script template: and %XX% , the main difference is that the entire template rendering is divided into two steps: first render all variables, and then render the remaining %XX% variables; there are no special requirements when rendering the former, but when rendering the latter, it is necessary to ensure that the length and number of lines of the text before and after the variable rendering remain unchanged.

This script will decompress embed-bin as a compressed package. This is mainly because the relevant data may be large (hundreds of megabytes or even GB) when we use it internally. If you only need a small script, you can remove the code related to compression. .

In addition, this script will perform an MD5 check before execution, which is mainly to prevent the incomplete download of the script in some cases. But because embed-bin itself is a compressed package, you can delete the verification-related code to speed up the installation (the reason we keep it internally is because the content of our embed is more than a compressed package or even more than one file, and on the other hand, it is for give a better error message).

This script also provides the ability to pass parameters and specify some default values. This is because in some cases the relevant steps may be abnormal and it is time-consuming to execute all the steps. In actual use, you can delete the script parameters according to actual needs.

The parameters of the script are given by the template rendering engine, which is mainly for maintainability. If you prefer to write related content in the script, you can modify the relevant parts

render script

Not much to say, just go to the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 twenty one
 twenty two
 twenty three
 twenty four
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333

 //go:embed "header.sh.tmpl"
 var headerTemplate string

 type headerOptions struct {
 name string
 MD5 string

 Opts *Opts

 *ArchiveOptions
 }

 type ArchiveOptions struct {
 DefaultPrefix string
 AutoExtract bool
 AutoExecute bool
 Command string // Use $PREFIX to refer to prefix

 Filename string // For builder use, will not enter the final file
 }

 func (o *ArchiveOptions) QuotedCommand() string {
 return shells. Quote(o. Command)
 }

 func renderHeaders (o *headerOptions) ([] byte , error ) {
 t := template. New( "" )

 tt, err := t.Parse(headerTemplate)
 if err != nil {
 return nil , ee. Wrap(err, "invalid template" )
 }

 b := bytes. Buffer{}

 err = tt. Execute(&b, o)
 if err != nil {
 return nil , err
 }

 return b. Bytes(), nil
 }

 func getHeaders (o *headerOptions) ([] byte , error ) {
 tmpl, err := renderHeaders(o)
 if err != nil {
 return nil , err
 }

 lines := bytes. Count(tmpl, [] byte ( "\n" )) + 1

 tmpl = bytes. ReplaceAll(tmpl, [] byte ( "@LINES@" ), [] byte (strconv. Itoa(lines)))

 replaceAndFillSpace(tmpl, "@BLOCK_SIZE@" , blockSize)

 return tmpl, nil
 }

 func replaceAndFillSpace (data [] byte , old string , new int64 ) {
 oldBytes := [] byte (old)
 newString := strconv. FormatInt( new , 10 )

 newWithExtraSpace := append ([] byte (newString), bytes. Repeat([] byte { ' ' }, len (old)- len (newString))...)

 // assert len(old) == len(newWithExtraSpace)

 // Apply replacements to buffer.
 start := 0
 for {
 i := bytes.Index(data[start:], oldBytes)
 if i == -1 {
 return // stop
 }

 start += i
 start += copy (data[start:], newWithExtraSpace)
 }
 }

 type Opts struct {
 All []*Opt
 }

 func (opts *Opts) FlagNames() string {
 b := strings. Builder{}
 for _, opt := range opts. All {
 b. WriteString(opt. Name)
 if len (opt.Arg) != 0 {
 b. WriteString( ":" )
 }
 }

 return b. String()
 }

 func (opts *Opts) Usage() string {
 b := strings. Builder{}

 b. WriteString( "Usage: $0 [options]\n\n" )

 all := make ([][ 2 ] string , 0 , 1 + len (opts. All))

 nameLen := 2

 all = append (all, [ 2 ] string { "-h" , "Print this help message and exit" })

 for _, opt := range opts. All {
 bb := strings. Builder{}
 bb. WriteString( "-" )
 bb.WriteString(opt.Name)

 if opt. Arg != "" {
 bb. WriteString( " [" )
 bb. WriteString(opt. Arg)
 bb. WriteString( "]" )
 }

 name := bb. String()

 if len (name) > nameLen {
 nameLen = len (name)
 }

 all = append (all, [ 2 ] string {name, opt. Help})
 }

 for _, a := range all {
 b. WriteString(a[ 0 ])
 b.WriteString(strings.Repeat( " " , nameLen- len (a[ 0 ])))
 b. WriteString( "\t" )
 b. WriteString(a[ 1 ])
 b. WriteString( "\n" )
 }

 return b. String()
 }

 type Opt struct {
 name string
 Arg string
 Help string
 Action OptAction
 }

 type OptAction interface {
 DoIfSet() [] string
 }

 type DoAndExitAction struct {
 Do [] string
 ExitCode int
 }

 func (a *DoAndExitAction) DoIfSet() [] string {
 r := append ([] string {}, a.Do...)
 r = append (r, "exit " +strconv.Itoa(a.ExitCode))
 return r
 }

 type DoAndContinueAction struct {
 Do [] string
 }

 func (a *DoAndContinueAction) DoIfSet() [] string {
 return a.Do
 }

 func SimpleSetEnvAction (envName string , envValue interface {}) *DoAndContinueAction {
 return &DoAndContinueAction{
 Do: [] string {fmt. Sprintf( "%s=%v" , envName, envValue)},
 }
 }

 type Builder struct {
 name string

 ArchiveOptions *ArchiveOptions
 }

 func openAndWrite (filename string , w io. Writer) ( int64 , error ) {
 f, err := os. Open(filename)
 if err != nil {
 return 0 , err
 }
 defer f. Close()

 return io. Copy(w, f)
 }

 func fillAndSetHeader (prefix, filename string , f io.Writer, headers [] byte , offset int64 ) ( int64 , error ) {

 fileLength, err := openAndWrite(filename, f)
 if err != nil {
 return 0 , ee.Wrap(err, "cannot append data for " +prefix)
 }

 firstOffset := offset
 firstBytes := blockSize - (firstOffset % blockSize)
 replaceAndFillSpace(headers, fmt.Sprintf( "@%s_FIRST_OFFSET@" , prefix), firstOffset)
 replaceAndFillSpace(headers, fmt.Sprintf( "@%s_FIRST_BYTES@" , prefix), firstBytes)

 copy2Start := firstOffset + firstBytes
 copy2Skip := copy2Start / blockSize
 copy2Blocks := (fileLength - copy2Start + firstOffset) / blockSize
 replaceAndFillSpace(headers, fmt.Sprintf( "@%s_BLOCK_OFFSET@" , prefix), copy2Skip)
 replaceAndFillSpace(headers, fmt.Sprintf( "@%s_BLOCKS_COUNT@" , prefix), copy2Blocks)

 copy3Start := (copy2Skip + copy2Blocks) * blockSize
 copy3Size := fileLength - firstBytes - (copy2Blocks * blockSize)
 replaceAndFillSpace(headers, fmt.Sprintf( "@%s_LAST_OFFSET@" , prefix), copy3Start)
 replaceAndFillSpace(headers, fmt.Sprintf( "@%s_LAST_BYTES@" , prefix), copy3Size)

 return fileLength, nil
 }

 func (b *Builder) Build(saveTo string ) error {
 header := &headerOptions{
 Name: b.Name,
 ArchiveOptions: b. ArchiveOptions,
 Opts: &Opts{},
 }

 fileMD5 := md5. New()

 var dataSize int64

 if header. ArchiveOptions != nil {
 if header. ArchiveOptions. AutoExtract {
 header.Opts.All = append (header.Opts.All, &Opt{
 Name: "E" ,
 Help: "Do not extract archive" ,
 Action: SimpleSetEnvAction( "EXTRACT" , 0 ),
 })
 } else {
 header.Opts.All = append (header.Opts.All, &Opt{
 Name: "e" ,
 Help: "Also extract archive" ,
 Action: SimpleSetEnvAction( "EXTRACT" , 1 ),
 })
 }

 header.Opts.All = append (header.Opts.All, &Opt{
 Name: "f" ,
 Help: "Force extract archive" ,
 Action: SimpleSetEnvAction( "FORCE_EXTRACT" , 1 ),
 })

 prefixOpt := &Opt{
 Name: "d" ,
 Arg: "DIR" ,
 Help: "Extract to directory" ,
 Action: &DoAndContinueAction{
 Do: [] string { `PREFIX="${OPTARG}"` },
 },
 }
 if header.ArchiveOptions.DefaultPrefix != "" {
 prefixOpt.Help += fmt.Sprintf( " (default: %s)" , header.ArchiveOptions.DefaultPrefix)
 }

 header.Opts.All = append (header.Opts.All, prefixOpt)

 if header. ArchiveOptions. Command != "" {
 if header. ArchiveOptions. AutoExecute {
 header.Opts.All = append (header.Opts.All, &Opt{
 Name: "X" ,
 Help: "Do not execute command" ,
 Action: SimpleSetEnvAction( "EXECUTE" , 0 ),
 })
 } else {
 header.Opts.All = append (header.Opts.All, &Opt{
 Name: "x" ,
 Help: "Also execute the command" ,
 Action: SimpleSetEnvAction( "EXECUTE" , 1 ),
 })
 }
 }

 n, err := openAndWrite(header. ArchiveOptions. Filename, fileMD5)
 if err != nil {
 return ee.Wrap(err, "failed to read archive file to get md5" )
 }
 dataSize += n
 }

 _ = dataSize

 header.MD5 = hex.EncodeToString(fileMD5.Sum( nil ))

 headers, err := getHeaders(header)
 if err != nil {
 return ee. Wrap(err, "failed to get headers" )
 }

 f, err := os.OpenFile(saveTo, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0755 )
 if err != nil {
 return ee. Wrap(err, "failed to write file" )
 }
 defer f. Close()

 // write header
 headersLen, err := f.Write(headers)
 if err != nil {
 return ee.Wrap(err, "failed to write headers" )
 }

 currentOffset := int64 (headersLen)

 // embed archive
 if header. ArchiveOptions != nil {
 n, err := fillAndSetHeader( "ARCHIVE" , header. ArchiveOptions. Filename, f, headers, currentOffset)
 if err != nil {
 return ee. Wrap(err, "failed to embed installer" )
 }
 currentOffset += n
 }

 _ = currentOffset

 // rewrite headers
 _, err = f. Seek( 0 , 0 )
 if err != nil {
 return ee. Wrap(err, "failed to seek file" )
 }
 newHeadersLen, err := f.Write(headers)
 if err != nil {
 return ee. Wrap(err, "failed to rewrite headers" )
 }
 if headersLen != newHeadersLen {
 return ee. New( "headers unexpected change after rewrite" )
 }

 return nil
 }

use is

 b := &Builder{
 Name: name,
 ArchiveOptions: &binbundler. ArchiveOptions{
 DefaultPrefix: "/path/to/extract" ,
 AutoExtract: true ,
 AutoExecute: true ,
 Command: "bash $PREFIX/install.sh" , # Installation command, simple and directly executable, complex can use an additional script
 Filename: "/path/to/embed" ,
 },
 }
 err = b. Build( "/path/to/script-save-to.sh" )

Throughout the script, relevant template variables are dynamically inserted and relevant offsets are calculated

postscript

This article is more just to provide a way of thinking (use dd to decompress and dynamically generate opt to control the execution process). Compared with using grep and other means to locate binary content on the Internet, it is more efficient and easy to maintain.

On this basis, in fact, more things can be achieved (dependency verification, installation of multiple files, etc.), welcome to try

This article is transferred from: https://articles.singee.me/embed-bin-in-shell-script
This site is only for collection, and the copyright belongs to the original author.