Production-ready shell startup scripts: The Set Builtin
• 7 minute read
bash, shell, scripts
Table of contents
When you build an application and make it available through container technology, either you have an ENTRYPOINT or CMD instructions at the end of its Dockerfile. Depending on which framework you're using and some requirements you have, sometimes it's better to have a bash script responsible for running your project. When it's available, generally, you'll see a bunch of commands that are executed, like the following script I created for the project Django Multiple Schemas:
#!/usr/bin/env bash
python manage.py makemigrations
python manage.py migrate
python manage.py seed --create-super-user
python manage.py runserver 0.0.0.0:8000Let's suppose the command python manage.py migrate failed its execution. What would happen 🤔? The answer is counter-intuitive, but the script would run fine, even with the error 🤯. It would execute the command python manage.py seed --create-super-user followed by python manage.py runserver 0.0.0.0:8000. How to fix that? Let's know a bit about some arguments of The Set Builtin.
Exit immediately if any command returns a non-zero status
Let's suppose the following script:
#!/usr/bin/env bash
python the_set_builtin/sample_1.py
python the_set_builtin/sample_2_force_error.py
python the_set_builtin/sample_3.pyIf we execute it, we'll have the following output:
▶ docker-compose up why-use-argument-e-with-bug
Creating the-set-builtin_why-use-argument-e-with-bug_1 ... done
Attaching to the-set-builtin_why-use-argument-e-with-bug_1
why-use-argument-e-with-bug_1 | I am the /app/the_set_builtin/sample_1.py!
why-use-argument-e-with-bug_1 | I am the /app/the_set_builtin/sample_2_force_error.py!
why-use-argument-e-with-bug_1 | Let me force an error 👀
why-use-argument-e-with-bug_1 | I am the /app/the_set_builtin/sample_3.py!
the-set-builtin_why-use-argument-e-with-bug_1 exited with code 0The exit code is 0 😠. Now, if you include the option set -e and execute it again, the output changes, fixing the unexpected behavior:
▶ docker-compose up why-use-argument-e-with-fix
Creating the-set-builtin_why-use-argument-e-with-fix_1 ... done
Attaching to the-set-builtin_why-use-argument-e-with-fix_1
why-use-argument-e-with-fix_1 | I am the /app/the_set_builtin/sample_1.py!
why-use-argument-e-with-fix_1 | I am the /app/the_set_builtin/sample_2_force_error.py!
why-use-argument-e-with-fix_1 | Let me force an error 👀
the-set-builtin_why-use-argument-e-with-fix_1 exited with code 1Now the exit code is 1 🥳. According to the documentation about the argument -e:
Exit immediately if a pipeline, which may consist of a single simple command, a list, or a compound command returns a non-zero status.
This argument is quite good! It can protect us from bugs in production. Though it can solve many potential problems, we may need an environment variable. To illustrate, see the PORT:
#!/usr/bin/env bash
python manage.py migrate
python manage.py collectstatic --no-input
gunicorn -cpython:gunicorn_config -b 0.0.0.0:${PORT} aladdin.wsgiIf the variable is not provided, our application might be running without a critical parameter. Is there another argument that can stop the script if this variable is missing?
Treat unset variables as an error when substituting
Now we have the following script:
#!/usr/bin/env bash
python the_set_builtin/sample_1.py
python the_set_builtin/sample_2_force_error.py
python the_set_builtin/sample_3.py
python the_set_builtin/sample_4_env_variable.py "$ALADDIN_WISH"
python the_set_builtin/sample_5.pyIf you run, you'll get the following output:
▶ docker-compose up why-use-argument-u-with-bug
Starting the-set-builtin_why-use-argument-u-with-bug_1 ... done
Attaching to the-set-builtin_why-use-argument-u-with-bug_1
why-use-argument-u-with-bug_1 | I am the /app/the_set_builtin/sample_1.py!
why-use-argument-u-with-bug_1 | I am the /app/the_set_builtin/sample_2_force_error.py!
why-use-argument-u-with-bug_1 | Let me force an error 👀
why-use-argument-u-with-bug_1 | I am the /app/the_set_builtin/sample_3.py!
why-use-argument-u-with-bug_1 | I am the /app/the_set_builtin/sample_4_env_variable.py!
why-use-argument-u-with-bug_1 | Look! I received the following arguments: ['the_set_builtin/sample_4_env_variable.py', '']
why-use-argument-u-with-bug_1 | Length: 2
why-use-argument-u-with-bug_1 | I am the /app/the_set_builtin/sample_5.py!
the-set-builtin_why-use-argument-u-with-bug_1 exited with code 0With the option set -u, here is what you get:
▶ docker-compose up why-use-argument-u-with-fix
Starting the-set-builtin_why-use-argument-u-with-fix_1 ... done
Attaching to the-set-builtin_why-use-argument-u-with-fix_1
why-use-argument-u-with-fix_1 | I am the /app/the_set_builtin/sample_1.py!
why-use-argument-u-with-fix_1 | I am the /app/the_set_builtin/sample_2_force_error.py!
why-use-argument-u-with-fix_1 | Let me force an error 👀
why-use-argument-u-with-fix_1 | I am the /app/the_set_builtin/sample_3.py!
why-use-argument-u-with-fix_1 | ./scripts/with-argument-u.sh: line 10: ALADDIN_WISH: unbound variable
the-set-builtin_why-use-argument-u-with-fix_1 exited with code 1It's rocking 🤘! Let's see the explanation of -u:
Treat unset variables and parameters other than the special parameters ‘@’ or ‘*’ as an error when performing parameter expansion. An error message will be written to the standard error, and a non-interactive shell will exit.
Including both options in our scripts (set -eu) and we are almost good to go! I wrote "almost" because we can add another layer of protection. Let's see the last one.
A pipeline should produce a failure return code if any command fails
To illustrate, imagine you have the script below. Notice that we're using set -e argument to make it safer:
#!/usr/bin/env bash
# https://www.gnu.org/software/bash/manual/bash.html#The-Set-Builtin
# • -e: Exit immediately if a command exits with a non-zero status.
set -e
this-is-a-fake-command-my-friend | echo "You...are late."
echo "A thousand apologies, O patient one."In case the second command fails in the pipeline, here's the output:
▶ docker-compose up why-use-option-pipefail-with-bug
Starting the-set-builtin_why-use-option-pipefail-with-bug_1 ... done
Attaching to the-set-builtin_why-use-option-pipefail-with-bug_1
why-use-option-pipefail-with-bug_1 | ./scripts/without-option-pipefail.sh: line 7: this-is-a-fake-command-my-friend: command not found
why-use-option-pipefail-with-bug_1 | You...are late.
why-use-option-pipefail-with-bug_1 | A thousand apologies, O patient one.
the-set-builtin_why-use-option-pipefail-with-bug_1 exited with code 0Including the option set -e -o pipefail, the output changes to the following:
▶ docker-compose up why-use-option-pipefail-with-fix
Starting the-set-builtin_why-use-option-pipefail-with-fix_1 ... done
Attaching to the-set-builtin_why-use-option-pipefail-with-fix_1
why-use-option-pipefail-with-fix_1 | ./scripts/with-option-pipefail.sh: line 9: this-is-a-fake-command-my-friend: command not found
why-use-option-pipefail-with-fix_1 | You...are late.
the-set-builtin_why-use-option-pipefail-with-fix_1 exited with code 127Its explanation:
If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully. This option is disabled by default.
Shield you from working outside working hours
Wrapping everything up, this is the entry your bash script should have at the top: set -eu -o pipefail. Always insert it in all of your scripts, always. This simple protection will help you a lot during TSHOOT 🙏. One example is when you're running your application without a migration that should have been applied.
You can check out all the code I put in this article on GitHub. There you'll find a docker-compose file with all the services testing each of the arguments I showed here 🤙.