Shell

Basic loop to iterate over lines in a file

for pkg in $(cat pkgs.txt); do sudo apt purge "$pkg" -y; done

More complex loop using ‘if’ statement

Useful for control actions, cleaning up output, etc.

for node in $(cat nodes.txt); do \
  echo "Node: ${node}"; \
  ssh -q -t "$node" 'if [[ $(lsblk | grep -i lvm) ]]; then sudo apt install mdadm -y; fi'; \
done

Checking busy log files for their contents

This does not hang your console as opposed to using tail -f.

watch -n 0.5 sudo tail /var/log/named/queries.log

Alternative conditional logic in loop

Declare nodes variable separately or prepend to loop and separate with semicolon.

for node in "${nodes[@]}"; do \
  ping -c 2 -W 0.1 "$node" > /dev/null && \
  echo "OK: ${node}" || echo "NOT OK: ${node}"; \
done

‘while’ loop to iterate over lines in a file

Avoids calls to cat as is the case with the for loop example. Using madison command rather than policy seems to be slightly faster.

while read pkg; do \
  if [[ $(apt-cache madison "$pkg") ]]; then \
    echo "OK: ${pkg} exists in some repo"; \
  else \
    echo "NOT OK: ${pkg} doesn't exist in any repo"; \
  fi; \
done < pkgs.txt

Match lines into an array

types=($(grep -oE 'pattern' input.txt))

Grab block of text between two patterns

sed -n '/pattern1/,/pattern2/p' input.txt

Octal permissions for a file or directory

stat -c '%a' /etc/passwd

Grab last character from string

last_character=${string_variable:-1}

Parse output of ‘grep’ into commands

grep -rl '\-\- MARK \-\-' /var/log/* | \
  while read line; do \
    echo "Working with file '${line}'"; \
    grep MARK "$line" | tail -n1; \
  done

Include lines before/after a ‘grep’ match

grep -B 3 -A 3 -i "hv_fcopy" /var/log/messages

Find all unique directories in listed directories that contain files modified 10 minutes ago since the command was ran

ls | xargs -I {} find {} -type f -mmin -10 | cut -d "/" -f2 | sort -u

Find all files in the current directories that were modified at least a minute ago, are larger than 500MB, and long list them

find . -type f -mmin -1 -size +500M -exec ls -lsh {} \;

Find all files in the current directories that were modified at least a day ago, are larger than 2GB, and empty their contents

find . -type f -mtime -1 -size +2G -exec bash -c 'echo > {}' \;

Run command against a list of directories

ls | xargs -I {} git -C {} pull

Step-by-step debug Bash scripts

Move ahead with Enter key.

set -x
trap read debug

Change timezone interactively

dpkg-reconfigure tzdata

Search binary file while ignoring case

grep -ai "end:" /var/log/syslog

System call info for a directory listing

strace -c ls test/

Show line number during script run

echo "DEBUG: ${LINENO}"

Remove duplicated lines in order

awk '!visited[$0]++' your_file > deduplicated_file

Run local script on a remote endpoint

ssh -q <username>@<endpoint> "sudo bash -s" < local_script.sh

mkdir new_directory && cd $_

function mkcd () {
  mkdir "$1"
  cd "$1"
}

From here.

Recall argument to last used command

$_
!$
Alt + .
!:1
!:1-2

From here.

Get SSH key fingerprint

SHA-256
MD5

ssh-keygen -lf ~/.ssh/id_rsa.pub

ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub

Find broken symbolic links

find . -xtype l

Bulk fix relative symbolic links

find . -lname '<relative-to-source target>*' \
  -exec sh -c 'ln -sfn "<new relative-to-source target>/$(basename $0)" $0' {} \;

Run remote script on remote endpoint

ssh -q <username>@<endpoint> './location/to/script'

Create ISO from directory

The -l doesn’t truncate long names and hyphens with underscores aren’t replaced using -iso-level 4.

genisoimage -o data.iso -iso-level 4 -R -l data/

List ISO file contents without mounting

isoinfo -l -i data.iso

Simple colouring for log files

Works for both static and running output.

cat test.log | perl -pe 's/^\[\*\].*/\e[0;36m$&\e[0m/g; s/^\[\+\].*/\e[0;32m$&\e[0m/g; s/^\[\!\].*/\e[0;31m$&\e[0m/g'

From here.

Suppress Python warnings

For situations like these.

export PYTHONWARNINGS='ignore'

Remove last column using delimiter

$ string='my_underscored_string_12345'
$ echo "$string" | rev | cut -d '_' -f 2- | rev
my_underscored_string

Disregard alias execution

$ halt -p
REALLY!? -p
$ alias halt
alias halt='echo "REALLY!?"'
$ \halt -p
Connection to example.com closed by remote host.

Pretty print CSV files

function pretty_csv {
    perl -pe 's/((?<=,)|(?<=^)),/ ,/g;' "$@" | column -t -s, | less  -F -S -X -K
}

pretty_csv data.csv
pretty_csv < data.csv
sort data.csv | pretty_csv

From here

Pretty print TSV files

function pretty_tsv {
    perl -pe 's/((?<=\t)|(?<=^))\t/ \t/g;' "$@" | column -t -s $'\t' | less  -F -S -X -K
}

pretty_tsv data.tsv
pretty_tsv < data.tsv
sort data.tsv | pretty_tsv

Diff two files and save output to file

diff -u file1 file2 > files.diff

Show build information for cloud image

$ cat /etc/cloud/build.info
build_name: server
serial: 20201211.1

Show top disk usage and exclude directories

du -Sh / --exclude=/{proc,sys,dev,var} | sort -rh | head -n 10

Use ’:’ for an infinite loop

while :; do "looping"; done

Execute Bash to ‘unsource’ variables

exec /bin/bash

Use binary version of ‘time’

This provides access to more information.

$(which time) --verbose echo "test"

From here.

Use ‘perf stat’ to repeat a command

Also provides additional useful measurements.

perf stat --null --repeat 5 --table echo "test"

More examples here.

Change key value in array of JSON

.parameters.vmObjects.value |= map(if .vmName == "router" then .moduleSnapshot = "fixed" else . end)

Use indirect references for dynamic variables

for host in "${hosts[@]}"; do
  declare "parent_disk_${host}=$parent_disk"
done

for host in "${hosts[@]}"; do
  parent_disk="parent_disk_${host}"
  echo "${!parent_disk}"
done

Test terminal’s colors

msgcat --color=test

Bulk rename files in place

find . -type f -name '<file name>' -execdir mv {} "description.txt" \;

Encode with Base64 on a single line

echo "text" | base64 -w 0

Convert PEM to single-line

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' combined.pem

Standard output to a new directory

echo "something" | install -D /dev/stdin directory/file.txt

Find and delete files older than 1 year

find /the/dir/to/start/in -type f -mtime +365 -ls -exec rm -f -- {} \;

View permissions as a tree

-p: permissions
-u: username/userid
-f: full path
-i: don’t print indentation lines
-d: print directories only

tree -pufid

Use wildcard in ‘pip uninstall’

pip freeze | grep "azure*" | xargs -n 1 pip uninstall -y

Show transaction history for a package

dnf history list <package>

Show transaction’s historical info

dnf history info <transaction ID>

Undo last transaction

dnf history undo last

List SystemD timers

systemctl list-timers

Show execution of timer and service

journalctl -u name.timer
journalctl -u name.service

Copy remote directory to local system

rsync
scp

rsync -azvhP <user>@<remote>:<remote path> <local path>

scp -rCp <user>@<remote>:<remote path> <local path>

Overwrite existing directory

rsync -av --delete ~/new/ ~/old

Count number of installed kernels

$ sudo dnf list --installed kernel-core* | tail -n +2 | wc -l
9

Increase number of installed kernels

This goes in /etc/dnf/dnf.conf.

...
installonly_limit=10
...

Pin specific kernel version

$ sudo dnf install python3-dnf-plugins-extras-versionlock
$ # List kernel packages
$ rpm -qa kernel
kernel-6.0.18-300.fc37.x86_64
kernel-6.1.7-200.fc37.x86_64
kernel-6.1.8-200.fc37.x86_64
$ Add pin
$ sudo dnf versionlock add kernel-6.0.18-300.fc37.x86_64
Last metadata expiration check: 3:51:11 ago on E 30 jaan  2023 15:47:21.
Adding versionlock on: kernel-0:6.0.18-300.fc37.*
$ # Remove pin
$ sudo dnf versionlock delete kernel-6.0.18-300.fc37.x86_64
...

Undo ad hoc changes made to a service

For example systemd-resolved.

$ systemctl revert systemd-resolved.service
Removed "/etc/systemd/system/systemd-resolved.service.d/override.conf".
Removed "/etc/systemd/system/systemd-resolved.service.d".
$ systemctl restart systemd-resolved.service

Back up a file using brace expansion

Equivalent to cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.

cp /etc/ssh/sshd_config{,.bak}

The same can be applied to directories:

cp -aR public{,.bak}

Restore a backed up file

Equivalent to cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config.

cp /etc/ssh/sshd_config{.bak,}

Download older version of a kernel

koji download-build --arch=x86_64 <kernel package name>

More information about Koji
A list of all kernels built for Fedora
A list of downloadable kernels built for Fedora
- Some may be removed

Fedora Discussion here.

Parallel execution of lines from a file

readarray -t items < items.txt
parallel -kj 20 echo {1} ::: "${items[@]}"

Verify SSL certificate against domain

openssl s_client -connect google.com:443 2> /dev/null | openssl x509 -noout -dates

Run programs with ‘systemd-run’

This will leverage SystemD’s features.

$ systemd-run env
Running as unit: run-19945.service
$ journalctl -u run-19945.service
Sep 08 07:37:21 bupkis systemd[1]: Starting /usr/bin/env...
Sep 08 07:37:21 bupkis systemd[1]: Started /usr/bin/env.
Sep 08 07:37:21 bupkis env[19948]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Sep 08 07:37:21 bupkis env[19948]: LANG=en_US.UTF-8
Sep 08 07:37:21 bupkis env[19948]: BOOT_IMAGE=/vmlinuz-3.11.0-0.rc5.git6.2.fc20.x86_64

Documentation here.

Working with ‘screen’

Start a session: screen
Start any long-running commands
Detach from session: Ctrl+A+D
List sessions: screen -ls
Attach to a running session: screen -r <session ID>

Don’t send anything identifiable over SSH

ssh -a -i /dev/null -o IdentityAgent=/dev/null whoami.filippo.io

Using Ctrl keys

Ctrl + a: move to the beginning of the line
Ctrl + d: if you’ve type something, it deletes the character under the cursor, otherwise it quits the current shell
Ctrl + e: move to the end of the line
Ctrl + k: delete all text from the cursor to the end of the line
Ctrl + l: equivalent to clear
Ctrl + p: same as Up arrow
Ctrl + n: same as Down arrow
Ctrl + s: to stop output to terminal
Ctrl + q: to resume output to terminal after Ctrl + s
Ctrl + r: begins a backward search through command history, keep pressing to continue moving backwards
Ctrl + t: transpose the character before the cursor with the one under the cursor
- Esc + t: transposes the two words before the cursor
Ctrl + u: cut the line before the cursor
- Ctrl + y to paste it
Ctrl + w: cut the word before the cursor
- Ctrl + y to paste it
Ctrl + x + Backspace: delete all text from the beginning of line to the cursor
Ctrl + x + Ctrl + e: launch editor defined by $EDITOR to input your command
- exit the editor (e.g. wq) and the command will be ran automatically
- useful for multi-line commands
Ctrl + z: stop current running process and keep it in background
- fg to continue the process in the foreground
- bg to continue the process in the background
Ctrl + _: undo typing

From here.

Use ampersand in ‘sed’ for efficiency

The ampersand means that the entirety of what was matched will be used in the replacement.

sed 's/create/&-repo/' input.txt