Skip to content
Snippets

Shell

Basic loop to iterate over lines in a file

Terminal window
for pkg in $(cat pkgs.txt); do sudo apt purge "$pkg" -y; done

More complex loop using ‘if’ statement

Useful for control actions, cleaning up output, etc.

Terminal window
for node in $(cat nodes.txt); do \
  echo "Node: ${node}"; \
  ssh -q -t "$node" 'if [[ $(lsblk | grep -i lvm) ]]; then sudo apt install mdadm -y; fi'; \
done

Checking busy log files for their contents

This does not hang your console as opposed to using tail -f.

Terminal window
watch -n 0.5 sudo tail /var/log/named/queries.log

Alternative conditional logic in loop

Declare nodes variable separately or prepend to loop and separate with semicolon.

Terminal window
for node in "${nodes[@]}"; do \
  ping -c 2 -W 0.1 "$node" > /dev/null && \
  echo "OK: ${node}" || echo "NOT OK: ${node}"; \
done

‘while’ loop to iterate over lines in a file

Avoids calls to cat as is the case with the for loop example. Using madison command rather than policy seems to be slightly faster.

Terminal window
while read pkg; do \
  if [[ $(apt-cache madison "$pkg") ]]; then \
    echo "OK: ${pkg} exists in some repo"; \
  else \
    echo "NOT OK: ${pkg} doesn't exist in any repo"; \
  fi; \
done < pkgs.txt

Match lines into an array

Terminal window
types=($(grep -oE 'pattern' input.txt))

Grab block of text between two patterns

Terminal window
sed -n '/pattern1/,/pattern2/p' input.txt

Octal permissions for a file or directory

Terminal window
stat -c '%a' /etc/passwd

Grab last character from string

Terminal window
last_character=${string_variable:-1}

Parse output of ‘grep’ into commands

Terminal window
grep -rl '\-\- MARK \-\-' /var/log/* | \
  while read line; do \
    echo "Working with file '${line}'"; \
    grep MARK "$line" | tail -n1; \
  done

Include lines before/after a ‘grep’ match

Terminal window
grep -B 3 -A 3 -i "hv_fcopy" /var/log/messages

Find all unique directories in listed directories that contain files modified 10 minutes ago since the command was ran

Terminal window
ls | xargs -I {} find {} -type f -mmin -10 | cut -d "/" -f2 | sort -u

Find all files in the current directories that were modified at least a minute ago, are larger than 500MB, and long list them

Terminal window
find . -type f -mmin -1 -size +500M -exec ls -lsh {} \;

Find all files in the current directories that were modified at least a day ago, are larger than 2GB, and empty their contents

Terminal window
find . -type f -mtime -1 -size +2G -exec bash -c 'echo > {}' \;

Run command against a list of directories

Terminal window
ls | xargs -I {} git -C {} pull

Step-by-step debug Bash scripts

Move ahead with Enter key.

Terminal window
set -x
trap read debug

Change timezone interactively

Terminal window
dpkg-reconfigure tzdata

Search binary file while ignoring case

Terminal window
grep -ai "end:" /var/log/syslog

System call info for a directory listing

Terminal window
strace -c ls test/

Show line number during script run

Terminal window
echo "DEBUG: ${LINENO}"

Remove duplicated lines in order

Terminal window
awk '!visited[$0]++' your_file > deduplicated_file

Run local script on a remote endpoint

Terminal window
ssh -q <username>@<endpoint> "sudo bash -s" < local_script.sh

Create new directory and change into it

Terminal window
mkdir new_directory && cd $_

Recall argument to last used command

Terminal window
$_
!$
Alt + .
!:1
!:1-2

From here.

Get SSH key fingerprint

Terminal window
ssh-keygen -lf ~/.ssh/id_rsa.pub
Terminal window
find . -xtype l
Terminal window
find . -lname '<relative-to-source target>*' \
  -exec sh -c 'ln -sfn "<new relative-to-source target>/$(basename $0)" $0' {} \;

Run remote script on remote endpoint

Terminal window
ssh -q <username>@<endpoint> './location/to/script'

Create ISO from directory

The -l doesn’t truncate long names and hyphens with underscores aren’t replaced using -iso-level 4.

Terminal window
genisoimage -o data.iso -iso-level 4 -R -l data/

List ISO file contents without mounting

Terminal window
isoinfo -l -i data.iso

Simple colouring for log files

Works for both static and running output.

Terminal window
cat test.log | perl -pe 's/^\[\*\].*/\e[0;36m$&\e[0m/g; s/^\[\+\].*/\e[0;32m$&\e[0m/g; s/^\[\!\].*/\e[0;31m$&\e[0m/g'

From here.

Suppress Python warnings

For situations like these.

Terminal window
export PYTHONWARNINGS='ignore'

Remove last column using delimiter

Terminal window
$ string='my_underscored_string_12345'
$ echo "$string" | rev | cut -d '_' -f 2- | rev
my_underscored_string

Disregard alias execution

Terminal window
$ halt -p
REALLY!? -p
$ alias halt
alias halt='echo "REALLY!?"'
$ \halt -p
Connection to example.com closed by remote host.

Pretty print CSV files

Terminal window
function pretty_csv {
    perl -pe 's/((?<=,)|(?<=^)),/ ,/g;' "$@" | column -t -s, | less  -F -S -X -K
}


pretty_csv data.csv
pretty_csv < data.csv
sort data.csv | pretty_csv

From here

Pretty print TSV files

Terminal window
function pretty_tsv {
    perl -pe 's/((?<=\t)|(?<=^))\t/ \t/g;' "$@" | column -t -s $'\t' | less  -F -S -X -K
}


pretty_tsv data.tsv
pretty_tsv < data.tsv
sort data.tsv | pretty_tsv

Diff two files and save output to file

Terminal window
diff -u file1 file2 > files.diff

Show build information for cloud image

Terminal window
$ cat /etc/cloud/build.info
build_name: server
serial: 20201211.1

Show top disk usage and exclude directories

Terminal window
du -Sh / --exclude=/{proc,sys,dev,var} | sort -rh | head -n 10

Use ’:’ for an infinite loop

Terminal window
while :; do "looping"; done

Execute Bash to ‘unsource’ variables

Terminal window
exec /bin/bash

Use binary version of ‘time’

This provides access to more information.

Terminal window
$(which time) --verbose echo "test"

From here.

Use ‘perf stat’ to repeat a command

Also provides additional useful measurements.

Terminal window
perf stat --null --repeat 5 --table echo "test"

More examples here.

Change key value in array of JSON

.parameters.vmObjects.value |= map(if .vmName == "router" then .moduleSnapshot = "fixed" else . end)

Use indirect references for dynamic variables

Terminal window
for host in "${hosts[@]}"; do
  declare "parent_disk_${host}=$parent_disk"
done


for host in "${hosts[@]}"; do
  parent_disk="parent_disk_${host}"
  echo "${!parent_disk}"
done

Test terminal’s colors

Terminal window
msgcat --color=test

Bulk rename files in place

Terminal window
find . -type f -name '<file name>' -execdir mv {} "description.txt" \;

Encode with Base64 on a single line

Terminal window
echo "text" | base64 -w 0

Convert PEM to single-line

Terminal window
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' combined.pem

Standard output to a new directory

Terminal window
echo "something" | install -D /dev/stdin directory/file.txt

Find and delete files older than 1 year

Terminal window
find /the/dir/to/start/in -type f -mtime +365 -ls -exec rm -f -- {} \;

View permissions as a tree

  • -p: permissions
  • -u: username/userid
  • -f: full path
  • -i: don’t print indentation lines
  • -d: print directories only
Terminal window
tree -pufid

Use wildcard in ‘pip uninstall’

Terminal window
pip freeze | grep "azure*" | xargs -n 1 pip uninstall -y

Show transaction history for a package

Terminal window
dnf history list <package>

Show transaction’s historical info

Terminal window
dnf history info <transaction ID>

More on ‘history’

Undo last transaction

Terminal window
dnf history undo last

List SystemD timers

Terminal window
systemctl list-timers

Show execution of timer and service

Terminal window
journalctl -u name.timer
journalctl -u name.service

Copy remote directory to local system

Terminal window
rsync -azvhP <user>@<remote>:<remote path> <local path>

Overwrite existing directory

Terminal window
rsync -av --delete ~/new/ ~/old

Count number of installed kernels

Terminal window
$ sudo dnf list --installed kernel-core* | tail -n +2 | wc -l
9

Increase number of installed kernels

This goes in /etc/dnf/dnf.conf.

...
installonly_limit=10
...

Pin specific kernel version

Terminal window
$ sudo dnf install python3-dnf-plugins-extras-versionlock
$ # List kernel packages
$ rpm -qa kernel
kernel-6.0.18-300.fc37.x86_64
kernel-6.1.7-200.fc37.x86_64
kernel-6.1.8-200.fc37.x86_64
$ Add pin
$ sudo dnf versionlock add kernel-6.0.18-300.fc37.x86_64
Last metadata expiration check: 3:51:11 ago on E 30 jaan  2023 15:47:21.
Adding versionlock on: kernel-0:6.0.18-300.fc37.*
$ # Remove pin
$ sudo dnf versionlock delete kernel-6.0.18-300.fc37.x86_64
...

Undo ad hoc changes made to a service

For example systemd-resolved.

Terminal window
$ systemctl revert systemd-resolved.service
Removed "/etc/systemd/system/systemd-resolved.service.d/override.conf".
Removed "/etc/systemd/system/systemd-resolved.service.d".
$ systemctl restart systemd-resolved.service

Back up a file using brace expansion

Equivalent to cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.

Terminal window
cp /etc/ssh/sshd_config{,.bak}

The same can be applied to directories:

Terminal window
cp -aR public{,.bak}

More on brace expansion.

Restore a backed up file

Equivalent to cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config.

Terminal window
cp /etc/ssh/sshd_config{.bak,}

Download older version of a kernel

Terminal window
koji download-build --arch=x86_64 <kernel package name>

Fedora Discussion here.

Parallel execution of lines from a file

Terminal window
readarray -t items < items.txt
parallel -kj 20 echo {1} ::: "${items[@]}"

Verify SSL certificate against domain

Terminal window
openssl s_client -connect google.com:443 2> /dev/null | openssl x509 -noout -dates

Run programs with ‘systemd-run’

This will leverage SystemD’s features.

Terminal window
$ systemd-run env
Running as unit: run-19945.service
$ journalctl -u run-19945.service
Sep 08 07:37:21 bupkis systemd[1]: Starting /usr/bin/env...
Sep 08 07:37:21 bupkis systemd[1]: Started /usr/bin/env.
Sep 08 07:37:21 bupkis env[19948]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Sep 08 07:37:21 bupkis env[19948]: LANG=en_US.UTF-8
Sep 08 07:37:21 bupkis env[19948]: BOOT_IMAGE=/vmlinuz-3.11.0-0.rc5.git6.2.fc20.x86_64

Documentation here.

Working with ‘screen’

  1. Start a session: screen
  2. Start any long-running commands
  3. Detach from session: Ctrl+A+D
  4. List sessions: screen -ls
  5. Attach to a running session: screen -r <session ID>

Don’t send anything identifiable over SSH

Terminal window
ssh -a -i /dev/null -o IdentityAgent=/dev/null whoami.filippo.io

Using Ctrl keys

  • Ctrl + a: move to the beginning of the line
  • Ctrl + d: if you’ve type something, it deletes the character under the cursor, otherwise it quits the current shell
  • Ctrl + e: move to the end of the line
  • Ctrl + k: delete all text from the cursor to the end of the line
  • Ctrl + l: equivalent to clear
  • Ctrl + p: same as Up arrow
  • Ctrl + n: same as Down arrow
  • Ctrl + s: to stop output to terminal
  • Ctrl + q: to resume output to terminal after Ctrl + s
  • Ctrl + r: begins a backward search through command history, keep pressing to continue moving backwards
  • Ctrl + t: transpose the character before the cursor with the one under the cursor
    • Esc + t: transposes the two words before the cursor
  • Ctrl + u: cut the line before the cursor
    • Ctrl + y to paste it
  • Ctrl + w: cut the word before the cursor
    • Ctrl + y to paste it
  • Ctrl + x + Backspace: delete all text from the beginning of line to the cursor
  • Ctrl + x + Ctrl + e: launch editor defined by $EDITOR to input your command
    • exit the editor (e.g. wq) and the command will be ran automatically
    • useful for multi-line commands
  • Ctrl + z: stop current running process and keep it in background
    • fg to continue the process in the foreground
    • bg to continue the process in the background
  • Ctrl + _: undo typing

From here.

Use ampersand in ‘sed’ for efficiency

The ampersand means that the entirety of what was matched will be used in the replacement.

Terminal window
sed 's/create/&-repo/' input.txt