The White Rabbit

Table of Content

In this post, I'll outline a minimal set of Bash commands, complete with examples, designed to address a broad range of use cases.

Let's be honest, we developers have a reputation for taking shortcuts, but there are smarter ways to embrace our laziness.

Even with the rise of Prompt Engineering and the reliance on AI tools like GPT (and previously Stack Overflow) for generating code, it's still beneficial to have a basic set of commands at your fingertips. Remembering these can decrease your stress and enhance your efficiency when working in Linux server environments.

The most common tasks usually involve searching for and manipulating text data, mainly using grep, awk and sed.

The Magnificent Three: grep, awk, sed

Let's directly dive into the examples using the most useful Bash Commands.

It will preserve links (-d), attributes (-p) and is recursive for contained subfolders (-r).

Count Number of Occurrences

grep -c "Login successful" auth.log

Find Lines Without the Search Term

grep -v "OK" healthcheck.log

Print Specific Columns

Using awk, let's extract first and fourth columns from access.log

awk '{print $1, $4}' access.log

Sum a Column of Numbers

awk '{sum += $3} END {print sum}' data.log

Filter Lines Based on a Condition

awk '$5 > 100 {print $0}' metrics.log

Extract Lines between "Start-Section" and "End-Section" (inclusive)

awk '/Start-Section/,/End-Section/' example.txt

Note: it also supports regex.

Remove Duplicate Lines in a File (Without Sorting)

awk '!seen[$0]++' file.txt

Convert CSV to JSON

awk -F',' '{printf "{\"field1\": \"%s\", \"field2\": \"%s\"}\n", $1, $2}' data.csv

Convert Space Delimited File to CSV

awk '{print $1","$2","$3}' file.txt > file.csv

Find and Replace Text

sed 's/oldtext/newtext/g' file.txt

Note: it supports regex. For example, the following replaces the beginning of each line (^) in file.txt with the string newtext:

sed 's/^/newtext/g' file.txt

Delete Lines Containing a Specific String

sed '/unwanted/d' config.txt

Using cut

Extract first column

cut -d',' -f1 data.csv

Display the Third to Fifth Characters of Each Line

cut -c3-5 file.txt

Select Multiple Columns

cut -d':' -f1,3 /etc/passwd

Extract a Range of Columns

echo "one two three four" | cut -d' ' -f2-3

Other Utility Functions

Full Copy of Files

cp -dpr source_directory destination_directory

Search by Name (Case Insensitive)

For directories (-type d) in the whole system (/)

find / -type d -iname "dirname" 2>/dev/null

or for specific files under a specific path

find /var/log -name "*.log"

Search Files Modified in the Last 7 Days

find /home/user -mtime -7

Execute a Command for a List of File

find /backup -name "*.tar.gz" -exec tar xzf {} \;

For each file found by find, the -exec option executes the specified command (in this case, tar xzf {}).

{} is a placeholder that represents the current file being processed.

tar xzf {} extracts the contents of the tar archive specified by {}.

The \; indicates the end of the -exec command.

Find Files bigger than 100 Megabites

find /var/log -size +100M

Find Files and Count Lines Containing a Match

find /path/to/directory -type f -exec grep "pattern" {} \; | wc -l

Replace Text in Files Found Recursively

find /path/to/directory -type f -exec sed -i 's/old_text/new_text/g' {} \;

List Top 10 Memory Consuming Processes

ps aux --sort=-%mem | head -n 11

Create a Compressed Archive of Files Modified Recently

find /path/to/directory -mtime -7 -type f -print0 | tar -czvf archive.tar.gz --null -T -

When used with the find command, the -print0 option outputs the complete filename followed by a null character instead of the usual newline character, which -print employs. This functionality is crucial for handling filenames that include unusual characters, such as spaces, newlines, or other characters that might otherwise be misinterpreted by the shell or various programs.

Monitor a Log File in Real Time for Errors

tail -f /var/log/application.log | grep --line-buffered "ERROR"

The --line-buffered option causing grep using line buffer, meaning writing output each time it saw a newline, instead of waiting to reach 4096 bytes by default

Compare Two Sorted Files

The comm -12 command prints the common lines between two sorted files. Note that the input files should be sorted for accurate results.

comm -12 <(sort file1.txt) <(sort file2.txt)

For example, suppose file1.txt contains:

apple
banana
cherry

And file2.txt contains:

banana
cherry
date

After running the command, the output will be:

banana
cherry

Archive and Compress Multiple Directories

tar -czvf archive.tar.gz /dir1 /dir2 /dir3

Extract to Standard Output

unzip -p file.zip

This can be particularly useful when you want to view the contents of a file quickly or pipe it into another command without actually extracting and saving the files on your filesystem.