8 Advanced Linux Command Line Tools

The Power of Moreutils

Introduction

We often start our command-line journey with the essential coreutils package, which provides fundamental tools for file, text, and shell operations. These utilities are the backbone of any Unix-like system, handling everything from copying files with cp to manipulating text with cat. However, as the complexity grows and we transiton to power users, we often find that coreutils alone aren’t enough to tackle more demanding tasks.

In this blog, we’ll explore eight powerful tools from moreutils that can help us become more efficient in using command line.

Watch companion video:

Moreutils Overview

Moreutils is a package of additional command-line tools designed to fill in the gaps left by coreutils. Created by Joey Hess and further extended by the GNU Linux community in 2007, moreutils offers specialized utilities that can enhance your productivity and streamline your workflow.

While tools like grep, sed, and awkare often used for more advanced text processing, moreutils provides a set of unique commands that extend command-line capabilities even further.

Here’s a list of all commands in the package, we will look in detail at some of them in this blog:

• chronic: Runs a command quietly, only outputting if the command fails. This is particularly useful for cron jobs where you only want to be notified when something goes wrong.
• combine: Merges lines from two files using Boolean operations like AND, OR, and XOR, offering more control over file merging tasks.
• errno: Allows you to look up errno names and descriptions, helping you quickly understand error codes without needing to consult documentation.
• ifdata: Retrieves network interface information without requiring you to parse the output of ifconfig, making network management more straightforward.
• ifne: Runs a program only if the standard input is not empty. This is useful for conditional execution based on input.
• isutf8: Checks if a file or standard input is in UTF-8 format, ensuring data encoding consistency across your projects.
• lckdo: Executes a program while holding a lock, preventing other processes from running the same command simultaneously, which is crucial in multi-process environments.
• mispipe: Pipes two commands together, returning the exit
• parallel: Executes multiple commands concurrently, taking full advantage of multi-core processors to speed up operations like batch processing or testing.
• pee: Similar to tee, but instead of writing to multiple files, it sends the standard input to multiple commands simultaneously, allowing you to process input through several commands at once.
• sponge: Soaks up the standard input before writing it to a file, preventing issues with overwriting files that are still being read, a common problem when chaining commands.
• ts: Adds a timestamp to each line of input, which is particularly useful for logging or monitoring the timing of events in real-time.
• vidir: Opens a directory in your text editor, allowing you to rename files as if they were lines in a text file, which is much faster than renaming files one by one.
• vipe: Inserts a text editor into a pipeline, enabling you to manually edit the content of a command’s output before passing it on to the next command.
• zrun: Automatically uncompresses arguments to a command, simplifying the process of working with compressed files.

Command Operations Basics

Before diving into the specific tools, it’s helpful to review some command-line basics. Understanding operators like pipes (|), command substitution ($()), and logical operators (&&, ||) is essential for using both coreutils and moreutils effectively.

1. Pipes (|): Allow output of one command to be used as input for another. Example: ls -l | grep ".txt"
2. Command Substitution ($()): Uses output of a command as an argument for another. Example: echo "Today is $(date)"
3. Logical Operators:

• && (AND): Second command runs only if first succeeds.
• || (OR): Second command runs only if first fails.

Example:

mkdir new_directory && cd new_directory
grep "pattern" file.txt || echo "Pattern not found"

With this out of the way, let’s explore the most useful command line utilities in the package.

combine: Merging Files with Boolean Logic

Photo by Ksenia on Unsplash

The combine command from moreutils lets you merge lines from two files using Boolean operations. This is particularly useful when you need to compare or combine data from different sources.

Here’s a quick example using two simple log files:

Contents of syslog_day1.txt:

Error: Disk quota exceeded
Warning: CPU temperature high
Info: System rebooted

Contents of syslog_day2.txt:

Error: Disk quota exceeded
Warning: CPU temperature high
Info: New user added

Using combine

• AND operation: Find common lines in both files.

combine ./syslog_day1.txt and ./syslog_day2.txt

Output:

Error: Disk quota exceeded
Warning: CPU temperature high

• OR operation: Merge all lines from both files.

combine ./syslog_day1.txt or ./syslog_day2.txt

Output:

Error: Disk quota exceeded
Warning: CPU temperature high
Info: System rebooted
Info: New user added

• XOR operation: Find lines unique to each file.

combine ./syslog_day1.txt xor ./syslog_day2.txt

Output:

Info: System rebooted
Info: New user added

• NOT operation: Find lines in the first file that are not in the second.

combine ./syslog_day1.txt not ./syslog_day2.txt

Output:

Info: System rebooted

ifne: Execute Command If Output Not Empty

Photo by Lidia Nemiroff on Unsplash

The ifne command is a simple but powerful tool that runs a command only if the standard output (stdout) is not empty. This is especially useful in scripts where you want to execute follow-up actions based on whether a previous command produced any output.

Let’s say we have a log file, and want to compress it only if it contains specific entries using the example files from previous chapter.

Using ifne

You can combine grep with ifne to achieve this:

grep "Error" syslog.txt | ifne gzip syslog.txt

Explanation:

• grep "Error" syslog.txt: Searches for the term “Error” in the log file.
• | ifne gzip syslog.txt: If “Error” is found (i.e., if the output is not empty), gzip compresses the syslog.txt file.

If the grep command finds “Error” in the log, ifne ensures that the log file is compressed. If there are no “Error” entries, the file remains unchanged, and no compression is performed.

parallel: Running Multiple Jobs Simultaneously

Photo by Bernd 📷 Dittrich on Unsplash

The parallel command allows us to run multiple jobs concurrently, leveraging the full power of all CPU cores. This can significantly speed up tasks that would otherwise run sequentially.

Example: Running Load Tests with curl

Imagine you want to simulate multiple requests to a website to test its load handling. Instead of running these requests one after another, we can use parallel to run them simultaneously.

Here’s how we can do it:

seq 1 100 | parallel -j10 curl -s -o /dev/null http://example.com

• seq 1 100: Generates a sequence of numbers from 1 to 100, representing 100 requests.
• | parallel -j10: Runs the following command (curl in this case) in parallel, with 10 jobs running simultaneously (-j10).
• curl -s -o /dev/null http://example.com: Sends a request to http://example.com and discards the output (-s for silent mode, -o /dev/null to discard output).

With this setup, 10 curl requests are sent to the server at a time, making it much faster to simulate the 100 total requests. This is particularly useful for load testing or any scenario where you need to run many similar operations in parallel.

In the video, you can see how parallel improves speed of accessing URL endpoints vs sequential operation.

pee: Sending Input to Multiple Commands Simultaneously

Photo by Victor Barrios on Unsplash

The pee command in moreutils is similar to tee, but instead of writing input to multiple files, it sends the standard input to multiple commands simultaneously. This is particularly useful when we want to process the same input in different ways at the same time.

From the log files we’ve seen earlier we want to simultaneously search for different types of entries, such as “Error”, “Warning”, and “Info”. We can use pee to run multiple grep commands in parallel on the same input.

cat syslog.txt | pee 'grep Error' 'grep Warning' 'grep Info'

• cat syslog.txt: Outputs the contents of syslog.txt.
• | pee 'grep Error' 'grep Warning' 'grep Info': Sends the output to three grep commands simultaneously, each filtering for “Error”, “Warning”, or “Info” entries.

This setup allows us to extract different types of log entries at the same time, rather than running multiple commands one after the other. Each grep command processes the same input stream, but with a different filter.

sponge: Replace File Content In-Place

Photo by Marjan Blan on Unsplash

The sponge command reads the entire input before writing it to a file. This prevents the common problem of overwriting a file while it’s still being read, which can happen when you try to redirect output directly into the same file you’re reading from.

Let’s say we have a file with unsorted data, and you want to sort the contents and save the result back into the same file. Normally, trying to do this directly can cause issues because the file might be overwritten before the sort is completed.

Here’s how you can use sponge to avoid this problem:

sort unsorted.txt | sponge unsorted.txt

• sort unsorted.txt: Sorts the contents of unsorted.txt.
• | sponge unsorted.txt: sponge reads the sorted output fully before writing it back to unsorted.txt, ensuring that the operation is safe and won’t corrupt the file.

ts: Adding Timestamps to Each Line of Input

Photo by Djim Loic on Unsplash

The ts command prepends a timestamp to each line of input. This is particularly useful for logging, monitoring, or debugging, where knowing the exact time of each event is important.

If we have a log file and you want to add timestamps to each entry as they are being written, you can use ts:

tail -f /var/log/syslog | ts

Explanation:

• tail -f /var/log/syslog: Continuously outputs new lines added to the syslog file.
• | ts: Prepends a timestamp to each new line, showing exactly when each log entry was recorded.

This makes it easy to see when each event in the log occurred, which can be crucial for debugging and system monitoring.

vidir: Editing Directories and Files in Vim Buffer

Photo by Maarten van den Heuvel on Unsplash

The vidir command allows us to edit filenames within a directory as if they were lines in a text file. This makes batch renaming of files much more straightforward and faster than using mv repeatedly.

in Neovim plugins like `oil.nvim` or `mini.files` provide much more extended functionally. Learn more here.

Example: Batch Rename Files

Suppose we have a directory full of files and you want to rename them all at once:

vidir .

• vidir .: Opens the filenames in the current directory in your default text editor.
• You can edit the filenames directly in the text editor. Once you save and close the editor, vidir applies the changes, renaming the files accordingly.

This is a quick and efficient way to batch rename files without needing to write a script or manually move each file.

vipe: Editing Pipeline Content with Text Editor

Photo by Jens Freudenau on Unsplash

The vipe command inserts a text editor into a pipeline, allowing us to manually edit the content being passed through the pipeline before sending it to the next command.

Imagine we’re filtering some logs for specific entries and want to make some manual edits before saving the results:

grep "Error" syslog.txt | vipe | tee errors.txt

• grep "Error" syslog.txt: Finds all lines containing “Error” in syslog.txt.
• | vipe: Opens the grep output in your text editor, where you can manually make changes.
• | tee errors.txt: Saves the final, edited output to errors.txt while also displaying it.

This is particularly useful when you need to review or modify the output of a command before it’s passed along to the next step in a pipeline, offering a flexible and interactive way to handle data.

Closing Thoughts

We’ve explored some of the standout tools in moreutils, each offering a unique way to enhance our command-line workflow.

The key here isn’t to overwhelm yourself by trying to use every tool all at once. Instead, focus on integrating one or two that really resonate with your workflow. Whether it’s the convenience of sponge for safe file edits or the versatility of vipe for interactive pipeline editing, start with what feels most useful to you.

As you get comfortable with these tools, you’ll likely find that they become indispensable parts of your daily routine. Just like any great toolset, it’s all about finding what works best for you and making it a natural extension of how you work. Before long, you might wonder how you ever got by without them.

Thanks for taking the time to read this post. I hope you found it interesting and informative.

🔗 Connect with me on LinkedIn

🌐 Visit my Website

📺 Subscribe to my YouTube Channel