Linux systems expose a huge amount of information as plain text: logs, configuration files, command output, and status reports. That means DevOps engineers do not just read text; they filter it, search it, transform it, count it, and pipe it into other tools.
The commands in this lesson turn the shell into a lightweight data-processing environment.
grepgrep searches for lines that match a pattern.
grep error app.log
grep -i failed app.log
grep -n timeout app.log
grep -r nginx /etcUseful flags:
-i ignores case-n shows line numbers-r searches recursivelygrep supports regular expressions, which let you search by pattern instead of fixed text.
grep '^ERROR' app.log
grep '200$' access.log
grep '[0-9][0-9][0-9]' access.logExamples:
^ERROR means lines starting with ERROR200$ means lines ending with 200[0-9] matches a digitTip: Start simple with fixed text, then add regex only when needed.
cutcut extracts parts of each line.
cut -d: -f1 /etc/passwd
cut -d' ' -f1 access.logHere:
-d sets the delimiter-f chooses the field numberFor /etc/passwd, fields are separated by :. The first field is the username.
sort orders lines. uniq removes adjacent duplicate lines.
sort names.txt
sort access.log | uniq
sort access.log | uniq -cuniq -c also counts repeated entries.
This is helpful when summarizing logs or finding repeated values.
wcwc stands for word count, but it can count more than words.
wc app.log
wc -l app.log
wc -w notes.txt
wc -c payload.jsonUseful flags:
-l counts lines-w counts words-c counts bytesExample use case:
grep ERROR app.log | wc -lThis counts how many error lines appear in the log.
awkawk is a powerful text-processing language. For beginners, think of it as a tool for working with columns and applying logic.
awk '{print $1}' access.log
awk '/ERROR/ {print $0}' app.log
awk -F: '{print $1, $7}' /etc/passwdKey ideas:
$1, $2, $3 refer to fields-F: sets the field separator to :/pattern/ { action } runs the action only on matching linesA realistic example:
awk '$9 == 500 {print $1, $7}' access.logIn a typical web log, this might print client IPs and paths for HTTP 500 responses.
sedsed is a stream editor. It is often used for quick substitutions.
sed 's/http:/https:/g' urls.txt
sed 's/DEBUG/INFO/g' app.logThe s/old/new/g pattern means substitute all matches on each line.
You can modify a file directly with -i.
sed -i 's/listen 80/listen 8080/' nginx.confBe careful with in-place editing, especially on production configs.
Note: It is often smart to copy a config file before running
sed -ion it.
A pipe sends the output of one command into another command.
cat access.log | grep 500 | wc -lThis works, but you can usually skip cat and write:
grep 500 access.log | wc -lRedirection sends output to files.
grep ERROR app.log > errors.txt
grep WARN app.log >> errors.txt
command 2> error-output.txt> overwrites a file>> appends to a file2> redirects standard errorSuppose you want to find the most common failing endpoint from a web log.
awk '$9 >= 500 {print $7}' access.log | sort | uniq -c | sort -nrThis pipeline does four things:
That is a strong example of why text processing matters. You are turning raw logs into actionable insight with one command chain.
grep to find matching linescut for simple field extractionawk for field logic and patternssed for substitutions and quick editssort, uniq, and wc for summarizing outputYou do not need to master every advanced flag today. Learn the core use cases first, then build from there.
Question: If you want only log lines that begin with ERROR, which grep pattern from the lesson should you use?
<MCQExercise exercise={{
id: 'linux-text-processing-q1',
title: 'Exercise 1: Anchoring a grep search',
description: 'Pick the grep pattern that matches lines starting with ERROR.',
options: [
{ id: 'a', text: 'grep \'200$\' app.log' },
{ id: 'b', text: 'grep \'[0-9][0-9][0-9]\' app.log' },
{ id: 'c', text: 'grep \'^ERROR\' app.log' },
{ id: 'd', text: 'grep -r /etc app.log' },
],
correctId: 'c',
explanation: 'The caret anchor means start of line, so ^ERROR finds lines whose first characters are ERROR. That is exactly the example the tutorial used to introduce simple regular expressions in grep.',
wrongExplanations: {
a: '200$ matches lines ending with 200, not lines beginning with ERROR.',
b: 'This pattern finds any three digits in a row. It does not require the line to start with ERROR.',
d: 'The -r flag controls recursive searching, but this command does not express the needed start-of-line pattern at all.',
},
}} />
Question: What does cut -d: -f1 /etc/passwd return?
<MCQExercise exercise={{
id: 'linux-text-processing-q2',
title: 'Exercise 2: Splitting fields with cut',
description: 'Determine what cut prints when it uses : as the delimiter and field 1.',
options: [
{ id: 'a', text: 'The first field from each colon-separated line, which is usually the username.' },
{ id: 'b', text: 'Only the first colon character found in the whole file.' },
{ id: 'c', text: 'The final field on each line, such as the login shell.' },
{ id: 'd', text: 'The file permissions for /etc/passwd.' },
],
correctId: 'a',
explanation: 'The lesson explains that -d sets the delimiter and -f selects the field number. In /etc/passwd, field 1 is the username, so this command is a simple way to list account names.',
wrongExplanations: {
b: 'cut operates line by line and prints fields, not individual delimiter characters.',
c: 'Field 1 is the first field, not the last one. The shell path is farther to the right in each passwd entry.',
d: 'Permissions come from commands like ls -l. cut just slices text fields from each line.',
},
}} />
Scenario: You want to know which HTTP path appears most often among 500-level responses.
Question: What is the main job of sort | uniq -c in the tutorial pipeline after awk prints the request path?
<MCQExercise exercise={{
id: 'linux-text-processing-q3',
title: 'Exercise 3: Counting repeated values',
description: 'Choose why sort and uniq -c are paired in the error-analysis pipeline.',
options: [
{ id: 'a', text: 'They change every path into uppercase text before printing it.' },
{ id: 'b', text: 'They install a log parser package that understands HTTP status codes.' },
{ id: 'c', text: 'They group identical paths together and count how often each one appears.' },
{ id: 'd', text: 'They remove all duplicate lines without preserving any count information.' },
],
correctId: 'c',
explanation: 'The tutorial shows sort followed by uniq -c because uniq only counts adjacent duplicates. Sorting groups identical paths together first, and uniq -c then reports how many times each failing endpoint appeared.',
wrongExplanations: {
a: 'Neither sort nor uniq changes case here. They organize and count lines.',
b: 'These are standard shell tools, not package installation commands.',
d: 'uniq -c does preserve counts. The count is the point of using it in that pipeline.',
},
}} />
grep finds lines, sed edits them, awk works with fields. Choose by what you need, not habit.grep -r "text" . for a codebase and grep -c when you only need the count.awk '{print $3}' beats cut on whitespace, because it treats runs of spaces as one separator.sed -i edits in place: and BSD sed on macOS requires an argument, so sed -i '' there and sed -i on Linux.sort | uniq -c | sort -rn is the log-analysis idiom worth memorising.uniq only collapses adjacent duplicates, so it is useless without sorting first.2>&1 merges stderr into stdout; without it, piping silently drops your error messages.sed -i 's/old/new/g' file.txt, -i edits in place, s substitutes, and g replaces every occurrence on each line rather than just the first. There is one portability trap: GNU sed on Linux accepts bare -i, while BSD sed on macOS requires a backup suffix, so the same command needs sed -i '' 's/old/new/g' there. Scripts that must run on both should use perl -pi -e 's/old/new/g', which behaves identically everywhere.
cut -d' ' -f3 treats every single delimiter as a boundary, so two consecutive spaces produce an empty field, which makes it unreliable on command output aligned with variable whitespace. awk '{print $3}' treats any run of whitespace as one separator by default, which is almost always what you want for ps, df or log output. cut is the better tool for genuinely delimited data such as CSV or /etc/passwd, where the delimiter is exactly one character.
sort file | uniq -c | sort -rn | head. The first sort is mandatory, because uniq only collapses adjacent duplicates and silently does nothing otherwise, that is the most common mistake with it. uniq -c prefixes each line with its count, and sort -rn orders by that count descending. Combined with awk to pull out a field, it answers most log questions: awk '{print $1}' access.log | sort | uniq -c | sort -rn | head gives your top client IPs.
grep -rn "pattern" ., r recurses, n adds line numbers so the output is actionable. Add -i for case-insensitive and --include="*.py" to limit by file type. grep -rl lists only the matching filenames, which is what you want when feeding results into another command. For large repositories, ripgrep (rg) is considerably faster and respects .gitignore by default, which usually removes more noise than any set of --exclude flags.
Because a pipe carries stdout only, and errors go to stderr. command | grep error therefore filters the normal output and lets the actual error print straight to your terminal, unfiltered. Redirect stderr into stdout first with command 2>&1 | grep error. The order matters: 2>&1 >file and >file 2>&1 do different things, because redirections are processed left to right. The second is the one that sends both to the file.
Understand Linux permissions: rwx for user, group and other, octal modes like 755 and 644, chmod versus chown, setuid, the sticky bit and umask defaults.
Manage Linux accounts safely: useradd versus adduser, why usermod -aG needs a new login, the destructive -G mistake, sudo versus su and the account files.
Find, watch and stop Linux processes: use ps and top, send SIGTERM before SIGKILL, spot zombie states, read load average and find which process owns a port.