teach pipes: the tutorial covered redirection but never chaining commands

- Navigate the DAIC filesystem confidently

- Create, copy, move, and delete files and directories

- Redirect command output (stdout/stderr) to files

- Chain commands together with pipes

- Find files and search their contents

- Write simple shell scripts to automate tasks

The `2>&1` syntax means "send stderr (2) to wherever stdout (1) is going." The `&>` shorthand is a bash extension: it is not POSIX, so use `> file 2>&1` in scripts that must run under `sh`.

To send output to another command rather than to a file, use a pipe (see [Combining commands with pipes](#combining-commands-with-pipes)).

**Discard output entirely:**

```shell-session

$ mycommand > /dev/null 2>&1      # Discard everything

$ grep -l "import" src/*.py        # Just show filenames

```

### Combining commands with pipes

Redirection sends a command's output to a file. A pipe (`|`) sends it to another command instead, which reads it as input:

```shell-session

$ ls src | wc -l

2

```

`ls src` produced a list of filenames and, instead of printing them, the shell handed them to `wc -l`, which counted the lines. No temporary file was involved.

Pipes chain as far as you need, with each stage reading the previous stage's stdout:

```shell-session

$ grep -r "epochs" src/ | wc -l

8

$ grep -r "epochs" src/ | grep -v "range" | wc -l

6

$ find . -name "*.py" | sort | head -n 2

./src/evaluate.py

./src/train.py

```

The `-v` option inverts a match, so the second command counts the lines that contain "epochs" but not "range".

**A pipe carries stdout only.** Errors still go to your terminal:

```shell-session

$ grep -r "import" src/ nonexistent/ | wc -l

grep: nonexistent/: No such file or directory

2

```

The error message bypassed `wc` and appeared on screen; only the two matching lines were counted. To send stderr through the pipe as well, redirect it into stdout first:

```shell-session

$ grep -r "import" src/ nonexistent/ 2>&1 | wc -l

3

```

Now the error counts as a line like any other.

{{% alert title="Why this matters for HPC" color="info" %}}

Cluster commands print more than you want to read. Pipes let you narrow the output on the spot - counting your queued jobs, or filtering a long node list down to the ones you care about - without writing anything to disk. See the [Slurm tutorial](/tutorials/slurm/) for the commands worth filtering.

{{% /alert %}}

### Exercise 4: Find and search

1. Find all files modified in the last day:

   $ find . -type d -name "data"

   ```

4. Count how many Python files your project contains:

   ```shell-session

   $ find . -name "*.py" | wc -l

   ```

{{% alert title="Check your work" color="info" %}}

The `find . -mtime -1` command should list files you recently created. The `grep -n` command shows line numbers where "print" appears. The directory search should show `./data` (and any other data directories you created).

The `find . -mtime -1` command should list files you recently created. The `grep -n` command shows line numbers where "print" appears. The directory search should show `./data` (and any other data directories you created). The count should be `2`: `train.py` and `evaluate.py`.

{{% /alert %}}

## Part 7: Automating with scripts

| Delete | `rm file` or `rm -r dir` |

| Find files | `find . -name "*.py"` |

| Search contents | `grep "pattern" file` |

| Pipe into another command | `command \| wc -l` |

| Make script executable | `chmod +x script.sh` |

## What's next?