This the multi-page printable view of this section.Click here to print.

Return to the regular view of this page.

Shell Scripting

Notes on how to perform tasks within shell scripting, mostly bash, on Linux

Table of Contents

1 - Bash Scripts

Notes on how to perform tasks with Bash

1.1 - Read data to script

Notes on how to read data into a Bash script

1.1.1 - Read file line by line

How to read a file line by line in Bash

Reading a file into a bash script on a per-line basis is pretty simple:

1while read -r line; do command "${line}"; done < src.file

here we read a line from stdin and as long as a line was read the line is passed to command as a single command argument.

the -r option prevents backslash escapes in the line being interpreted.

If you want to prevent leading or trailling whitespace from being removed from the line then add IFS= to the line:

1while IFS= read -r line; do command "${line}"; done < src.file

Within scripts

The above is useful when writing something quick on the command line, but in a script it's more customary to write this across multiple lines.

What I tend to do is wrap the while loop within ( ) with the input redirect after the parentheses. To me this makes it clearer on what the bounds of the script relates to the input.

For example, here we have a script which writes the content of a file to stdout but adds line numbers to the output:

#!/bin/bash input="$1" ( lc=1 while IFS= read -r line do printf "%03d %s\n" ${lc} "${line}" lc=$((lc+1)) done ) < "${input}"

When run against itself:

$ ./ 001 #!/bin/bash 002 input="$1" 003 ( 004 lc=1 005 while IFS= read -r line 006 do 007 printf "%03d %s\n" ${lc} "${line}" 008 lc=$((lc+1)) 009 done 010 ) < "${input}"

2 - Cut

Remove sections from each line of files


cut OPTION... [FILE]...


Print selected parts of lines from each FILE to standard output.

With no FILE, or when FILE is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.

-b, --bytes=LIST
select only these bytes
-c, --characters=LIST
select only these characters
-d, --delimiter=DELIM
use DELIM instead of TAB for field delimiter
-f, --fields=LIST
select only these fields; also print any line that contains no delimiter character, unless the -s option is specified
complement the set of selected bytes, characters or fields
-s, --only-delimited
do not print lines not containing delimiters
use STRING as the output delimiter the default is to use the input delimiter
-z, --zero-terminated
line delimiter is NUL, not newline
display this help and exit
output version information and exit

Use one, and only one of -b, -c or -f. Each LIST is made up of one range, or many ranges separated by commas. Selected input is written in the same order that it is read, and is written exactly once. Each range is one of:

N'th byte, character or field, counted from 1
from N'th byte, character or field, to end of line
from N'th to M'th (included) byte, character or field
from first to M'th (included) byte, character or field

2.1 - Cut last column

How to get the last column using cut

To get the last field in a line using cut is simple, just wrap cut with a pair of rev commands and then extract the first field:

1rev | cut -f1 -d' ' | rev

This works as rev simply reverses the characters of each line, pipe it through cut which extracts the required fields (counting in reverse), and then reverses it again so the content is in the original order.

The only thing you have to remember is that, once reversed, field 1 is actually the last, 2 the one before the last and so on.

3 - Pipes

Pipes and Pipelines

In most script languages, a pipe is where the output of one command is passed as the input of a second command. It allows the chaining of multiple commands without the need of intermediary files.

File handles

Every command has 3 files open by default:

File # Name Script Description
Read Create Append
0stdin < Input into the command
1stdout >&1 >>&1 Output from the command
2stderr >&2 >>&2 Errors from the command

3.1 - Here document

Sending multiple lines into a pipe

A Here document is a set of lines in a script which is to be sent to a command as it's input.

It consists of <<deliminator followed by multiple lines and then a line with just demiminator to show the end of the document.

For example, we have a 4 line document that will be sent to the command as it's standard input:

1command <<TERMINATOR
2line 1
3line 2
4line $someVar
5line 4

See also: Here strings

3.2 - Here strings

Sending a string into a pipe

Here strings are similar to Here documents but allows you to do this on one line:

1command <<<"some string"

For example, the following commands do the same thing, count the number of words in a string:

1echo "This is a test" | wc -w
3wc -w <<<"This is a test"

You can also pipe the output of commands as a here string:

1ps -fe | wc -l
3wc -l <<<$(ps -fe)

See also: Here document