Linux Shell Skills

Readings

The readings below are assigned through Canvas:

1. Linux Shell Skills from Prof. Shedden’s 2016 Course notes

2. A tmux Primer by Daniel Meissler

3. Statistics and Computation Service by UM ITS

Terminals

If you are using a computer running a Mac or Linux OS you have access to a Linux terminal installed on your computer as Terminal.

If you are using a computer running a Windows OS, you will need to install a terminal or shell program capable of acting as such. Most people use Putty but Powershell may also be an option.

Experienced users may be interested in zsh but we will not make use of it for this class.

Connecting to UM Machines

  ssh uniqname@login.itd.umich.edu
  ssh uniqname@scs.dsc.umich.edu
  ssh uniqname@mario.dsc.umich.edu
  ssh uniqname@luigi.dsc.umich.edu

File system

In Linux essentially everything is a file: this includes program executables, system configurations, as well as your own data and source files.

Files are organized hierarchically into directories beginning with the root directory /. Directories can contain files and sub-directories with locations in the directory hierarchy separated by a /. This collection of directories and files is called a file tree.

Use the following commands to navigate and interact with the file tree:

• ls (list files), ls -a, ls -l
• cd (change directories)
• pwd (print the current or working directory)
• mkdir (make directory), mkdir -p
• rmdir (remove directory)
• rm (remove a file), rm -r
• mv Move a file or directory
• find Find a file.

In working with files, it is helpful to know:

• . refers to the current directory,
• .. refers to the parent directory, one step up the file tree.
• cd invoked with no arguments will return you to your home directory.
• Configuration files and others used by programs are often named as hidden files. The names of hidden files begin with a .. To see these files, use ls -a.
• Use filename wildcards to refer to groups of files matching specific patterns:
  • * matches any sequence of characters
  • ? matches any single character.

Text Editors

In order to edit files in the shell, you will need to use a text editor. Some popular choices are:

• emacs
• vi / vim
• nano

You can find links to tutorials in Prof Shedden’s notes assigned as reading. I personally use emacs and that will be the editor I use in examples presented to the class.

If you do not already have a preferred text editor, please pick one to learn for the course, find a tutorial on it, and work through that tutorial.

Terminal Multiplexers

A terminal multiplexer allows you to invoke multiple shells from the same terminal connection and to keep these sessions running after you log off. The two most common are screen and tmux, with the latter being the preferred option for this course.

When using a terminal multiplexer with a networked file system such as AFS, be aware that your credentials or “ticket” for accessing the networked files will typically expire after a fixed amount of time (e.g. 24 hours). You can renew this ticket for a fixed amount of time using kinit:

kinit -4d
aklog

Transferring data

There are many ways to transfer data to a remote server using the shell. Three common ways to do this from the command line are:

+scp to copy to/from your local computer,

+wget to download directly from the web,

+sftp or ‘secure file transfer protocol’ for transferring large volumes of data.

To transfer a single smallish file from the working directory on your local machine to your AFS space:

scp ./local_file.ext uniqname@login.itd.umich.edu:~/remote_directory/

To transfer a file from the remote directory to your local computer reverse the arguments:

scp uniqname@login.itd.umich.edu:~/remote_directory/remote_file.ext ./

For larger transfers you should use sftp for efficiency. Transfer data using the login pool to avoid adding strain to the computation servers.

To download data directly from a website to a remote server use a web browser to find the URL to the file and use wget:

wget https://remote.url.edu/path/to/file/data.txt

Make sure you only download from trusted sources!

Use sftp for interactive sessions, read more using man sftp on one of the University linux servers.

Pipes and file redirections

We will review some of the examples from sections 2.3.4 and 2.3.5 of Data Science at the Command Line. You may wish to read all of section 2.3.

Compression and archiving

Large files often contain redundant data and can be stored using less space on disk in a compressed format. Depending on the system and the file, compression can make reading from or writing to a file more efficient as reading the bits off disk is an “I/O-bound” task while decoding/decompressing is a “CPU-bound” task. This is particularly useful on shared systems with I/O bottlenecks.

gzip file.txt

gunzip file.gz

gzip -d file.gz

gunzip -c file.gz > file.txt

tar cvfz name.tgz ./parent_folder

tar xvfz name.tgz

Other common tools

You will likley find the following command line tools useful:

• head - read the first n lines of a file

• tail - read the last n lines of a file

• wc - count words or use wc -l to count lines

• grep - find lines in files that match string patterns

• sort - sort a file on one or more fields

• cut - extract select columns from a delimited file

• paste - concatenate files line by line

• join - merge two files based on a common field.

• nl - number the lines in a file.

We will look at examples in class as time permits. You may also wish to read chapter 5 of Data Science at the Command Line.

Shell scripting

A shell script is a program constructed from shell commands. You can view an example here.

For more on shell scripting, see Chapter 4 from Data Science at the Command Line by Jeroen Janssens.

Statistics 506, Fall 2019 Homepage