Unix

https://www.chiark.greenend.org.uk/~sgtatham/agedu/

Filesystem and storage

The /dev directory contains files representing each attached device; each device on the system is represented by a file. The devices sda1, sdb, etc. are usually the hard drive or flash drives and their partitions.

Logical labels are used for drives that are mounted on the filesystem. These logical labels vary depending on where the drives are mounted, which means that the same hard drive might have different labels at different times.

Originally, floppy drives were labelled as fd0 and hard drives as hda. Old legacy hard drives that use IDE or E-IDE interfaces are still represented in the form hda. Newer Serial ATA (SATA) interface drives and Small Computer System Interface (SCSI) hard drives are represented as sda. Drives can be split up into sections known as partitions, which are represented in the labeling system with numbers.

When systems have more than one hard drive, the labels are named by incrementing the last letter in alphabetical order. The serial letter after the label is often referred to as the major number.

Device file	Description
sda	First SATA hard drive
sdb	Second SATA hard drive
sdc	Third SATA hard drive
sdn	nth SATA hard drives

Drives can be split into partitions in order to manage and separate information. For example, you may want to separate your hard drive so that your swap file, home directory, and / directory are all on separate partitions. You may want to do this for the ability to share resources with relaxed permissions. Partitions are labelled with a minor number that comes after the drive designation. In this manner, the first partition on the first SATA drive would be sda1 and the second partition would be sda2, and so on.

Partition	Description
sda1	The first partition on the first SATA drive
sda2	The second partition on the first SATA drive
sda3	The third partition on the first SATA drive
sdan	The nth partition on the first SATA drive

To view the partitions, use fdisk -l (the -l switch lists all partitions of all the drives).

Character and block devices

In the /dev directory some of the device files have a "c" or "b" in the first position of the permission string. These letters represent the two ways that devices transfer data in and out.

The "c" stands for character and are character devices. External devices that interact with the system by sending and receiving data character by character, such as mice or keyboards are character devices.

The "b" stands for block devices and they communicate in blocks of data (multiple bytes at a time) and include devices like hard drives and DVD drives. These devices require higher-speed data throughput and therefore send and receive data in blocks (many characters or bytes at a time).

The command lsblk (list block) lists some basic information about each block device listed in /dev. The result is similar to the output from fdisk -l but it will also display devices with multiple partitions in a tree that shows each device with its partitions as branches. This will display information on the mount point of the drive, which is the position the drive was attached to the filesystem.

Mounting and unmounting

A storage device must first be physically connected to the filesystem and then logically attached to the filesystem in order for the data to be made available to the operating system. In other words, even if the device is physically attached to the system, it is not necessarily logically attached and available to the operating system. Storage devices typically automount, automatically become attached to the filesystem, when they are connected.

The term mount is a legacy from the early days of computing when storage tapes (before hard disks) had to be physically mounted to the computer system. The point in the directory tree where devices are attached is known as the mount point. The two main mount points are /mnt and /media. As per convention, devices such as external USB devices and flash drives can be manually mounted at /mnt but then automatically mounted, the /media directory is used (even though any directory can be used).

To mount a drive on the filesystem, use the mount command. The mount point for the device should be an empty directory; if you mount a device on a directory that has subdirectories and files, the mounted device will cover the contents of the directory making them unavailable. To mount a new hard drive sdb1 at the /mnt directory enter the following:

mount /dev/sdb1 /mnt

The filesystems on a system that are mounted at boot-time are kept in a file at /etc/fstab (filesystem table), which is read by the system at every bootup.

To unmount enter the following:

umount /dev/sdb1

Note that a device that is busy, if the system is reading or writing to the device, it cannot be unmounted.

Monitoring

The command df (disk free) provides basic information on any mounted devices.

The command fsck (filesystem check) checks the filesystem for errors and repairs the damage, if possible, or allocates the bad area into a bad blocks table to mark it as bad. To run the fsck command, you need to specify the device file to check. It is important to note that you must unmount the drive before running a filesystem check or else you will receive an error.

umount /dev/sdb1
# -p will automatically repair any problems
fsck -p /dev/sdb1

Logging

Log files store information about events that occur when the operating system and applications are run, including any errors and security alerts. Information will be logged automatically based on a series of rules.

A daemon called syslogd is used to automatically log events and there are several variations of syslog, including rsyslog and syslog-ng that are used on different distributions. The rsyslog config file is /etc/rsyslog.conf. The Rules section of this file is where you can set the rules as to what will be logged.

The rsyslog rules determine:

what kind of information is logged
what programs have their messages logged, and
where that log is stored.

Each line of the rules is a separate logging rule that says what messages are logged and where they are logged to. The basic format for these rules is as follows:

facility.priority            action

The facility keyword references the program (i.e. software), such as mail, kernal, or lpr, whose messages are being logged. The priority keyword determines what kind of messages to log for that program. The action keyword references the location where the log will be sent.

The following are a list of valid codes for facility

auth, authpriv - Security/authorisation messages
cron - clock daemons
daemon - other daemons
kern - kernal message
lpr - printing system
mail - mail system
user - generic user-level messages

An astertisk wildcard (*) in place of a word refers to all facilities and you can select more than one facility by listing them together and comma-separated.

The priority indicates what kinds of messages to log. Codes are listed from lowest priority, starting at debug, to highest priority, ending at panic. If the priority is *, messages of all priorities are logged. When you specify a priority, messages of that priority and higher are logged. Below is the full list of valid codes for priority:

debug
info
notice
warning
warn
error
err
crit
alert
emerg
panic

The codes warn, error, and panic have been deprecated and should not be used. The action is usually a filename and location where the logs should be sent. Note that generally, log files are sent to the /var/log directory with a filename that describes the facility that generated them, such as auth. This means that logs generated by the auth facility would be sent to /var/log.auth.log.

The following will log mail events of all priorities to /var/log/mail:

mail.* /var/log/mail

The following will log kernel events of critical (crit) or higher to /var/log/kernel:

kern.crit /var/log/kernel

The following will log all events of the emergency (emerg) priority to all logged-on users:

*.emerg :omusmsg:*

logrotate

Log rotation is the process of regularly archiving log files by moving them to some other location, leaving you with a fresh log file. That archived location will then get cleaned up after a specified period of time. A cron job is used to schedule logrotate and you can configure the regularity by editing /etc/logrotate.conf.

In /etc/logrotate.conf, the unit of time is first set (e.g. weekly, which is the default). Any number after this will refer to this unit of time. The default is to rotate logs every 4 weeks (rotate 4). A new empty log file is created when old ones are rotated (create) and you can compress the rotated log files (by setting compress).

At the end of each rotation period, the log files are renamed and pushed toward the end of the chain of logs as a new log file is created, replacing the current log file. For example, /var/log.auth will become /var/log.auth.1, then /var/log.auth.2, and so on. If logs are rotated every four weeks and four set of backups are kept, there will be no /var/log.auth.5 as the previous /var/log.auth.4 will be deleted.

See man logrotate for more details.

Processes

The kernel, the inner core of the operating system, assigns a unique process ID (PID) to each process sequentially, as the processes are created. Running the ps command by itself lists the processes invoked by the currently logged-in user and processes that are running on that terminal.

Running the ps command with aux will show all processes running on the system for all users and the processes are displayed in the order they were started (since the kernel assigns PIDs in order).

The nice command is used to influence the priority of a process to the kernel, since the kernel has the final say over process priorities. The command renice can be used to (re)set a job's priority and requires the PID of the process. This can also be done using top by pressing the "r" key and then supplying the PID and a nice value.

The kill command has 64 different kill signals and each performs something slightly different. Below are a list of the most useful and commonly used signals.

Signal name	Number	Description
SIGHUP	1	This is known as the Hangup (HUP) signal. It stops the designated process and restarts it with the same PID
SIGINT	2	This is the Interrupt (INT) signal. It is a weak kill signal that isn't guaranteed to work, but it works in most cases.
SIGQUIT	3	This is known as the core dump. It terminates the process and saves the process information in memory, and then it saves this information in the current working directory to a file named core.
SIGTERM	15	This is the Termination (TERM) signal and is the default kill signal.
SIGKILL	9	This is the absolute kill signal. It forces the process to stop by sending the process's resources to /dev/null.

The killall command can also be used to kill a process but takes the name of the process instead of the PID. Jobs can be killed via top as well; simply press "k" and enter the PID.

To start a job in the background append an ampersand to the end of the command. If you want to move a process running in the background to the foreground use the fg command, which required the PID.

The at command is used to set up the daemon atd, which is useful for scheduling a job to run once at some point in time. The syntax is simply the at command followed by the time to execute the process.

at 7:20pm
at noon

Afterwards enter the command to be executed at the specified time.

File permission

Set default permissions allocated to files and directories using umask (user file-creation mask). The umask method represents the permissions you want to remove from the base permissions on a file or directory to make them more secure.

The umask is a three-digit octal number corresponding to the three permission digits and is used to subtract from the permissions to arrive at a new permission status. When a new file or directory is created, its permissions are set to the default value and then subtracted from the value in umask.

For example if umask is set to 022 (default in most Debian systems) and the default file permission is 666, then the resulting permission will be 644. The default directory permission is 777 and with a umask of 022, the resulting permission is 755.

Each user can set a personal default in their profile.

# find out current umask
umask

# set new umask
umask 007

Special permissions

In addition to the three general-purpose permissions, rwx, there are three special permissions that are slightly more complicated. There special permissions are:

set user ID (SUID)
set group ID (SGID)
sticky bit

Temporary permission can be granted to a user by setting the SUID bit on a file. For example, /etc/shadow can only be accessed by root but in order for users to change their password, they need access to the file. Setting the SUID on a file is not a typical operation but you can use chmod to achieve it.

chmod 4664 filename

SGID also grants temporary elevated permissions but it grants the permissions of the file owner's group, rather than of the file's owner. This means that someone without execute permission can execute a file if the owner belongs to the group that has permission to execute the file. A file with the SUID bit set will show an s in place of an x.

The SGID bit is represented as 2 before the regular permissions, so a new file with 644 permissions would be represented as 2644 and this is done via chmod.

chmod 2644 filename

The sticky bit is a permission bit that you can set on a directory to allow a user to delete or rename files within that directory. However, the sticky bit is a legacy of older Unix systems and modern systems ignore it.

These special permissions can be used to exploit systems through privilege escalation, whereby a regular user gains root or sysadmin privileges.

Networking

The ifconfig command is one of the most basic tools for examining and interacting with active network interfaces. You can use it to query your active network connections by simply entering ifconfig in the terminal.

eth0: flags=4163<UP, Broadcast, RUNNING, MULTICAST> mtu 1500
vinet addr:192.168.181.131 netmask 255.255.255.0
Bcast:192.168.181.255
--snip--
lo Linkencap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
--snip--
wlan0 Link encap:EthernetHWaddr 00:c0:ca:3f:ee:02

Tips

Find all top level directories

# -path pattern - File name matches shell pattern pattern.
# -prune True; if the file is a directory, do not descend into it.
find . -type d -path "*/*" -prune

View older "last" logs

last -2000 -f /var/log/wtmp.1 | less

Default group for newly created files requires the setgid bit on the directory.

# https://serverfault.com/questions/96338/specify-default-group-and-permissions-for-new-files-in-a-certain-directory
mkdir blah
chown dtang.ngs blah
ls -lhd blah
drwxrwxr-x 2 dtang ngs 10 Mar 18 14:59 blah

# set the setgid bit
chmod g+s blah
ls -lhd blah
drwxrwsr-x 2 dtang ngs 10 Mar 18 14:59 blah

touch blah/foo
ls -lh blah
total 0
-rw-rw-r-- 1 dtang ngs 0 Mar 18 15:01 foo

Wrap or fold text; -w tells the width of the text, where 80 is standard. and -s tells to break at spaces, and not in words.

fold -w 80 -s text.txt