File management

GNU coreutils

  • test – check file types and compare values
  • uniq – report or omit repeated lines
  • dirname – strip last component from file name
  • basename – strip directory and suffix from filenames
  • pathchk – check whether file names are valid or portable
  • dd – convert and copy a file
  • readlink – print resolved symbolic links or canonical file names

ls – list directory contents

-A, –almost-alldo not list implied . and ..
-h, –human-readablewith -l and -s, print sizes like 1K 234M 2G etc.
-k, –kibibytesdefault to 1024-byte blocks for disk usage; used only with -s and per directory totals
-luse a long listing format
-n, –numeric-uid-gidlike -l, but list numeric user and group IDs
-r, –reversereverse order while sorting
-R, –recursivelist subdirectories recursively
-Ssort by file size, largest first
-tsort by time, newest first; see –time
-d, –directorylist directories themselves, not their contents
# ls -rlS

List only directories:

ls -l -d */

cp – copy files and directories

Merge two folders using cp:

cp -rl source/folder destination
rm -r source/folder
-l, --linkhard link files instead of copying

Note that the source folder shouldn’t be in use, otherwise there’s a chance that new files in the source will be deleted without having been linked…

mkdir – make directories

move all files except some

mkdir archived
mv !(archived) archived/

Create several folders at once

mkdir /docs/folder{1..5}

du – estimate file space usage

-h, --human-readableprint sizes in human readable format (e.g., 1K 234M 2G)
-x, --one-file-systemskip directories on different file systems
-d, --max-depth=Nprint the total for a directory (or file, with –all) only if it is N or fewer levels below the command line argument;
–max-depth=0 is the same as –summarize
--apparent-sizeprint apparent sizes, rather than disk usage;
although the apparent size is usually smaller, it may be larger due to holes in (‘sparse’) files,
internal fragmentation, indirect blocks, and the like
du -hxd 1 Documents/

Something important to note on compressed filesystems, du might show a different size for two identical datasets while ls won’t. To avoid this problem, one should use –apparent-size.

du --apparent-size -hxd 1 Documents/

Coupled with sort:

du --apparent-size -hxd1 Documents/archived | sort -hr
du --apparent-size -hxd1 Documents/archived | sort --human-numeric-sort --reverse

cut – remove sections from each line of files

btrfs subvolume list -a /mnt/storage | cut -d' ' -f9

sort – sort lines of text files

btrfs subvolume list -a /mnt/storage | cut -d' ' -f9 | sort

realpath – print the resolved path

leo@computer:~$ realpath todo
me@computer:~$ realpath ../../bin/cd

tee – read from standard input and write to standard output and files

tar – an archiving utility

-x, --extract, --getExtract files from an archive. Arguments are optional. When given, they specify names of the archive members to be extracted.
-v, --verboseVerbosely list files processed.
-f, --file=ARCHIVEUse archive file or device ARCHIVE.
-z, --gzip, --gunzip, --ungzipFilter the archive through gzip(1).
-C, --directory=DIRChange to DIR before performing any operations. This option is order-sensitive, i.e. it affects all options that follow.

Untar shit

tar -xvf DKU_16.4.0.tar
tar -zxvf lnx_drv_celerity8_2220f1.tgz

tail – output the last part of files

head – output the first part of files

head file.txt               # first 10 lines
tail file.txt               # last 10 lines
head -n 20 file.txt         # first 20 lines
tail -n 20 file.txt         # last 20 lines
head -20 file.txt           # first 20 lines
tail -20 file.txt           # last 20 lines
head -n -5 file.txt         # all lines except the 5 last
tail -n +5 file.txt         # all lines except the 4 first, starts at line 5

Others utilities

watch – execute a program periodically, showing output fullscreen

-d, --differences [permanent]Highlight the differences between successive updates.
-n, --interval secondsSpecify update interval.

To watch the contents of a directory change, you could use:

watch -d ls -l-d

Watch open ports variation:

watch -d -n 0.5 ss -tulpn

xargs – build and execute command lines from standard input

-0, --nullInput items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (every character is taken literally). Disables the end of file string, which is treated like any other argument. Useful when input items might contain white space, quote marks, or backslashes. The GNU find -print0 option produces input suitable for this mode.
-I replace-strReplace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separator is the newline character. Implies -x and -L 1.
-P max-procs
Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time.

Removes all BTRFS subvolumes in one go (there’s no recursive switch…)

btrfs subvolume list . | grep timeshift-btrfs | cut -d ' ' -f 9 | xargs -I % btrfs subvolume delete /mnt/KINGSTON_SA1000M8/%

Use several processes

find /path -name '*.foo' | xargs -P 24 -I '{}' /cpu/bound/process '{}' -o '{}'.out

lsof – list open files

lsof +r2 | grep '/some/dir'

ncdu – NCurses Disk Usage