Common File Operations in Linux Shell Scripting

Let’s try a few of these things out:reader@ubuntu:~$ cd /var/log/reader@ubuntu:/var/log$ ls -ltotal 3688<SNIPPED>drwxr-xr-x 2 root root 4096 Apr 17 20:22 dist-upgrade-rw-r — r — 1 root root 550975 Aug 18 13:35 dpkg.

log-rw-r — r — 1 root root 32160 Aug 11 10:15 faillog<SNIPPED>-rw — — — — 1 root root 64320 Aug 11 10:15 tallylog<SNIPPED>reader@ubuntu:/var/log$ cp dpkg.

log /home/reader/reader@ubuntu:/var/log$ ls -l /home/reader/total 552-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 testdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 testfile-rwxr-xr — 1 reader reader 0 Aug 18 14:00 testfilecopydrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:/var/log$ cp tallylog /home/reader/cp: cannot open ‘tallylog’ for reading: Permission deniedreader@ubuntu:/var/log$So, what happened?.We used cd to change the directory to /var/log/.

We listed the files there using ls with the long option.

We copied a file with a relative path that we were able to read, but that was owned by root:root, to the fully qualified /home/reader/ directory.

When we listed /home/reader/ with the fully qualified path, we saw that the copied file was now owned by reader:reader.

When we tried to do the same for the tallylog file, we got the error cannot open ‘tallylog’ for reading: Permission denied.

This should not be unexpected, since we do not have any read permissions on that file, so copying would be hard.

This should answer two of the three questions.

But what about directories?.Let’s try to copy the /tmp/ directory into our home directory:reader@ubuntu:/var/log$ cdreader@ubuntu:~$ cp /tmp/ .

cp: -r not specified; omitting directory ‘/tmp/’reader@ubuntu:~$ ls -ltotal 552-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 testdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 testfile-rwxr-xr — 1 reader reader 0 Aug 18 14:00 testfilecopydrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$ cp -r /tmp/ .

cp: cannot access ‘/tmp/systemd-private-72bcf47b69464914b021b421d5999bbe-systemd-timesyncd.

service-LeF05x’: Permission deniedcp: cannot access ‘/tmp/systemd-private-72bcf47b69464914b021b421d5999bbe-systemd-resolved.

service-ApdzhW’: Permission deniedreader@ubuntu:~$ ls -ltotal 556-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 testdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 testfile-rwxr-xr — 1 reader reader 0 Aug 18 14:00 testfilecopydrwxrwxr-t 9 reader reader 4096 Aug 18 14:38 tmpdrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$For such a simple exercise, a lot actually happened!.First, we navigate back to our home directory using cd without any arguments; a neat little trick in itself.

Next, we try to copy the entire /tmp/ directory to .

(which is shorthand for current directory).

However, this fails with the error -r not specified; omitting directory ‘/tmp/’.

We list the directory to check this, and indeed, it seems like nothing happened.

When we add the -r, as specified by the error, and retry the command, we get some Permission denied errors.

This is not unexpected, since not all files inside the /tmp/ directory will be readable to us.

Even though we got the errors, when we now check the contents of our home directory, we can see the tmp directory there!.So, using the -r option, which is short for –recursive, allows us to copy directories and everything that’s in them.

RemovingAfter copying some files and directories into our home directory (which is a safe bet, because we know for sure that we can write there!), we’re left with a little mess.

Instead of only creating files, let’s use the rm command to remove some duplicate items:reader@ubuntu:~$ ls -ltotal 556-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 testdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 testfile-rwxr-xr — 1 reader reader 0 Aug 18 14:00 testfilecopydrwxrwxr-t 9 reader reader 4096 Aug 18 14:38 tmpdrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$ rm testfilecopyreader@ubuntu:~$ rm tmp/rm: cannot remove ‘tmp/’: Is a directoryreader@ubuntu:~$ rm -r tmp/reader@ubuntu:~$ ls -ltotal 552-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 testdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 testfiledrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$Using rm followed by a filename deletes it.

As you might notice, there is no Are you sure?.prompt.

This can actually be enabled by using the -i flag, but by default this is not the case.

Consider that rm also allows you to use wildcards, such as * (which matches everything), which will delete every file that is matched (and can be deleted by the user).

In short, this is a great way to lose your files really quickly!When we tried to use rm with the name of a directory, however, it gave the error cannot remove ‘tmp/’: Is a directory.

This is very similar to the cp command, and luckily for us, the remediation is also the same: add -r for a recursive delete!.Again, this is a great way to lose files; a single command lets you delete your entire home directory and everything in it, without so much as a warning.

Consider this your warning!.Especially when using in combination with the -f flag, which is short for –force, which will ensure that rm never prompts and starts deleting right away.

Renaming, moving, and linkingSometimes, we do not just want to create or delete a file, we might need to rename one.

Weirdly, Linux does not have anything that sounds like rename; however, the mv command (for move) does accomplish the functionality that we want.

Similar to the cp command, it takes a source file and destination file as arguments, and looks like this:reader@ubuntu:~$ ls -ltotal 552-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 testdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 testfiledrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$ mv testfile renamedtestfilereader@ubuntu:~$ mv testdir/ renamedtestdirreader@ubuntu:~$ ls -ltotal 552-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 renamedtestdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 renamedtestfiledrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$As you can see, the mv command is really simple to use.

It even works for directories, without needing a special option such as the -r we saw for cp and rm.

It does, however, get a little more complex when we introduce wildcards, but don’t worry about that for now.

The commands we used in the preceding code are relative, but they work just as well fully qualified or mixed.

Sometimes, you’ll want to move a file from one directory into another.

If you think about it, this is actually a rename of the fully qualified file name!.No data is being touched, but you just want to reach the file somewhere else.

So, using mv umaskfile umaskdir/ will move the umaskfile into umaskdir/:reader@ubuntu:~$ ls -ltotal 16-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 renamedtestdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 renamedtestfiledrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$ mv umaskfile umaskdir/reader@ubuntu:~$ ls -ltotal 16-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 renamedtestdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 renamedtestfiledrwxrwx — — 2 reader reader 4096 Aug 19 10:37 umaskdirreader@ubuntu:~$ ls -l umaskdir/total 0-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$Finally, we have the ln command, which stands for linking.

This is the Linux way of creating links between files, which are closest to the shortcuts that Windows uses.

There are two types of links: symbolic links (also called soft links) and hard links.

The difference is found deeper in the filesystem workings: a symbolic link refers to the filename (or directory name), whereas a hard link links to inode that stores the contents of the file or directory.

For scripting, if you’re using links, you’re probably using symbolic links, so let’s see those in action:reader@ubuntu:~$ ls -ltotal 552-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

log-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 renamedtestdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 renamedtestfiledrwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$ ln -s /var/log/auth.

logreader@ubuntu:~$ ln -s /var/log/auth.

log link-to-auth.

logreader@ubuntu:~$ ln -s /tmp/reader@ubuntu:~$ ln -s /tmp/ link-to-tmpreader@ubuntu:~$ ls -ltotal 552lrwxrwxrwx 1 reader reader 17 Aug 18 15:07 auth.

log -> /var/log/auth.

log-rw-r — r — 1 reader reader 550975 Aug 18 14:20 dpkg.

loglrwxrwxrwx 1 reader reader 17 Aug 18 15:08 link-to-auth.

log -> /var/log/auth.

loglrwxrwxrwx 1 reader reader 5 Aug 18 15:08 link-to-tmp -> /tmp/-rw-rw-r — 1 reader reader 69 Jul 14 13:18 nanofile.

txtdrwxrwxr-x 2 reader reader 4096 Aug 4 16:16 renamedtestdir-rwxr-xr — 1 reader reader 0 Aug 4 13:44 renamedtestfilelrwxrwxrwx 1 reader reader 5 Aug 18 15:08 tmp -> /tmp/drwxrwx — — 2 reader reader 4096 Aug 4 16:18 umaskdir-rw-rw — — 1 reader games 0 Aug 4 16:18 umaskfilereader@ubuntu:~$We created two types of symbolic link using ln -s (which is short for –symbolic): to the /var/log/auth.

log file first, and to the /tmp/ directory after.

We are seeing two different ways of using ln -s: without a second argument, it creates the link with the same name as the thing we’re linking to; otherwise, we can give our own name for the link as the second argument (as can be seen with the link-to-auth.

log and link-to-tmp/ links).

We can now read the contents of /var/log/auth.

log by either interacting with /home/reader/auth.

log or /home/reader/link-to-auth.

log.

If we want to navigate to /tmp/, we can now use either /home/reader/tmp/ or /home/reader/link-to-tmp/ in combination with cd.

While this example isn’t particularly useful in your day-to-day work (unless typing /var/log/auth.

log instead of auth.

log saves you tons of time), linking prevents duplicate copies of files while maintaining easy access.

NoteAn important concept in linking (and Linux filesystems in general) is the inode.

Every file (whatever the type, so including directories) has an inode, which describes the attributes and disk block locations of that file.

In this context, attributes include things like ownership and permissions, as well as last change, access and modification timestamps.

In linking, soft links have their own inodes, while hard links refer to the same inode.

If you found this article interesting and helpful, you can explore Learn Linux Shell Scripting — Fundamentals of Bash 4.

4 to create and maintain powerful Bash scripts for automation and administration.

Following an approach that takes you through the fundamentals of Linux, Bash, and shell scripting through example and exercise-driven learning, Learn Linux Shell Scripting — Fundamentals of Bash 4.

4 is a must-read for new and existing system administrators.

.. More details

Leave a Reply