05 - Docker Advanced Usage - Docker File System Internals
we're going to learn about the Docker file system. Even if you're good at finding config and system files on a regular operating system, tracking things down on a virtualized system can be a challenge.
Let's see how Docker manages things. Docker files as with other virtualization systems are usually owned by Root.
How does Docker mange files?
Therefore to view these files, we're going to have to have root privileges. "Sudo su" is a way to ensure that all the subsequent commands in this shell will be run as Root. Let's go to /var/lib/docker, which is where all the Docker files both system files and conflict files live.
Type "ls" for list to list the files and directories in this directory. And the first one that might catch your attention is repositories-aufs. Let's read the contents of repositories-aufs. Instantly we can see that it's going to be very difficult to read this file. Simply using "cat" to print the contents to the screen is not going to come out in any format that's usable. Therefore, we should "cat" but then type the output of "cat" to a Python program -mjson.tool, which will usefully reformat the contents of the file. Now we see this is a list of image repositories. It's obvious that this can, under certain circumstances, be quite useful.
However, let's move on. What else is in this directory? Let's go to the directory. Containers list its contents and there seem to be a rather large number of sub-directories. Each one of these is entitled with a long and unattractive hex code that we'll just "cd" that is change directory into one of these. And here we'll discover that tab complete is our best friend. Just type the first couple of letters or characters of a title then hit "tab" and if there's an entry in this directory that uniquely begins with these characters the system will complete for us. Enter "ls" to list the contents of this directory and we see a number of files.
Let's take a look at config.json but again using the Python json.tool to format it a little better. It's a little longer but we have to scroll up again to see everything that was there. But we see this file contains many of the environment variables of this image. When Docker starts up an image, it will read this file and assign these variables to the instance that we will be running. So for instance, the path which will tell the system where to find certain programs that can be invoked from the command line is defined here. The host name is defined here. Many of the interfaces are defined.
Let's go back up two directory levels back to the Docker directory and take a look at the aufs directory. "Ls," the interesting directory name here is diff. We'll see why in just a minute it was called diff but let's move into that directory. "Ls" to list all the contents and here is a really depressingly long list of hex really inexplicable and unreadable hex code titles. Each one of course because of its color you can see is a directory all its own. So let's quickly "cd" into the first of them 029 and then hit tab for tab complete. "Ls" to list and this looks familiar. This looks like a regular file system root directory. There is etc and home and lib and var just as you would find in any UNIX based root system.
However, it turns out this isn't really the operating system root. If we take a look let's say at etc, we'll see a list of many of the normal files that you would find. So let's go to network for instance and there is a directory if-up.d, which is a normal directory to find in the network etc/network directory. But the files while they look as though they are working system files in fact aren't and the clue is in the name of the master directory the higher level directory diff.
Diff you may already know is a UNIX or Linux command line tool that when followed by the names of two files will output the differences between these two files.
Docker images snapshots
Let's go back up three levels. And once again take a look at all of these directories but this time using "ls -l," which will provide us with a long list. "L" is for long. That is a list of all the same directories but with their owner's Root and Root. This is owned by Root and the group called Root is the group that it belongs to. Its file size is a directory that usually you would have a file size of 4,096 bytes. Its date of creation, August 21. And the time of creation 18:16. Note how close together in time many of these directories are. The one was created 18:16 at 6:16 in the evening. One at 18:06, one at 18:19, one at 18:16 presumably some a few seconds earlier or later.
Where do these come from? I didn't create separate images a couple of minutes apart through the evening on the 21st nor the afternoon on the 21st or the 24th or at any other time. I wasn't that busy. What are these directories? Turns out these are snapshots of images at various points in their life cycle and they contain references to all the key system files that might have been in use while they were active. The files in the directory diff allow you to compare the state of system files from one image to another image or I should say even better from an image at one point to an image at a later point.
Let's demonstrate. Diff usually works by typing "diff" and then the name and location of one file and then the name and location of a different file. And hitting "Enter" diff will show you the differences between these two files. These two files actually don't exist. I am going to paste a command, which is obviously going to be a little bit longer just because of the file locations we're working with but this is diff. We're comparing the file found in var/lib/docker/aufs/diff and then this image or this snapshot inside it the file var/log/d message and I'm comparing that to the d message file, the contents of the d message file in a different image. Let's see what happens. In fact, it seems there are no differences. Most likely because there were no entries in the d message log but you can see how it works and how it could be extremely useful to have this tool at your disposal.
Let's see how Docker manages things. Docker files as with other virtualization systems are usually owned by Root.
How does Docker mange files?
Therefore to view these files, we're going to have to have root privileges. "Sudo su" is a way to ensure that all the subsequent commands in this shell will be run as Root. Let's go to /var/lib/docker, which is where all the Docker files both system files and conflict files live.
Type "ls" for list to list the files and directories in this directory. And the first one that might catch your attention is repositories-aufs. Let's read the contents of repositories-aufs. Instantly we can see that it's going to be very difficult to read this file. Simply using "cat" to print the contents to the screen is not going to come out in any format that's usable. Therefore, we should "cat" but then type the output of "cat" to a Python program -mjson.tool, which will usefully reformat the contents of the file. Now we see this is a list of image repositories. It's obvious that this can, under certain circumstances, be quite useful.
However, let's move on. What else is in this directory? Let's go to the directory. Containers list its contents and there seem to be a rather large number of sub-directories. Each one of these is entitled with a long and unattractive hex code that we'll just "cd" that is change directory into one of these. And here we'll discover that tab complete is our best friend. Just type the first couple of letters or characters of a title then hit "tab" and if there's an entry in this directory that uniquely begins with these characters the system will complete for us. Enter "ls" to list the contents of this directory and we see a number of files.
Let's take a look at config.json but again using the Python json.tool to format it a little better. It's a little longer but we have to scroll up again to see everything that was there. But we see this file contains many of the environment variables of this image. When Docker starts up an image, it will read this file and assign these variables to the instance that we will be running. So for instance, the path which will tell the system where to find certain programs that can be invoked from the command line is defined here. The host name is defined here. Many of the interfaces are defined.
Let's go back up two directory levels back to the Docker directory and take a look at the aufs directory. "Ls," the interesting directory name here is diff. We'll see why in just a minute it was called diff but let's move into that directory. "Ls" to list all the contents and here is a really depressingly long list of hex really inexplicable and unreadable hex code titles. Each one of course because of its color you can see is a directory all its own. So let's quickly "cd" into the first of them 029 and then hit tab for tab complete. "Ls" to list and this looks familiar. This looks like a regular file system root directory. There is etc and home and lib and var just as you would find in any UNIX based root system.
However, it turns out this isn't really the operating system root. If we take a look let's say at etc, we'll see a list of many of the normal files that you would find. So let's go to network for instance and there is a directory if-up.d, which is a normal directory to find in the network etc/network directory. But the files while they look as though they are working system files in fact aren't and the clue is in the name of the master directory the higher level directory diff.
Diff you may already know is a UNIX or Linux command line tool that when followed by the names of two files will output the differences between these two files.
Docker images snapshots
Let's go back up three levels. And once again take a look at all of these directories but this time using "ls -l," which will provide us with a long list. "L" is for long. That is a list of all the same directories but with their owner's Root and Root. This is owned by Root and the group called Root is the group that it belongs to. Its file size is a directory that usually you would have a file size of 4,096 bytes. Its date of creation, August 21. And the time of creation 18:16. Note how close together in time many of these directories are. The one was created 18:16 at 6:16 in the evening. One at 18:06, one at 18:19, one at 18:16 presumably some a few seconds earlier or later.
Where do these come from? I didn't create separate images a couple of minutes apart through the evening on the 21st nor the afternoon on the 21st or the 24th or at any other time. I wasn't that busy. What are these directories? Turns out these are snapshots of images at various points in their life cycle and they contain references to all the key system files that might have been in use while they were active. The files in the directory diff allow you to compare the state of system files from one image to another image or I should say even better from an image at one point to an image at a later point.
Let's demonstrate. Diff usually works by typing "diff" and then the name and location of one file and then the name and location of a different file. And hitting "Enter" diff will show you the differences between these two files. These two files actually don't exist. I am going to paste a command, which is obviously going to be a little bit longer just because of the file locations we're working with but this is diff. We're comparing the file found in var/lib/docker/aufs/diff and then this image or this snapshot inside it the file var/log/d message and I'm comparing that to the d message file, the contents of the d message file in a different image. Let's see what happens. In fact, it seems there are no differences. Most likely because there were no entries in the d message log but you can see how it works and how it could be extremely useful to have this tool at your disposal.
Comments
Post a Comment