Pages

Saturday, June 27, 2015

Extending Storage on an Fedora Atomic Host

I had to spend some time understanding how to use docker-storage-setup on an Atomic host. The tool docker-storage-setup comes by default and makes the configuration of storage on your Atomic host easier. I didn't read any of the provided documentation (although that probably would have helped) other than the script itself.  So, pardon me if this is a duplicate of other info out there.  It was a great way to learn more about it.  The goal here is to add more disk space to an Atomic host.  By default, the cloud image that you download has one device (vda) that is 6GB in size.  When I'm testing many, many docker builds and iterating through the Fedora-Dockerfiles repo, that's just not enough space.  So, I need to know how to expand it.

To provide some context about my environment, I'm using a local KVM environment to hack around in.  The first thing I'll do is go ahead and add a few extra disks to my environment so I can do some testing of docker-storage-setup.  Here is what we will be modifying on our running Atomic VM:

My VM is called: atomic1
New disk 1: vdb (logical name presented to VM)
New disk 2: vdc (logical name presented to VM)
New disk 3: vdd (logical name presented to VM)

As with anything you do regarding storage, make sure you have a backup.

Here is what it looks like on the Atomic VM before I add my disks:

       

# atomic host status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME            REFSPEC                                                
* 2015-06-27 20:22:47     22.50     0eca6e0777     fedora-atomic     fedora-atomic:fedora-atomic/f22/x86_64/docker-host     
  2015-05-21 19:01:46     22.17     06a63ecfcf     fedora-atomic     fedora-atomic:fedora-atomic/f22/x86_64/docker-host     


# fdisk -l | grep vd
Disk /dev/vda: 6 GiB, 6442450944 bytes, 12582912 sectors
/dev/vda1  *      2048   616447   614400  300M 83 Linux
/dev/vda2       616448 12582911 11966464  5.7G 8e Linux LVM



As you can see, I only have a vda disk.  I need to create 3 additional disks.  I do this on my KVM hypervisor that I am running Atomic on.  In my case it's a Fedora 21 host.

       
# for i in $(seq 1 3); do qemu-img create -f qcow2 -o preallocation=metadata disk$i.qcow2 4G &> /dev/null; chown qemu.qemu disk$i.qcow2 && chmod 744 disk$i.qcow2; done && ls -ltr disk*
-rwxr--r--. 1 qemu qemu 4295884800 Jun 27 21:22 disk1.qcow2
-rwxr--r--. 1 qemu qemu 4295884800 Jun 27 21:22 disk2.qcow2
-rwxr--r--. 1 qemu qemu 4295884800 Jun 27 21:22 disk3.qcow2



Now that I have 3 new disks, I want to attach them to my running Atomic VM.  Note that I am starting with vdb because the VM already has a vda.

       
# virsh attach-disk atomic1 /extra/libvirt/images/disk1.qcow2 vdb --targetbus virtio  --live
Disk attached successfully

# virsh attach-disk atomic1 /extra/libvirt/images/disk2.qcow2 vdc --targetbus virtio  --live
Disk attached successfully

# virsh attach-disk atomic1 /extra/libvirt/images/disk3.qcow2 vdd --targetbus virtio  --live
Disk attached successfully


Now, back on the Atomic VM you can see the new disks. You don't need to partition them, or pvcreate them.  The docker-storage-setup script will handle all that.

       
# fdisk -l | grep vd
Disk /dev/vda: 6 GiB, 6442450944 bytes, 12582912 sectors
/dev/vda1  *      2048   616447   614400  300M 83 Linux
/dev/vda2       616448 12582911 11966464  5.7G 8e Linux LVM
Disk /dev/vdb: 4 GiB, 4295884800 bytes, 8390400 sectors
Disk /dev/vdc: 4 GiB, 4295884800 bytes, 8390400 sectors
Disk /dev/vdd: 4 GiB, 4295884800 bytes, 8390400 sectors



Now, I can start playing around with docker-storage-setup.  There's at least two scenarios that I want to evaluate.  The first is adding a new disk to my host as a new phyiscal volume, VG and LV.  I want that to be what Docker uses for storage. After that, I want to extend that volume with the other two disks.  So, when I am finished, I will have a total of ~ 12GB of space for my Docker images.  I can get instructions on how to do this by looking at the /bin/docker-storage-setup script.  It says:

       
# This section reads the config file (/
# Currently supported options:
# DEVS: A quoted, space-separated list of devices to be used.  This currently
#       expects the devices to be unpartitioned drives.  If "VG" is not
#       specified, then use of the root disk's extra space is implied.
#
# VG:   The volume group to use for docker storage.  Defaults to the volume
#       group where the root filesystem resides.  If VG is specified and the
#       volume group does not exist, it will be created (which requires that
#       "DEVS" be nonempty, since we don't currently support putting a second
#       partition on the root disk).


Let's take a look at the current configuration.

       
# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: atomicos-docker--pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 11.8 MB
 Data Space Total: 2.961 GB
 Data Space Available: 2.949 GB
 Metadata Space Used: 49.15 kB
 Metadata Space Total: 8.389 MB
 Metadata Space Available: 8.339 MB
 Udev Sync Supported: true
 Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 4.0.6-300.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 2
Total Memory: 1.954 GiB
Name: atomic-00.localdomain
ID: GBO5:RZYO:SGIO:IVQ4:IGIL:E55A:3YGF:CUWZ:LAAV:6Z4P:2WAI:BPD3


You can see that the Pool is atomicos-docker--pool.  We want to change that.

Scenario 1


For the first scenario, I want to go ahead and add the initial disk.  It's really, really easy.

1. Check the configuration before making the changes.

       
# pvs
  PV         VG       Fmt  Attr PSize PFree
  /dev/vda2  atomicos lvm2 a--  5.70g    0 

# lvs
  LV          VG       Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool atomicos twi-a-tz-- 2.76g             0.40   0.59                            
  root        atomicos -wi-ao---- 2.93g                                                    

# vgs
  VG       #PV #LV #SN Attr   VSize VFree
  atomicos   1   2   0 wz--n- 5.70g    0 



2. Create the file /etc/sysconfig/docker-storage-setup with the following entries.

       
# cat /etc/sysconfig/docker-storage-setup

DEVS="vdb"
VG="test-disk"


3. Run the command docker-storage-setup.

       
# docker-storage-setup 
  Volume group "test-disk" not found
  Cannot process volume group test-disk
0
Checking that no-one is using this disk right now ... OK

Disk /dev/vdb: 4 GiB, 4295884800 bytes, 8390400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0xe737456c.
Created a new partition 1 of type 'Linux LVM' and of size 4 GiB.
/dev/vdb2: 
New situation:

Device     Boot Start     End Sectors Size Id Type
/dev/vdb1        2048 8390399 8388352   4G 8e Linux LVM

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
  Physical volume "/dev/vdb1" successfully created
  Volume group "test-disk" successfully created
NOCHANGE: partition 2 is size 11966464. it cannot be grown
  Physical volume "/dev/vda2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
  Rounding up size to full physical extent 8.00 MiB
  Logical volume "docker-meta" created.
  Logical volume "docker-data" created.


4. Restart Docker to consume the new configuration.

       
# systemctl restart docker


5. Check the new configuration.

       
# pvs
  PV         VG        Fmt  Attr PSize PFree 
  /dev/vda2  atomicos  lvm2 a--  5.70g     0 
  /dev/vdb1  test-disk lvm2 a--  4.00g 84.00m

# vgs
  VG        #PV #LV #SN Attr   VSize VFree 
  atomicos    1   2   0 wz--n- 5.70g     0 
  test-disk   1   2   0 wz--n- 4.00g 84.00m

# lvs
  LV          VG        Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool atomicos  twi-a-tz-- 2.76g             0.40   0.59                            
  root        atomicos  -wi-ao---- 2.93g                                                    
  docker-data test-disk -wi-ao---- 3.91g                                                    
  docker-meta test-disk -wi-ao---- 8.00m                                                    

# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:0-8473021-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/test-disk/docker-data
 Metadata file: /dev/test-disk/docker-meta
 Data Space Used: 11.8 MB
 Data Space Total: 4.194 GB
 Data Space Available: 4.183 GB
 Metadata Space Used: 53.25 kB
 Metadata Space Total: 8.389 MB
 Metadata Space Available: 8.335 MB
 Udev Sync Supported: true
 Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 4.0.6-300.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 2
Total Memory: 1.954 GiB
Name: atomic-00.localdomain
ID: GBO5:RZYO:SGIO:IVQ4:IGIL:E55A:3YGF:CUWZ:LAAV:6Z4P:2WAI:BPD3


Now you can see that we are using the new disk and we have a "Data Space Total" of 4GB.

Before using this new storage, you will need to clean out /var/lib/docker and restart Docker.  The reason for this is that we are going from one thin pool volume to another.

Scenario 2


For the second scenario, I want to extend that so we have more space for the data file.  Again, really easy.

1. Modify the /etc/sysconfig/docker file to add the two new disks and run docker-storage-setup.

       
# cat /etc/sysconfig/docker-storage-setup

DEVS="vdc vdd"
VG="test-disk"

# docker-storage-setup 
0
Checking that no-one is using this disk right now ... OK

Disk /dev/vdc: 4 GiB, 4295884800 bytes, 8390400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0x2bd6f997.
Created a new partition 1 of type 'Linux LVM' and of size 4 GiB.
/dev/vdc2: 
New situation:

Device     Boot Start     End Sectors Size Id Type
/dev/vdc1        2048 8390399 8388352   4G 8e Linux LVM

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
  Physical volume "/dev/vdc1" successfully created
0
Checking that no-one is using this disk right now ... OK

Disk /dev/vdd: 4 GiB, 4295884800 bytes, 8390400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0x4d2c0a6f.
Created a new partition 1 of type 'Linux LVM' and of size 4 GiB.
/dev/vdd2: 
New situation:

Device     Boot Start     End Sectors Size Id Type
/dev/vdd1        2048 8390399 8388352   4G 8e Linux LVM

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
  Physical volume "/dev/vdd1" successfully created
  Volume group "test-disk" successfully extended
NOCHANGE: partition 2 is size 11966464. it cannot be grown
  Physical volume "/dev/vda2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
  Rounding size to boundary between physical extents: 16.00 MiB
  Size of logical volume test-disk/docker-meta changed from 8.00 MiB (2 extents) to 16.00 MiB (4 extents).
  Logical volume docker-meta successfully resized
  Size of logical volume test-disk/docker-data changed from 3.91 GiB (1000 extents) to 11.81 GiB (3024 extents).
  Logical volume docker-data successfully resized



2. Now restart Docker and check the new configuration.

       
# systemctl restart docker


# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:0-8473021-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/test-disk/docker-data
 Metadata file: /dev/test-disk/docker-meta
 Data Space Used: 11.8 MB
 Data Space Total: 12.68 GB
 Data Space Available: 12.67 GB
 Metadata Space Used: 90.11 kB
 Metadata Space Total: 16.78 MB
 Metadata Space Available: 16.69 MB
 Udev Sync Supported: true
 Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 4.0.6-300.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 2
Total Memory: 1.954 GiB
Name: atomic-00.localdomain
ID: GBO5:RZYO:SGIO:IVQ4:IGIL:E55A:3YGF:CUWZ:LAAV:6Z4P:2WAI:BPD3

# pvs
  PV         VG        Fmt  Attr PSize PFree  
  /dev/vda2  atomicos  lvm2 a--  5.70g      0 
  /dev/vdb1  test-disk lvm2 a--  4.00g      0 
  /dev/vdc1  test-disk lvm2 a--  4.00g      0 
  /dev/vdd1  test-disk lvm2 a--  4.00g 164.00m

# lvs
  LV          VG        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool atomicos  twi-a-tz--  2.76g             0.40   0.59                            
  root        atomicos  -wi-ao----  2.93g                                                    
  docker-data test-disk -wi-ao---- 11.81g                                                    
  docker-meta test-disk -wi-ao---- 16.00m                                                    

# vgs
  VG        #PV #LV #SN Attr   VSize  VFree  
  atomicos    1   2   0 wz--n-  5.70g      0 
  test-disk   3   2   0 wz--n- 11.99g 164.00m


That's it.  Enjoy your new disk space!

No comments:

Post a Comment