24 April 2013
Today I learned from the awesome bob how to add storage to a box,
including partitioning, mounting, and other things I'd heard of but had never
We had to add some storage to the elasticsearch boxes. We generate a lot of logs
and they all go to elasticsearch. By default when we provision servers we give
them 32GB but this wasn't enough. So here's how to do it:
- Add the extra disk through our virtualisation platform's control panel.
- Reboot the box so that it sees the new disk.
- Partition the new disk. There are many ways to partition a disk - we used cfdisk, which has a GUI.
This opens the GUI and then you tab through it to select options. We just
accepted all the defaults, so the only time we had to change it was at the end,
to select [write].
We did consider making the type Linux LVM as this would have meant we could extend it
later, but in this case we just accepted the default type of Linux.
This means that as well as /dev/sdb we have /dev/sdb1. (Extra learning points
here - disks are added alphabetically. So the existing disk was sda. If,
subsequent to this, we add another disk, it will be sdc.)
- Before we can mount the disk we need to give it a filesystem. We gave it ext4 because "everything uses ext4
these days" (bob). If you don't have a good reason for picking one of the others, then
- Now we mount the new disk to a temporary directory:
mount /dev/sdb1 /mnt/tmp
- At this point we need to stop elasticsearch and also disable puppet (otherwise it will just start elasticsearch again on its next run):
service elasticsearch-logging stop
puppet agent --disable
- Now we need to copy the elasticsearch indices to the new disk. Because the indices are just files you can rsync them, as long as elasticsearch has been stopped, which it has.
rsync -av --stats --progress /mnt/elasticsearch/*
The -v flag above is 'verbose', and in addition, the stats and progress options
give some more noisy output while the rsync is going on. Often, when using rsync
you would want to add -z, but since this is just from one disk to another and
not going over the wire then there's no need here. Finally, -a is 'archive'. As
the man page helpfully notes, this is "same as -rlptgoD (no -H)". I will leave
that as an exercise for the reader.
This stage takes some time.
- When it's finished, we did two things, just as a (slightly paranoid) check
to see if it worked:
- Run the same command again. If it has successfully rsynched everything, the two will be the same so it will output immediately.
-sh to check that /mnt/elasticsearch and /mnt/tmp are the same
- Instead of mounting the new disk in the same way as step 5 above, we can edit /etc/fstab. We add this line:
/dev/sdb1 /mnt/elasticsearch ext4 defaults 0
We should have thought about using Puppet here, as doing it like this means there are parts of the infrastructure that are not code.
- Next, we unmount sdb1 from /mnt/tmp:
- Then, we move the old elastic search directory and create a new one:
mv /mnt/elasticsearch /mnt/elasticsearch-old
- Now mount the sdb1 to the new elasticsearch. Because of step 9 above, we can pass either the mount directory or the directory to be mounted; we do not need both. This has the added advantage of testing our fstab setup.
- All done! Now we need to:
- Restart elastic search
- Re-enable puppet
- Delete /mnt/tmp and /mnt/elasticsearch-old