-
Not Synced
Hi, I'm Colin Watson, this is a recruitment drive.
-
Not Synced
I'm glad to see quite so many people here, that's very useful.
-
Not Synced
[video team]: (inaudible)
-
Not Synced
[Colin]: There's a little bit of feedback?
-
Not Synced
Okay, is that any better?
-
Not Synced
[video team]: (inaudible)
-
Not Synced
[Colin]: Alright.
-
Not Synced
Somehow over the years I've found myself
-
Not Synced
as the de facto GRUB maintainer in debian
-
Not Synced
and I'm also a GRUB developer over the years
-
Not Synced
and I'm here to persuade you all that this is a fun thing to do
-
Not Synced
and that can you help, because I need more people
-
Not Synced
So yay
-
Not Synced
grub has moved on a long way from it's beginnings
-
Not Synced
It's called the GRAND UNIFIED BOOT LOADER
-
Not Synced
which is a bit of an aspirational name
-
Not Synced
and certainly was an aspirational name
-
Not Synced
back in 1995 and back then most us of just
-
Not Synced
used it over LILO which is the really traditional
-
Not Synced
x86 loader, because you didn't have to remember to reinstall
-
Not Synced
your bootloader when you installed a new kernel which was really
-
Not Synced
annoying, but even so quite a few debian developers have
-
Not Synced
hacked on grub over the years and nowadays it's a very powerful
-
Not Synced
bootloader, it's been ported to many architectures
-
Not Synced
it's actually quite rewarding to hack on
-
Not Synced
so I'll be giving you a tour of its history
-
Not Synced
and of its design and suggesting a some particular areas
-
Not Synced
where we could really do with help
-
Not Synced
Now the grub project; this is grub 1
-
Not Synced
it was originally just called grub and now
-
Not Synced
we call it grub legacy, but it was started in 1995 by Erich Boleyn,
-
Not Synced
he was initally trying to build something to boot GNU/Hurd
-
Not Synced
and among the loaders of the day it was quite unusual in that
-
Not Synced
you know, you could edit its menus on the fly and
-
Not Synced
that sort of thing. It had an emacs or bash style interface to that.
-
Not Synced
Other loaders usually just let you append kernel line options and that was
-
Not Synced
about all you got
-
Not Synced
and grub even then had a reasonably capable file system
-
Not Synced
interface as well
-
Not Synced
so it was by and large good enough, so many people adopted it as it was.
-
Not Synced
and of course it was originally designed for the Hurd
-
Not Synced
to start with so it started out with a focus on a new boot method
-
Not Synced
they designed which is called multiboot and the history file says
-
Not Synced
they were determined not to add to the large number of mutually
-
Not Synced
incompatible PC boot methods
-
Not Synced
[laughter]
-
Not Synced
Yes, well.
-
Not Synced
grub did soon become a little bit more generic
-
Not Synced
it supported the methods for booting linux and such
-
Not Synced
and I am being a bit unfair
-
Not Synced
multiboot has been genuinely useful to people doing academic experiments
-
Not Synced
with kernels written from scratch and custom payloads
-
Not Synced
because they don't have to do all of that work from scratch
-
Not Synced
anyway I won't cover multiboot further.
-
Not Synced
This is a rough layout of grub legacy back in the day
-
Not Synced
You had stage 1, this was a tiny thing that fitted
-
Not Synced
in the 512 byte master boot record.
-
Not Synced
It just knew how to read the first sector of the next
-
Not Synced
stage along, stage 1.5, from a fixed location, and jump to it.
-
Not Synced
That was all it could do, it was really stupid.
-
Not Synced
There was a seperate stage 1.5 for each separate file
-
Not Synced
system type that grub understood.
-
Not Synced
And that had enough file system code to
-
Not Synced
read stage 2 from an ordinary file system.
-
Not Synced
So it's the usual kind of bootstrap your way gradually up the stack.
-
Not Synced
The real meat of the loader lived in stage 2 and that included
-
Not Synced
all the command line commands, configuration file parsing
-
Not Synced
and the code to actually boot a payload,
-
Not Synced
like a Linux kernel image,
-
Not Synced
which is probably what you wanted to do with it.
-
Not Synced
So far, so good.
-
Not Synced
But there were quite a few problems with the design
-
Not Synced
as it was initially put together.
-
Not Synced
The file system abstraction was pretty ad-hoc.
-
Not Synced
It was achieved by building a seperate stage 1.5 blob
-
Not Synced
for each file system that grub knew about
-
Not Synced
and shipping that in the package.
-
Not Synced
The terminal abstraction was pretty reasonable aswell,
-
Not Synced
but other than that there wasn't much in the way of
-
Not Synced
internal expressive par(sing??) going on.
-
Not Synced
So it was very difficult to extend it safely.
-
Not Synced
It's hard to work out how grub legacy could ever
-
Not Synced
have been taught to handle LVM, for instance,
-
Not Synced
because you would then have to have a stage 1.5 that knew
-
Not Synced
about each of the file systems on LVM and so on.
-
Not Synced
You end up with a combinatorial explosion.
-
Not Synced
So it wasn't very elegant, from that point of view.
-
Not Synced
There were lots of PC BIOS assumptions aswell.
-
Not Synced
It may have been grand and unified,
-
Not Synced
but it wasn't particularly portable at that point.
-
Not Synced
Fedora did manage to get it to work with UEFI.
-
Not Synced
Actually, probably Red Hat then managed to get it
-
Not Synced
to work with UEFI, but it wasn't at all
-
Not Synced
a straight-forward exercise and it was very hacky.
-
Not Synced
So as time went on, upstream maintenance kind of
-
Not Synced
fell by the way side, it was such a pain to work
-
Not Synced
on and even though there wasn't really anything newer
-
Not Synced
that was usable, nobody wasn really interested in the
-
Not Synced
upstream so the distributions did what they had to do.
-
Not Synced
This meant we have ended up with grub legacy packages
-
Not Synced
in Debian and Ubuntu and Fedora and SUSE and all the rest,
-
Not Synced
that are actually completely different products.
-
Not Synced
I don't just mean have a different patch set,
-
Not Synced
in the usual way that you get,
-
Not Synced
they have completely different installation
-
Not Synced
and configuration tools that don't exist in
-
Not Synced
other multiple branches.
-
Not Synced
So today it's impossible if someone comes to us upstream,
-
Not Synced
to support grub 0.97, because it just isn't enough of a thing by itself.
-
Not Synced
You have to figure out what distribution people are using
-
Not Synced
and tell them what to do for that.
-
Not Synced
So we end up punting a lot of that to the distributions to support.
-
Not Synced
So there is a grub 2 rewrite, started in around 2002.
-
Not Synced
This is a very rough timeline of it,
-
Not Synced
as you can see it was a pretty long project, pretty slow project.
-
Not Synced
Yoshinori K. Okuji, I hope I pronounced his name correctly,
-
Not Synced
he was the lead grub maintainer at the time.
-
Not Synced
He started work on what he called:
-
Not Synced
"Preliminary Universal Programming Architecture for GNU Grub"
-
Not Synced
or 'PUPA'.
-
Not Synced
It's GNU so there is a bad pun in the acronym,
-
Not Synced
a pupa is the next stage along from a larva or grub in an insect's lifecycle.
-
Not Synced
I'd say it was roughly usable by about 2007,
-
Not Synced
so there was a long period when it was very experimental.
-
Not Synced
My pretty biased viewpoint is that distributions started
-
Not Synced
adopting it properly and started to present it to users in about 2009.
-
Not Synced
A snowball effect kind of kicked in from there
-
Not Synced
and it became much better quality quickly,
-
Not Synced
until we finally got 2.0 out in 2012.
-
Not Synced
So I also noted in here the point where the first
-
Not Synced
non-x86 architecture port showed up, that was the PowerPC port
-
Not Synced
for New World Macs, in 2004.
-
Not Synced
Because that's a very unusual milestone
-
Not Synced
in something as architecture specific as bootloaders have historically tended to be.
-
Not Synced
But right now if you look at some core bits of grub 2,
-
Not Synced
you can kind of see the echos of bits of grub legacy in there.
-
Not Synced
There are a few common lines of code
-
Not Synced
but it was a pretty sweeping rewrite and there isn't much left.
-
Not Synced
This was a very large project by most standards,
-
Not Synced
the lead maintainship has changed hands 3 or 4 times along the way,
-
Not Synced
it took a lot of forward thinking for people to get involved early on.
-
Not Synced
So the design that we have now, is around a small kernel
-
Not Synced
that offers core features like initialisation,
-
Not Synced
a virtual file system layer, a module loader
-
Not Synced
almost everything is in modules, sound familar?
-
Not Synced
there is an insmod command,
-
Not Synced
there is considerable inspiration taken from
-
Not Synced
full-fledged operating systems here.
-
Not Synced
You mostly don't have to worry about much of this by hand,
-
Not Synced
grub-install which is the main installation tool,
-
Not Synced
works out the bare minimum set of modules you need
-
Not Synced
to read everything else from /boot/grub,
-
Not Synced
and builds what we call a core image.
-
Not Synced
That's the rough equivalent of what stage 1.5 was in grub legacy,
-
Not Synced
but this time it's built dynamically,
-
Not Synced
rather than all being shipped statically in the package.
-
Not Synced
And also at runtime, the loader will load some modules into itself automatically.
-
Not Synced
So for example if you request a command name that isn't currently loaded,
-
Not Synced
it will go off and load it for you automatically.
-
Not Synced
This architecture makes it a lot easier to
-
Not Synced
construct images that fit into a constrained space,
-
Not Synced
in a much more flexible way.
-
Not Synced
We make very heavy use of abstraction layers,
-
Not Synced
it's rather like the way, not quite identical,
-
Not Synced
but rather like the way Unix-like systems
-
Not Synced
present block devices and file systems.
-
Not Synced
And those are general composable,
-
Not Synced
you can quite easily boot from XFS on LVM on RAID, whatever.
-
Not Synced
There's even the loopback module, so you can treat a file as
-
Not Synced
if it were a disk, much like we have in Linux.
-
Not Synced
This makes it quite easy to do strange things,
-
Not Synced
like you can embed an entire Debian installation
-
Not Synced
in a file on a Windows system,
-
Not Synced
which we actually did with Ubuntu for a while
-
Not Synced
and it's quite scary but it largely works.
-
Not Synced
This design also lets us easily build userspace
-
Not Synced
tools from very near to the same code,
-
Not Synced
which is very important and I'll come onto that later.
-
Not Synced
As you can see from the random stats at the bottom,
-
Not Synced
there's very little assembly in this,
-
Not Synced
it's almost all portable C.
-
Not Synced
Almost all of the C is common,
-
Not Synced
the assembly is of course a lot more architecture specific
-
Not Synced
but for a PC BIOS we are only talking about
-
Not Synced
about 4 and a half thousand lines,
-
Not Synced
which is considerable but not too much of a maintenance burden.
-
Not Synced
So here is a brief summary of what architectures we have now,
-
Not Synced
we have a number of x86 platforms,
-
Not Synced
the newest of those is that you can now use grub
-
Not Synced
under Xen paravirtualization, with a bit of effort.
-
Not Synced
The newest architectures are arm and arm64,
-
Not Synced
which were added last year.
-
Not Synced
In practice only some arm32 devices actually work with this,
-
Not Synced
because they require the platform's Uboot to be built with
-
Not Synced
the Uboot API so that it can act as enough of a firmware for grub.
-
Not Synced
A lot of them don't have that turned on, but it's at least a start.
-
Not Synced
I believe that, in theory,
-
Not Synced
all of jessie's release architectures other than s390
-
Not Synced
(no it's s390x now, isn't it?)
-
Not Synced
have at least some level of grub support.
-
Not Synced
Which is pretty awesome.
-
Not Synced
In grub legacy the hardest and one of the most
-
Not Synced
common problems we had to debug were problems
-
Not Synced
with reading files from disk in one way or another.
-
Not Synced
So people had problems loading the kernel and initrd
-
Not Synced
because their filesystem was in some strange shape.
-
Not Synced
If you wanted to debug this you were usually stuck with
-
Not Synced
trying to set up something that roughly matched in an emulator
-
Not Synced
and either using just crude printf debugging
-
Not Synced
or in extreme cases trying to attach gdb to the emulated machine over qemu.
-
Not Synced
Some people in the audience may be familar with this routine,
-
Not Synced
it's not very much fun, especially without a gdb stub.
-
Not Synced
So in grub 2, as I said earlier,
-
Not Synced
we have much of the same code built into utilities called grub-probe and grub-fstest
-
Not Synced
and these use the operating system's block device as a backend.
-
Not Synced
So you can ask grub to use its own partition table and file system tools
-
Not Synced
and parsing code from right there in userspace.
-
Not Synced
This makes it very much easier to attack lots of common problems;
-
Not Synced
you can use gdb, you can use valgrind,
-
Not Synced
you can use the debugging tools of your choice.
-
Not Synced
Tom ?? was asking me earlier about f2fs and putting that together in grub 2,
-
Not Synced
it can almost entirely be done in userspace so you can put together a new module,
-
Not Synced
built a version of grub-probe that's linked against it
-
Not Synced
and iterate until you get something that works,
-
Not Synced
rather than having to constantly reboot.
-
Not Synced
Even having to reboot an emulated machine is a lot of work.
-
Not Synced
So a similar trick also gave us this spin-off benefit of grub-mount,
-
Not Synced
that's a FUSE file system implementation that uses grub's filesystem code.
-
Not Synced
So that gives you a guaranteed true read-only mount,
-
Not Synced
using the exact same view that the bootloader
-
Not Synced
will have when it tries to look at your system.
-
Not Synced
This lets you avoid the caveats that apply to
-
Not Synced
things like journalling filesystems in Linux,
-
Not Synced
which it turns out you sometimes can't safely read-only mount.
-
Not Synced
Or sometimes the kernel will simply refuse
-
Not Synced
and it's also useful for things like os-prober.
-
Not Synced
If somebody felt the urge to make that work
-
Not Synced
as a Hurd translator it would be a pretty good fit too.
-
Not Synced
Now, you can't do everything in userspace,
-
Not Synced
at boot time grub gives you a pretty nice bash style interactive
-
Not Synced
shell with runtime controllable debugging levels.
-
Not Synced
Set the debug variable to various things and
-
Not Synced
that will give you different amounts of spew on your console.
-
Not Synced
So you can often try things out on the fly,
-
Not Synced
to figure out what's going wrong.
-
Not Synced
You can do quick, miniature image builds using the grub-mkrescue command,
-
Not Synced
that gives you an iso image containing the version of grub that you're using,
-
Not Synced
which you can then boot in an emulator to try out.
-
Not Synced
A useful trick is to boot your real live system with such an image
-
Not Synced
with 'qemu -snapshot' so that it won't write anything back
-
Not Synced
and then you see how the loader that you're working with will boot your laptop,
-
Not Synced
without actually risking the possibility of breaking anything.
-
Not Synced
It's easy to pull out bits of configuration files,
-
Not Synced
run them, write new commands.
-
Not Synced
There is a Hello World command that's about 30 lines,
-
Not Synced
so it's quite easy to put new things together.
-
Not Synced
Getting onto to things that are a little bit move involved and/or broken,
-
Not Synced
one of the things that was part of debian's grub legacy changes
-
Not Synced
was the Debian specific update-grub script.
-
Not Synced
Now this worked by trying to guess which parts of
-
Not Synced
/boot/grub/menu.lst were user modified,
-
Not Synced
by way of magic comments and updating everything
-
Not Synced
else to match the current state of the system.
-
Not Synced
This was sort of OK, but it caused a lot of complaints,
-
Not Synced
sometimes it was just because people didn't read the comment
-
Not Synced
text which said which parts of the file were safe to edit.
-
Not Synced
But generally, mixing user-editable
-
Not Synced
and automatically updated content in the same file
-
Not Synced
usually turns out to be a bad idea
-
Not Synced
and we've seen that in various other places in Debian.
-
Not Synced
So for grub 2, my predecessors brought this upstream as grub-mkconfig
-
Not Synced
and made it generate the whole configuration file from a
-
Not Synced
small amount of user edited configuration in /etc/default/grub
-
Not Synced
and a bunch of scripts in /etc/grub.d.
-
Not Synced
You can still of course write your own grub.cfg directly
-
Not Synced
if you like, it is about the same length as it would be in grub 1,
-
Not Synced
but for a general purpose system we would prefer the autogenerated approach.
-
Not Synced
So far, so good.
-
Not Synced
Just about everything is customisable,
-
Not Synced
the problem is that you have to try quite hard in several places.
-
Not Synced
Anything involved requires editing quite complex shell scripts,
-
Not Synced
if those are changed then later you'll
-
Not Synced
have quite a difficult merge resolution to do.
-
Not Synced
Some changes require things like moving conffiles
-
Not Synced
around to different positions in the order which isn't
-
Not Synced
going to be handled well by dpkg conffile resolution in the future.
-
Not Synced
So all of this probably needs somebody, sadly,
-
Not Synced
to sit down and design a third iteration of the system
-
Not Synced
that uses a templating language or something.
-
Not Synced
but that hasn't really been at all started yet.
-
Not Synced
Ok, now onto some things that are still genuinely quite hard.
-
Not Synced
The PC BIOS architecture has, well,
-
Not Synced
secreted over 30 or 40 years of gradual development.
-
Not Synced
It's basically pretty awful for modern purposes.
-
Not Synced
The usual partition table format is called MBR,
-
Not Synced
for Master Boot Record, or sometimes the msdos partition table.
-
Not Synced
It doesn't offer any formal space for keeping boot code in,
-
Not Synced
you can cram a trivial loader into 446 bytes,
-
Not Synced
that basically gives you the debian mbr package,
-
Not Synced
which jumps to a loader somewhere else.
-
Not Synced
There are a few approaches to where you can keep
-
Not Synced
the rest of, the real meat of, the loader.
-
Not Synced
You can embed in a file system somewhere,
-
Not Synced
at an offset that you know won't change.
-
Not Synced
For instance you put it in a file and trust that
-
Not Synced
the file system is not going to move that around.
-
Not Synced
Sadly sometimes file systems do move things around,
-
Not Synced
you might run fsck or there are some file systems
-
Not Synced
that will do their own, I think tail packing in reiserfs is one of these,
-
Not Synced
that will sometimes move blocks around for you
-
Not Synced
on the fly and that doesn't play very well with a
-
Not Synced
bootloader that has hardcoded addresses in the MBR.
-
Not Synced
A few file systems support an embedding area at the start,
-
Not Synced
I think btrfs does and one or two others,
-
Not Synced
this is good but not really widespread enough
-
Not Synced
that we can build the whole installation strategy around it.
-
Not Synced
You can do what grub generally does now,
-
Not Synced
you can skate the edge of the specification,
-
Not Synced
that sort of really doesn't exist anywhere coherent,
-
Not Synced
and you can use what's sometimes called the boot track
-
Not Synced
or other names, that's the unallocated space between the MBR and the first partition
-
Not Synced
and this makes some people itchy, it's a bit nasty.
-
Not Synced
It means we have to do some alarming things to avoid conflicts
-
Not Synced
with other software that put things in the boot track.
-
Not Synced
Fortunately almost all of that software is evil.
-
Not Synced
[laughter]
-
Not Synced
There are a few bits of Windows software,
-
Not Synced
well some of them are backup utilities which I'm not sure
-
Not Synced
why they put things there, but they probably have a reason.
-
Not Synced
There are a few things on Windows that like to put a note in a
-
Not Synced
sector in the boot track to say that they have been installed,
-
Not Synced
so that you cannot simply uninstall them to work around their 30 day license trial terms,
-
Not Synced
you need to know to wipe that bit of the boot track aswell.
-
Not Synced
And that's all terribly nasty.
-
Not Synced
Grub actually now has error correction on some of it's own code in the core image.
-
Not Synced
Grub can literally cope with a couple of sectors of the
-
Not Synced
core image being overwritten with completely random data.
-
Not Synced
[laughter]
-
Not Synced
It's horrifying that we need this, but yes that surprisingly works.
-
Not Synced
Fortunately this is becoming less of a problem,
-
Not Synced
the alignment requirements on modern disks mean that for decent peformance
-
Not Synced
you want to have your first partition start at 1 megabyte or sometimes even more,
-
Not Synced
rather than the traditional 63 sectors.
-
Not Synced
So we usually now have plenty of space on MBR,
-
Not Synced
but not quite always.
-
Not Synced
Things are also unpredictable when you have multiple disks,
-
Not Synced
you can't tell from Linux which disk the system was actually booted from.
-
Not Synced
There are some things on some BIOS versions that let you make better guesses,
-
Not Synced
but it's unfortunately not universal.
-
Not Synced
You also can't tell what disk it's going to be booted from next time,
-
Not Synced
it's not necessarily the same.
-
Not Synced
Sometimes it depends on which order the devices are enumerated
-
Not Synced
by the PCI bus, even on the BIOS level.
-
Not Synced
The least bad answer is usually to install to all of the fixed disks on your system,
-
Not Synced
but then you annoy some people who have complicated multiboot setups,
-
Not Synced
so you can't really win.
-
Not Synced
The GUID Partition Table format, or GPT, is much better.
-
Not Synced
It's probably the best thing to have come out of the UEFI spec.
-
Not Synced
[knocks down debconf sign]
-
Not Synced
Excuse me, let me not do that. Right, I'll not do that again.
-
Not Synced
This gives you allocated space for
-
Not Synced
the bootloader code in the UEFI system partition,
-
Not Synced
in a partition type that we were able to safely allocate for
-
Not Synced
ourselves because now partition types are a GUID, they are ginormous.
-
Not Synced
We were able to allocate one for ourselves,
-
Not Synced
even when using that on top of BIOS, which you can do.
-
Not Synced
There are different intepretations of the spec,
-
Not Synced
unfortunately this causes some problems.
-
Not Synced
When you install a system with GPT,
-
Not Synced
you're supposed to put what's called a protective MBR in place,
-
Not Synced
that means that anything that trys to parse the disk as MBR
-
Not Synced
knows that it should stay away
-
Not Synced
and not try to scribble over the partition table.
-
Not Synced
The intepretation of that part of the spec is particularly bad.
-
Not Synced
Apple, at least, used to have a completely incompatible boths ways version,
-
Not Synced
you were required instead of a single giant partition to do the best job of
-
Not Synced
representing the GPT partitions in MBR.
-
Not Synced
That's incompatible both ways, I don't know if that's still the case.
-
Not Synced
If you're trying to use GPT on BIOS,
-
Not Synced
some PC class systems require you to set
-
Not Synced
the active flag on the protective MBR,
-
Not Synced
which the GPT spec explicitly forbids for some reason.
-
Not Synced
So you're stuck both ways round.
-
Not Synced
UEFI is where it's at for PC firmware, apparently.
-
Not Synced
Even if your new machine looks like it's BIOS,
-
Not Synced
it's almost certainly a legacy layer
-
Not Synced
implemented internally on top of UEFI.
-
Not Synced
The economics for firmware manufacturers
-
Not Synced
are much more favourable this way.
-
Not Synced
Instead of having to maintain their own,
-
Not Synced
I don't know however many thousand lines of code base that have accreted over the years,
-
Not Synced
they can now fork Intel's TianoCore as their starting point
-
Not Synced
and just do their driver layer on top of that.
-
Not Synced
And this seems to have attracted essentially all of the BIOS manufacturers.
-
Not Synced
We should generally expect even that legacy BIOS layer
-
Not Synced
to go away soon, the direction that we hear from firmware people is all about that
-
Not Synced
and we need to cope with that.
-
Not Synced
Now, grub's core support for UEFI is basically fine,
-
Not Synced
it has more or less the set of drivers you'd expect
-
Not Synced
including relatively arcane things like serial support.
-
Not Synced
Of course the big thing that's been recently
-
Not Synced
is something called "Secure Boot",
-
Not Synced
that's a mechanism by which firmware makes sure to
-
Not Synced
only ever execute signed code, in a pre-boot context.
-
Not Synced
It's obviously very possible for this to go very wrong,
-
Not Synced
from the point of view of user freedom.
-
Not Synced
The FSF calls Secure Boot, "Restricted Boot" for that reason.
-
Not Synced
But a lot of systems now come with Secure Boot enabled by default,
-
Not Synced
so we have to figure out how to work with it
-
Not Synced
in the same way that we've had to figure out
-
Not Synced
how to work with all kinds of devices over the years,
-
Not Synced
just as a hardware enablement matter.
-
Not Synced
But the important thing is to work out how to that
-
Not Synced
in ways that don't end up impinging on our user's freedom.
-
Not Synced
and that's been quite a difficult slog.
-
Not Synced
The community has managed to figure out schemes for this
-
Not Synced
that don't stop you modifying the operating system on your own computer,
-
Not Synced
which is the important thing.
-
Not Synced
We have this working on Ubuntu,
-
Not Synced
we talked about this at DebConf last year.
-
Not Synced
To get it working in Debian we need dak to sign
-
Not Synced
bootloader objects that we submit to it, with the Debian key.
-
Not Synced
I failed with pushing that forward so that
-
Not Synced
would be a great project for somebody to take up,
-
Not Synced
if I've missed it and somebody else is already working on that, brilliant!
-
Not Synced
I should also mention that there are some outstanding
-
Not Synced
issues with the layout of the EFI System Partition,
-
Not Synced
which is UEFI's place for putting things like bootloader code.
-
Not Synced
The spec proscribes how you're supposed to behave
-
Not Synced
on fixed disk versus removable disks,
-
Not Synced
we follow the spec but unfortunately some systems don't
-
Not Synced
and essentially require the removable layout.
-
Not Synced
Steve McIntyre's here at the back,
-
Not Synced
we've gone back and forth a bit on that.
-
Not Synced
We need some way to select the removable layout
-
Not Synced
and have that actually persist and ideally detect
-
Not Synced
that this is a problem in the first place,
-
Not Synced
so that we don't have to ask an incomprehensible question in the installer.
-
Not Synced
Now, more cheerily, a number of non-x86 architectures are in a pretty good state.
-
Not Synced
they get fairly limited testing at the moment.
-
Not Synced
I know that some people use them because occasionally
-
Not Synced
I get some bugs when things break but generally they work OK.
-
Not Synced
powerpc and sparc work fine, we sould probably consider
-
Not Synced
switching the default bootloader there at some point.
-
Not Synced
If you're a porter for those architectures,
-
Not Synced
please get in touch with me.
-
Not Synced
Some mips-type architectures would well too.
-
Not Synced
Ryan ?? lent me a Yeeloong laptop a little while back,
-
Not Synced
so that I could port the debian installer to it
-
Not Synced
and I found that grub was basically fine.
-
Not Synced
So I was able to build the installer on top of that.
-
Not Synced
It was a tenth of the work it might otherwise have been.
-
Not Synced
There's been a pretty pleasing trend towards having
-
Not Synced
new arches include a grub port very early on.
-
Not Synced
arm64, also powerpc64, has had grub working from very early in it's life.
-
Not Synced
This gives them a pretty full-feature loader
-
Not Synced
with not a lot of effort, so it actually
-
Not Synced
works out quite well for the ports now, I think.
-
Not Synced
The most recent arm64 port, I went back and looked at it,
-
Not Synced
it was about 2000 lines of patches to do the initial enablement.
-
Not Synced
And now that got to take the advantage of the arm port
-
Not Synced
that already existed and also of UEFI code,
-
Not Synced
so that helps a lot but still I think it
-
Not Synced
indicates a pretty porting friendly design.
-
Not Synced
OK, so if I haven't persuaded you already,
-
Not Synced
why do we default to grub on debian x86?
-
Not Synced
There are certainly other loaders,
-
Not Synced
there's the venerable LILO, which still largely works fine.
-
Not Synced
Syslinux and friends are actively maintained by some very smart people,
-
Not Synced
including folks who maintain the Linux boot protocol
-
Not Synced
and there's extlinux which is one of that family
-
Not Synced
which you can use in the same kind of way that you might use LILO.
-
Not Synced
There's yaboot and powerpc and so on and so forth.
-
Not Synced
You can even boot a kernel and an initramfs directly
-
Not Synced
from the UEFI environment, if you want to.
-
Not Synced
So some of those are some pretty good bootloaders,
-
Not Synced
they're almost all smaller and simpler than grub
-
Not Synced
and that appeals to a lot of people.
-
Not Synced
I do get the simplicity argument.
-
Not Synced
But I tend to argue that the result of that
-
Not Synced
is moving a good deal of complexity elsewhere.
-
Not Synced
If you have a bootloader that can only handle some setups,
-
Not Synced
the installer needs to know what those setups are.
-
Not Synced
It needs to forbid you from doing anything, that you won't be
-
Not Synced
able to boot, in your partitioner.
-
Not Synced
You end up with static /boot partitions scattered everywhere.
-
Not Synced
Having a bootloader that you can trust to handle almost anything
-
Not Synced
means that you don't have to think too hard when you are moving things around.
-
Not Synced
You can easily do things like having the bootloader notice
-
Not Synced
when the last boot failed and behave differently.
-
Not Synced
All that sort of thing is useful for a general purpose distro to be able to assume.
-
Not Synced
[Ben]: We used to support some bootloaders on mips,
-
Not Synced
but they couldn't load an initramfs, they could only load a kernel.
-
Not Synced
So for that reason the kernel configurations we used on mips
-
Not Synced
had to be restricted, they don't assume there is an initramfs
-
Not Synced
and you can't boot off filesystems other than ext2, 3 and 4.
-
Not Synced
[Colin]: That's CoLo, is it? Or cibyl?
-
Not Synced
[Ben]: I don't know. We have recently switched over,
-
Not Synced
but that has been a restriction up to and including wheezy.
-
Not Synced
[Colin]: Thanks. That's useful ammunition. [laughs]
-
Not Synced
I think that we do not yet have grub working on
-
Not Synced
all of the mips big endian architectures,
-
Not Synced
it's working on little endian,
-
Not Synced
but I think there are still some problems with big endian.
-
Not Synced
So we maybe can't save the world just yet,
-
Not Synced
but it might not be too far off. But thanks.
-
Not Synced
Plus Debian runs on lots of architectures,
-
Not Synced
generally we try as an architectural thing
-
Not Synced
to run the same software across different architectures so
-
Not Synced
that it all works kind of the same way.
-
Not Synced
We have the same interfaces, the same tools are available and so on.
-
Not Synced
And it makes a lot of sense to have that be the case with the bootloader too.
-
Not Synced
The installer would certainly be simplified by being able to
-
Not Synced
assume grub-mount is available on all architectures, for instance.
-
Not Synced
OK, so the important bit, this.
-
Not Synced
We've had a lot of people involved over the years
-
Not Synced
Too many to name, I've almost certainly missed some
-
Not Synced
Robert Millan, Felix Zielcke and Jordi Mallach
-
Not Synced
have done excellent jobs in Debian a few years ago.
-
Not Synced
And Vladimir Serbinenko does an excellent job as the upstream maintainer.
-
Not Synced
But as far as Debian maintenance goes,
-
Not Synced
it's mostly just me at the moment and has been
-
Not Synced
for the last couple of years and I can't do it all.
-
Not Synced
We'd probably have had grub 2 in 2.00
-
Not Synced
rather than 1.99 in wheezy,
-
Not Synced
if we had a bit more redundancy there.
-
Not Synced
I was kind of demotivated by the Secure Boot stuff at the time,
-
Not Synced
so ended up putting it off until too late
-
Not Synced
and that's exactly the kind of thing
-
Not Synced
where having more people involved is more helpful
-
Not Synced
there are lots of bugs, everybody's problem is critical for them,
-
Not Synced
quite understandably.
-
Not Synced
So keeping on top of the release critical list is absolute murder.
-
Not Synced
But there are a few specific problems I'd like to highlight,
-
Not Synced
that are more substantial and could really do with
-
Not Synced
somebody sitting down and thinking quite hard.
-
Not Synced
I won't go into too much detail,
-
Not Synced
as I've just had a sign from the video team that I'm low on time.
-
Not Synced
But you can ask me about any of them if you're interested.
-
Not Synced
I mentioned earlier that the system for generating
-
Not Synced
the config file has a bad case of second system effect
-
Not Synced
and really needs a third system
-
Not Synced
[laughter]
-
Not Synced
to make it easier to customise things.
-
Not Synced
All problems can be solved by another rewrite, right?
-
Not Synced
[laughter]
-
Not Synced
On MBR systems, we often have robustness problems
-
Not Synced
with grub not installing the boot code to the place
-
Not Synced
that your system actually boots from,
-
Not Synced
particularly when multiple disks are involved.
-
Not Synced
This tends to manifest as module loading failures
-
Not Synced
because what happens is, over time
-
Not Synced
the core image that you're actually using
-
Not Synced
that you didn't know about, becomes incompatible with new grub modules
-
Not Synced
because the ABI changes over time
-
Not Synced
and eventually becomes incompatible and can't boot anything
-
Not Synced
I think that needs some kind of overhaul somewhere,
-
Not Synced
though I'm not quite sure where.
-
Not Synced
There are the remaining bit of UEFI,
-
Not Synced
which I mentioned earlier.
-
Not Synced
There's getting signing sorted out in Debian,
-
Not Synced
getting a thing called MokManager,
-
Not Synced
which is part of a seperate project called shim,
-
Not Synced
nicely integrated to give users control over the whole signing process
-
Not Synced
which will let us require signed kernels under Secure Boot
-
Not Synced
and thus make Matthew Garrett want to kill me a little bit less
-
Not Synced
[laughter]
-
Not Synced
and there's the issues with EFI System Partition layout
-
Not Synced
There's some work to be done with upstream Xen
-
Not Synced
to layout effectively a new boot protocol for PV Grub 2
-
Not Synced
so we can have Xen guests install the bootloader
-
Not Synced
in a consistent place in the file system.
-
Not Synced
In fact, Ian Campbell recently sent me something that
-
Not Synced
might be useful for this, I didn't have time to read it.
-
Not Synced
and hopefully this will eventually let us kill off all
-
Not Synced
of the old strange things that live in Xen, to do this.
-
Not Synced
Finally, we should take much better advantage of grub's ports
-
Not Synced
and switch over to them on many more architectures than we have done to date.
-
Not Synced
I've just got onto questions, so excellent.
-
Not Synced
[Ian]: One of things I've found difficult getting to grips with grub 1
-
Not Synced
and grub 2 is that the documentation hasn't always been as good.
-
Not Synced
LILO has always been exceptional in that it came with
-
Not Synced
an extremely good document that described
-
Not Synced
its principles of operation and how to get it to do,
-
Not Synced
well, almost anything you might want it to.
-
Not Synced
Is that also something you're looking for help with?
-
Not Synced
[Colin]: Yes.
-
Not Synced
About a year or two ago,
-
Not Synced
I went through and tried to systematically documenting
-
Not Synced
all of the grub commands and overhauling some other things.
-
Not Synced
I got about half way through that,
-
Not Synced
before I ran out of time.
-
Not Synced
but one of the problems has been with grub 2
-
Not Synced
that the documentation got out of date during the rewrite
-
Not Synced
so nobody has an incentive to do it properly for their incremental changes.
-
Not Synced
So I think getting that into shape
-
Not Synced
would make it much easier to keep it in shape
-
Not Synced
It's better than it was in 2010, it's not where it should be.
-
Not Synced
So, yes.
-
Not Synced
Anybody else, anything from irc aswell?
-
Not Synced
[John]: This was a related question to your comment earlier
-
Not Synced
about how BIOSes are widely inconsistent about
-
Not Synced
how they choose which disk they're going to boot from.
-
Not Synced
Is there anything that grub could do to help us
-
Not Synced
with best practices for failover configurations?
-
Not Synced
So for example, let's say I have a server
-
Not Synced
and I want to install two disk that are mirrored,
-
Not Synced
using lvm or something else.
-
Not Synced
One of the challenges is, how do I configure grub
-
Not Synced
on both disk to be configured the right
-
Not Synced
way, despite all of the weirdness with BIOSes?
-
Not Synced
To have a reasonable easy to maintain setup,
-
Not Synced
so that where if one disk actually goes completely offline,
-
Not Synced
the system could automatically reboot.
-
Not Synced
[Colin]: One of the things I tried to do,
-
Not Synced
the first time I systematically
-
Not Synced
attacked this in Debian grub,
-
Not Synced
the thing I tried to do was to arrange that we
-
Not Synced
did as by-uuid install to disks, possibly as by-path, I forget.
-
Not Synced
So we encourage people to install to all of the disks on a system.
-
Not Synced
Also the grub-pc.postinst notices when a disk has gone away since it last checked,
-
Not Synced
and it's supposed to offer you the chance to update this.
-
Not Synced
But you're right it's a little bit fiddly to manage
-
Not Synced
particularly if you're in a system that's large enough
-
Not Synced
so that statistically disks are failing quite often
-
Not Synced
Then it's a problem.
-
Not Synced
I think what I'd probably like to do is to make sure that
-
Not Synced
we install to the disks that are associated with a RAID device or something
-
Not Synced
so you could just bootstrap off the RAID management.
-
Not Synced
I think some of that works but isn't hooked up very well in Debian.
-
Not Synced
But there are several things to do there.
-
Not Synced
I think there's also a bug in d-i where it doesn't install
-
Not Synced
to all of the fixed disks in your system but only one.
-
Not Synced
So there are several things to fix there.
-
Not Synced
There's a question over there.
-
Not Synced
[??]: I think my question could be summed up as:
-
Not Synced
"How tall do you have to be to ride this ride?"
-
Not Synced
[laughter]
-
Not Synced
It seems like getting started,
-
Not Synced
introducing regressions would be a really scary thing.
-
Not Synced
For someone who wanted to get started with the project
-
Not Synced
and I'm just wondering what kind of facilities do you have
-
Not Synced
for before we actually upload a version to
-
Not Synced
test across the broad range of architectures.
-
Not Synced
[Colin]: Right well, one of the points of this talk
-
Not Synced
was to try to emphasise that you don't actually have
-
Not Synced
to be as tall as people think you do.
-
Not Synced
But you're right about the need for regression testing.
-
Not Synced
One thing that really helps is that the different parts
-
Not Synced
of grub are much more isolated than they used to be.
-
Not Synced
So you can quite easily change one module,
-
Not Synced
test changes with that, without having to worry about
-
Not Synced
strange leakage all over the place.
-
Not Synced
But there are things that that doesn't cover,
-
Not Synced
there is some boot testing arrangement in the build system,
-
Not Synced
so there are some make targets that go off
-
Not Synced
and do boot checks of real systems against the current version of the code.
-
Not Synced
And that sort of thing helps a lot,
-
Not Synced
I would like to have grub hooked up better to some sort of autopkgtest arrangement
-
Not Synced
so that it does emulated boots across various types
-
Not Synced
of systems on ci.debian.net, I'd love help with that.
-
Not Synced
[Steve]: Now of course UEFI, there's a whole slew of things. Tom was asking
-
Not Synced
about MBR and a reliable fallback, of course. I'm not aware of any UEFI
-
Not Synced
implementation that actually does fallback for the system partition either.
-
Not Synced
So that is an utter trainwreck.
-
Not Synced
[Colin]: I suspect what you're,
-
Not Synced
maybe you're meant to use hardware RAID or something?
-
Not Synced
[Steve]: Clearly that's what you're expected to use and it's a mess.
-
Not Synced
We should definitely get together
-
Not Synced
and talk about the removable media path.
-
Not Synced
Again there isn't a good answer but let's try
-
Not Synced
and make it as not crap as we can.
-
Not Synced
[Colin]: Do you know of any way to detect that
-
Not Synced
system even if it's something like a blacklist of broken DMI decoders?
-
Not Synced
[Steve]: I think that's all we can do.
-
Not Synced
We had a system, as an example
-
Not Synced
for the people that may not have come across this,
-
Not Synced
there are firmware implementiatons out there
-
Not Synced
that straight after install may not actually recognise
-
Not Synced
that you've put grub into the path where you're meant to put it.
-
Not Synced
Once you've booted off a rescue system, however,
-
Not Synced
and done some not particularly well understood sequence of things,
-
Not Synced
including boot off a removable media path, reboot a couple more times
-
Not Synced
shuffle things around, spin twice clockwise...
-
Not Synced
[laughter]
-
Not Synced
suddenly they then will recognise that you've put grub there.
-
Not Synced
But god forbid, you ever want to update it again.
-
Not Synced
It's painful.
-
Not Synced
[Colin]: There are some exciting problems on Apple Macs,
-
Not Synced
that are a shadow of the same class of problem.
-
Not Synced
[Steve]: And of course Secure Boot,
-
Not Synced
kibi asked Steven about this a few days ago.
-
Not Synced
Of course, we've been talking about this for like 2 years.
-
Not Synced
I feel rubbish as well, that we haven't really got any further with it.
-
Not Synced
If someone really really wants to help get involved with that, oh absolutely,
-
Not Synced
there's a whole load of people who would love to see you help.
-
Not Synced
[Colin]: Not only are there are a whole load of people who would like to see you help,
-
Not Synced
but there's actually roughly a roadmap of what needs to be done.
-
Not Synced
You don't have to blaze a trail.
-
Not Synced
[??]: Scrape the names off the roadmap and put your name on it.
-
Not Synced
[laughter]
-
Not Synced
[Colin]: It's fine, we don't really care, we just want to see it done!
-
Not Synced
The things that need to be done are, as I say,
-
Not Synced
the next step I think is in dak and then
-
Not Synced
somebody needs to chase up getting shim into debian
-
Not Synced
with the right kind of signatures on it and so on
-
Not Synced
but none of it is actually fundamentally hard,
-
Not Synced
it just requires time to push all of that forward.
-
Not Synced
[Steve]: Exactly. I'll pass onto anybody else.
-
Not Synced
[video team]: Actually, we're out of time.
-
Not Synced
[Colin]: OK, thank you all very much.
-
Not Synced
[applause]
-
Not Synced