WEBVTT 99:59:59.999 --> 99:59:59.999 Hi, I'm Colin Watson, this is a recruitment drive. 99:59:59.999 --> 99:59:59.999 I'm glad to see quite so many people here, that's very useful. 99:59:59.999 --> 99:59:59.999 [video team]: (inaudible) 99:59:59.999 --> 99:59:59.999 [Colin]: There's a little bit of feedback? 99:59:59.999 --> 99:59:59.999 Okay, is that any better? 99:59:59.999 --> 99:59:59.999 [video team]: (inaudible) 99:59:59.999 --> 99:59:59.999 [Colin]: Alright. 99:59:59.999 --> 99:59:59.999 Somehow over the years I've found myself 99:59:59.999 --> 99:59:59.999 as the de facto GRUB maintainer in debian 99:59:59.999 --> 99:59:59.999 and I'm also a GRUB developer over the years 99:59:59.999 --> 99:59:59.999 and I'm here to persuade you all that this is a fun thing to do 99:59:59.999 --> 99:59:59.999 and that can you help, because I need more people 99:59:59.999 --> 99:59:59.999 So yay 99:59:59.999 --> 99:59:59.999 grub has moved on a long way from it's beginnings 99:59:59.999 --> 99:59:59.999 It's called the GRAND UNIFIED BOOT LOADER 99:59:59.999 --> 99:59:59.999 which is a bit of an aspirational name 99:59:59.999 --> 99:59:59.999 and certainly was an aspirational name 99:59:59.999 --> 99:59:59.999 back in 1995 and back then most us of just 99:59:59.999 --> 99:59:59.999 used it over LILO which is the really traditional 99:59:59.999 --> 99:59:59.999 x86 loader, because you didn't have to remember to reinstall 99:59:59.999 --> 99:59:59.999 your bootloader when you installed a new kernel which was really 99:59:59.999 --> 99:59:59.999 annoying, but even so quite a few debian developers have 99:59:59.999 --> 99:59:59.999 hacked on grub over the years and nowadays it's a very powerful 99:59:59.999 --> 99:59:59.999 bootloader, it's been ported to many architectures 99:59:59.999 --> 99:59:59.999 it's actually quite rewarding to hack on 99:59:59.999 --> 99:59:59.999 so I'll be giving you a tour of its history 99:59:59.999 --> 99:59:59.999 and of its design and suggesting a some particular areas 99:59:59.999 --> 99:59:59.999 where we could really do with help 99:59:59.999 --> 99:59:59.999 Now the grub project; this is grub 1 99:59:59.999 --> 99:59:59.999 it was originally just called grub and now 99:59:59.999 --> 99:59:59.999 we call it grub legacy, but it was started in 1995 by Erich Boleyn, 99:59:59.999 --> 99:59:59.999 he was initally trying to build something to boot GNU/Hurd 99:59:59.999 --> 99:59:59.999 and among the loaders of the day it was quite unusual in that 99:59:59.999 --> 99:59:59.999 you know, you could edit its menus on the fly and 99:59:59.999 --> 99:59:59.999 that sort of thing. It had an emacs or bash style interface to that. 99:59:59.999 --> 99:59:59.999 Other loaders usually just let you append kernel line options and that was 99:59:59.999 --> 99:59:59.999 about all you got 99:59:59.999 --> 99:59:59.999 and grub even then had a reasonably capable file system 99:59:59.999 --> 99:59:59.999 interface as well 99:59:59.999 --> 99:59:59.999 so it was by and large good enough, so many people adopted it as it was. 99:59:59.999 --> 99:59:59.999 and of course it was originally designed for the Hurd 99:59:59.999 --> 99:59:59.999 to start with so it started out with a focus on a new boot method 99:59:59.999 --> 99:59:59.999 they designed which is called multiboot and the history file says 99:59:59.999 --> 99:59:59.999 they were determined not to add to the large number of mutually 99:59:59.999 --> 99:59:59.999 incompatible PC boot methods 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 Yes, well. 99:59:59.999 --> 99:59:59.999 grub did soon become a little bit more generic 99:59:59.999 --> 99:59:59.999 it supported the methods for booting linux and such 99:59:59.999 --> 99:59:59.999 and I am being a bit unfair 99:59:59.999 --> 99:59:59.999 multiboot has been genuinely useful to people doing academic experiments 99:59:59.999 --> 99:59:59.999 with kernels written from scratch and custom payloads 99:59:59.999 --> 99:59:59.999 because they don't have to do all of that work from scratch 99:59:59.999 --> 99:59:59.999 anyway I won't cover multiboot further. 99:59:59.999 --> 99:59:59.999 This is a rough layout of grub legacy back in the day 99:59:59.999 --> 99:59:59.999 You had stage 1, this was a tiny thing that fitted 99:59:59.999 --> 99:59:59.999 in the 512 byte master boot record. 99:59:59.999 --> 99:59:59.999 It just knew how to read the first sector of the next 99:59:59.999 --> 99:59:59.999 stage along, stage 1.5, from a fixed location, and jump to it. 99:59:59.999 --> 99:59:59.999 That was all it could do, it was really stupid. 99:59:59.999 --> 99:59:59.999 There was a seperate stage 1.5 for each separate file 99:59:59.999 --> 99:59:59.999 system type that grub understood. 99:59:59.999 --> 99:59:59.999 And that had enough file system code to 99:59:59.999 --> 99:59:59.999 read stage 2 from an ordinary file system. 99:59:59.999 --> 99:59:59.999 So it's the usual kind of bootstrap your way gradually up the stack. 99:59:59.999 --> 99:59:59.999 The real meat of the loader lived in stage 2 and that included 99:59:59.999 --> 99:59:59.999 all the command line commands, configuration file parsing 99:59:59.999 --> 99:59:59.999 and the code to actually boot a payload, 99:59:59.999 --> 99:59:59.999 like a Linux kernel image, 99:59:59.999 --> 99:59:59.999 which is probably what you wanted to do with it. 99:59:59.999 --> 99:59:59.999 So far, so good. 99:59:59.999 --> 99:59:59.999 But there were quite a few problems with the design 99:59:59.999 --> 99:59:59.999 as it was initially put together. 99:59:59.999 --> 99:59:59.999 The file system abstraction was pretty ad-hoc. 99:59:59.999 --> 99:59:59.999 It was achieved by building a seperate stage 1.5 blob 99:59:59.999 --> 99:59:59.999 for each file system that grub knew about 99:59:59.999 --> 99:59:59.999 and shipping that in the package. 99:59:59.999 --> 99:59:59.999 The terminal abstraction was pretty reasonable aswell, 99:59:59.999 --> 99:59:59.999 but other than that there wasn't much in the way of 99:59:59.999 --> 99:59:59.999 internal expressive par(sing??) going on. 99:59:59.999 --> 99:59:59.999 So it was very difficult to extend it safely. 99:59:59.999 --> 99:59:59.999 It's hard to work out how grub legacy could ever 99:59:59.999 --> 99:59:59.999 have been taught to handle LVM, for instance, 99:59:59.999 --> 99:59:59.999 because you would then have to have a stage 1.5 that knew 99:59:59.999 --> 99:59:59.999 about each of the file systems on LVM and so on. 99:59:59.999 --> 99:59:59.999 You end up with a combinatorial explosion. 99:59:59.999 --> 99:59:59.999 So it wasn't very elegant, from that point of view. 99:59:59.999 --> 99:59:59.999 There were lots of PC BIOS assumptions aswell. 99:59:59.999 --> 99:59:59.999 It may have been grand and unified, 99:59:59.999 --> 99:59:59.999 but it wasn't particularly portable at that point. 99:59:59.999 --> 99:59:59.999 Fedora did manage to get it to work with UEFI. 99:59:59.999 --> 99:59:59.999 Actually, probably Red Hat then managed to get it 99:59:59.999 --> 99:59:59.999 to work with UEFI, but it wasn't at all 99:59:59.999 --> 99:59:59.999 a straight-forward exercise and it was very hacky. 99:59:59.999 --> 99:59:59.999 So as time went on, upstream maintenance kind of 99:59:59.999 --> 99:59:59.999 fell by the way side, it was such a pain to work 99:59:59.999 --> 99:59:59.999 on and even though there wasn't really anything newer 99:59:59.999 --> 99:59:59.999 that was usable, nobody wasn really interested in the 99:59:59.999 --> 99:59:59.999 upstream so the distributions did what they had to do. 99:59:59.999 --> 99:59:59.999 This meant we have ended up with grub legacy packages 99:59:59.999 --> 99:59:59.999 in Debian and Ubuntu and Fedora and SUSE and all the rest, 99:59:59.999 --> 99:59:59.999 that are actually completely different products. 99:59:59.999 --> 99:59:59.999 I don't just mean have a different patch set, 99:59:59.999 --> 99:59:59.999 in the usual way that you get, 99:59:59.999 --> 99:59:59.999 they have completely different installation 99:59:59.999 --> 99:59:59.999 and configuration tools that don't exist in 99:59:59.999 --> 99:59:59.999 other multiple branches. 99:59:59.999 --> 99:59:59.999 So today it's impossible if someone comes to us upstream, 99:59:59.999 --> 99:59:59.999 to support grub 0.97, because it just isn't enough of a thing by itself. 99:59:59.999 --> 99:59:59.999 You have to figure out what distribution people are using 99:59:59.999 --> 99:59:59.999 and tell them what to do for that. 99:59:59.999 --> 99:59:59.999 So we end up punting a lot of that to the distributions to support. 99:59:59.999 --> 99:59:59.999 So there is a grub 2 rewrite, started in around 2002. 99:59:59.999 --> 99:59:59.999 This is a very rough timeline of it, 99:59:59.999 --> 99:59:59.999 as you can see it was a pretty long project, pretty slow project. 99:59:59.999 --> 99:59:59.999 Yoshinori K. Okuji, I hope I pronounced his name correctly, 99:59:59.999 --> 99:59:59.999 he was the lead grub maintainer at the time. 99:59:59.999 --> 99:59:59.999 He started work on what he called: 99:59:59.999 --> 99:59:59.999 "Preliminary Universal Programming Architecture for GNU Grub" 99:59:59.999 --> 99:59:59.999 or 'PUPA'. 99:59:59.999 --> 99:59:59.999 It's GNU so there is a bad pun in the acronym, 99:59:59.999 --> 99:59:59.999 a pupa is the next stage along from a larva or grub in an insect's lifecycle. 99:59:59.999 --> 99:59:59.999 I'd say it was roughly usable by about 2007, 99:59:59.999 --> 99:59:59.999 so there was a long period when it was very experimental. 99:59:59.999 --> 99:59:59.999 My pretty biased viewpoint is that distributions started 99:59:59.999 --> 99:59:59.999 adopting it properly and started to present it to users in about 2009. 99:59:59.999 --> 99:59:59.999 A snowball effect kind of kicked in from there 99:59:59.999 --> 99:59:59.999 and it became much better quality quickly, 99:59:59.999 --> 99:59:59.999 until we finally got 2.0 out in 2012. 99:59:59.999 --> 99:59:59.999 So I also noted in here the point where the first 99:59:59.999 --> 99:59:59.999 non-x86 architecture port showed up, that was the PowerPC port 99:59:59.999 --> 99:59:59.999 for New World Macs, in 2004. 99:59:59.999 --> 99:59:59.999 Because that's a very unusual milestone 99:59:59.999 --> 99:59:59.999 in something as architecture specific as bootloaders have historically tended to be. 99:59:59.999 --> 99:59:59.999 But right now if you look at some core bits of grub 2, 99:59:59.999 --> 99:59:59.999 you can kind of see the echos of bits of grub legacy in there. 99:59:59.999 --> 99:59:59.999 There are a few common lines of code 99:59:59.999 --> 99:59:59.999 but it was a pretty sweeping rewrite and there isn't much left. 99:59:59.999 --> 99:59:59.999 This was a very large project by most standards, 99:59:59.999 --> 99:59:59.999 the lead maintainship has changed hands 3 or 4 times along the way, 99:59:59.999 --> 99:59:59.999 it took a lot of forward thinking for people to get involved early on. 99:59:59.999 --> 99:59:59.999 So the design that we have now, is around a small kernel 99:59:59.999 --> 99:59:59.999 that offers core features like initialisation, 99:59:59.999 --> 99:59:59.999 a virtual file system layer, a module loader 99:59:59.999 --> 99:59:59.999 almost everything is in modules, sound familar? 99:59:59.999 --> 99:59:59.999 there is an insmod command, 99:59:59.999 --> 99:59:59.999 there is considerable inspiration taken from 99:59:59.999 --> 99:59:59.999 full-fledged operating systems here. 99:59:59.999 --> 99:59:59.999 You mostly don't have to worry about much of this by hand, 99:59:59.999 --> 99:59:59.999 grub-install which is the main installation tool, 99:59:59.999 --> 99:59:59.999 works out the bare minimum set of modules you need 99:59:59.999 --> 99:59:59.999 to read everything else from /boot/grub, 99:59:59.999 --> 99:59:59.999 and builds what we call a core image. 99:59:59.999 --> 99:59:59.999 That's the rough equivalent of what stage 1.5 was in grub legacy, 99:59:59.999 --> 99:59:59.999 but this time it's built dynamically, 99:59:59.999 --> 99:59:59.999 rather than all being shipped statically in the package. 99:59:59.999 --> 99:59:59.999 And also at runtime, the loader will load some modules into itself automatically. 99:59:59.999 --> 99:59:59.999 So for example if you request a command name that isn't currently loaded, 99:59:59.999 --> 99:59:59.999 it will go off and load it for you automatically. 99:59:59.999 --> 99:59:59.999 This architecture makes it a lot easier to 99:59:59.999 --> 99:59:59.999 construct images that fit into a constrained space, 99:59:59.999 --> 99:59:59.999 in a much more flexible way. 99:59:59.999 --> 99:59:59.999 We make very heavy use of abstraction layers, 99:59:59.999 --> 99:59:59.999 it's rather like the way, not quite identical, 99:59:59.999 --> 99:59:59.999 but rather like the way Unix-like systems 99:59:59.999 --> 99:59:59.999 present block devices and file systems. 99:59:59.999 --> 99:59:59.999 And those are general composable, 99:59:59.999 --> 99:59:59.999 you can quite easily boot from XFS on LVM on RAID, whatever. 99:59:59.999 --> 99:59:59.999 There's even the loopback module, so you can treat a file as 99:59:59.999 --> 99:59:59.999 if it were a disk, much like we have in Linux. 99:59:59.999 --> 99:59:59.999 This makes it quite easy to do strange things, 99:59:59.999 --> 99:59:59.999 like you can embed an entire Debian installation 99:59:59.999 --> 99:59:59.999 in a file on a Windows system, 99:59:59.999 --> 99:59:59.999 which we actually did with Ubuntu for a while 99:59:59.999 --> 99:59:59.999 and it's quite scary but it largely works. 99:59:59.999 --> 99:59:59.999 This design also lets us easily build userspace 99:59:59.999 --> 99:59:59.999 tools from very near to the same code, 99:59:59.999 --> 99:59:59.999 which is very important and I'll come onto that later. 99:59:59.999 --> 99:59:59.999 As you can see from the random stats at the bottom, 99:59:59.999 --> 99:59:59.999 there's very little assembly in this, 99:59:59.999 --> 99:59:59.999 it's almost all portable C. 99:59:59.999 --> 99:59:59.999 Almost all of the C is common, 99:59:59.999 --> 99:59:59.999 the assembly is of course a lot more architecture specific 99:59:59.999 --> 99:59:59.999 but for a PC BIOS we are only talking about 99:59:59.999 --> 99:59:59.999 about 4 and a half thousand lines, 99:59:59.999 --> 99:59:59.999 which is considerable but not too much of a maintenance burden. 99:59:59.999 --> 99:59:59.999 So here is a brief summary of what architectures we have now, 99:59:59.999 --> 99:59:59.999 we have a number of x86 platforms, 99:59:59.999 --> 99:59:59.999 the newest of those is that you can now use grub 99:59:59.999 --> 99:59:59.999 under Xen paravirtualization, with a bit of effort. 99:59:59.999 --> 99:59:59.999 The newest architectures are arm and arm64, 99:59:59.999 --> 99:59:59.999 which were added last year. 99:59:59.999 --> 99:59:59.999 In practice only some arm32 devices actually work with this, 99:59:59.999 --> 99:59:59.999 because they require the platform's Uboot to be built with 99:59:59.999 --> 99:59:59.999 the Uboot API so that it can act as enough of a firmware for grub. 99:59:59.999 --> 99:59:59.999 A lot of them don't have that turned on, but it's at least a start. 99:59:59.999 --> 99:59:59.999 I believe that, in theory, 99:59:59.999 --> 99:59:59.999 all of jessie's release architectures other than s390 99:59:59.999 --> 99:59:59.999 (no it's s390x now, isn't it?) 99:59:59.999 --> 99:59:59.999 have at least some level of grub support. 99:59:59.999 --> 99:59:59.999 Which is pretty awesome. 99:59:59.999 --> 99:59:59.999 In grub legacy the hardest and one of the most 99:59:59.999 --> 99:59:59.999 common problems we had to debug were problems 99:59:59.999 --> 99:59:59.999 with reading files from disk in one way or another. 99:59:59.999 --> 99:59:59.999 So people had problems loading the kernel and initrd 99:59:59.999 --> 99:59:59.999 because their filesystem was in some strange shape. 99:59:59.999 --> 99:59:59.999 If you wanted to debug this you were usually stuck with 99:59:59.999 --> 99:59:59.999 trying to set up something that roughly matched in an emulator 99:59:59.999 --> 99:59:59.999 and either using just crude printf debugging 99:59:59.999 --> 99:59:59.999 or in extreme cases trying to attach gdb to the emulated machine over qemu. 99:59:59.999 --> 99:59:59.999 Some people in the audience may be familar with this routine, 99:59:59.999 --> 99:59:59.999 it's not very much fun, especially without a gdb stub. 99:59:59.999 --> 99:59:59.999 So in grub 2, as I said earlier, 99:59:59.999 --> 99:59:59.999 we have much of the same code built into utilities called grub-probe and grub-fstest 99:59:59.999 --> 99:59:59.999 and these use the operating system's block device as a backend. 99:59:59.999 --> 99:59:59.999 So you can ask grub to use its own partition table and file system tools 99:59:59.999 --> 99:59:59.999 and parsing code from right there in userspace. 99:59:59.999 --> 99:59:59.999 This makes it very much easier to attack lots of common problems; 99:59:59.999 --> 99:59:59.999 you can use gdb, you can use valgrind, 99:59:59.999 --> 99:59:59.999 you can use the debugging tools of your choice. 99:59:59.999 --> 99:59:59.999 Tom ?? was asking me earlier about f2fs and putting that together in grub 2, 99:59:59.999 --> 99:59:59.999 it can almost entirely be done in userspace so you can put together a new module, 99:59:59.999 --> 99:59:59.999 built a version of grub-probe that's linked against it 99:59:59.999 --> 99:59:59.999 and iterate until you get something that works, 99:59:59.999 --> 99:59:59.999 rather than having to constantly reboot. 99:59:59.999 --> 99:59:59.999 Even having to reboot an emulated machine is a lot of work. 99:59:59.999 --> 99:59:59.999 So a similar trick also gave us this spin-off benefit of grub-mount, 99:59:59.999 --> 99:59:59.999 that's a FUSE file system implementation that uses grub's filesystem code. 99:59:59.999 --> 99:59:59.999 So that gives you a guaranteed true read-only mount, 99:59:59.999 --> 99:59:59.999 using the exact same view that the bootloader 99:59:59.999 --> 99:59:59.999 will have when it tries to look at your system. 99:59:59.999 --> 99:59:59.999 This lets you avoid the caveats that apply to 99:59:59.999 --> 99:59:59.999 things like journalling filesystems in Linux, 99:59:59.999 --> 99:59:59.999 which it turns out you sometimes can't safely read-only mount. 99:59:59.999 --> 99:59:59.999 Or sometimes the kernel will simply refuse 99:59:59.999 --> 99:59:59.999 and it's also useful for things like os-prober. 99:59:59.999 --> 99:59:59.999 If somebody felt the urge to make that work 99:59:59.999 --> 99:59:59.999 as a Hurd translator it would be a pretty good fit too. 99:59:59.999 --> 99:59:59.999 Now, you can't do everything in userspace, 99:59:59.999 --> 99:59:59.999 at boot time grub gives you a pretty nice bash style interactive 99:59:59.999 --> 99:59:59.999 shell with runtime controllable debugging levels. 99:59:59.999 --> 99:59:59.999 Set the debug variable to various things and 99:59:59.999 --> 99:59:59.999 that will give you different amounts of spew on your console. 99:59:59.999 --> 99:59:59.999 So you can often try things out on the fly, 99:59:59.999 --> 99:59:59.999 to figure out what's going wrong. 99:59:59.999 --> 99:59:59.999 You can do quick, miniature image builds using the grub-mkrescue command, 99:59:59.999 --> 99:59:59.999 that gives you an iso image containing the version of grub that you're using, 99:59:59.999 --> 99:59:59.999 which you can then boot in an emulator to try out. 99:59:59.999 --> 99:59:59.999 A useful trick is to boot your real live system with such an image 99:59:59.999 --> 99:59:59.999 with 'qemu -snapshot' so that it won't write anything back 99:59:59.999 --> 99:59:59.999 and then you see how the loader that you're working with will boot your laptop, 99:59:59.999 --> 99:59:59.999 without actually risking the possibility of breaking anything. 99:59:59.999 --> 99:59:59.999 It's easy to pull out bits of configuration files, 99:59:59.999 --> 99:59:59.999 run them, write new commands. 99:59:59.999 --> 99:59:59.999 There is a Hello World command that's about 30 lines, 99:59:59.999 --> 99:59:59.999 so it's quite easy to put new things together. 99:59:59.999 --> 99:59:59.999 Getting onto to things that are a little bit move involved and/or broken, 99:59:59.999 --> 99:59:59.999 one of the things that was part of debian's grub legacy changes 99:59:59.999 --> 99:59:59.999 was the Debian specific update-grub script. 99:59:59.999 --> 99:59:59.999 Now this worked by trying to guess which parts of 99:59:59.999 --> 99:59:59.999 /boot/grub/menu.lst were user modified, 99:59:59.999 --> 99:59:59.999 by way of magic comments and updating everything 99:59:59.999 --> 99:59:59.999 else to match the current state of the system. 99:59:59.999 --> 99:59:59.999 This was sort of OK, but it caused a lot of complaints, 99:59:59.999 --> 99:59:59.999 sometimes it was just because people didn't read the comment 99:59:59.999 --> 99:59:59.999 text which said which parts of the file were safe to edit. 99:59:59.999 --> 99:59:59.999 But generally, mixing user-editable 99:59:59.999 --> 99:59:59.999 and automatically updated content in the same file 99:59:59.999 --> 99:59:59.999 usually turns out to be a bad idea 99:59:59.999 --> 99:59:59.999 and we've seen that in various other places in Debian. 99:59:59.999 --> 99:59:59.999 So for grub 2, my predecessors brought this upstream as grub-mkconfig 99:59:59.999 --> 99:59:59.999 and made it generate the whole configuration file from a 99:59:59.999 --> 99:59:59.999 small amount of user edited configuration in /etc/default/grub 99:59:59.999 --> 99:59:59.999 and a bunch of scripts in /etc/grub.d. 99:59:59.999 --> 99:59:59.999 You can still of course write your own grub.cfg directly 99:59:59.999 --> 99:59:59.999 if you like, it is about the same length as it would be in grub 1, 99:59:59.999 --> 99:59:59.999 but for a general purpose system we would prefer the autogenerated approach. 99:59:59.999 --> 99:59:59.999 So far, so good. 99:59:59.999 --> 99:59:59.999 Just about everything is customisable, 99:59:59.999 --> 99:59:59.999 the problem is that you have to try quite hard in several places. 99:59:59.999 --> 99:59:59.999 Anything involved requires editing quite complex shell scripts, 99:59:59.999 --> 99:59:59.999 if those are changed then later you'll 99:59:59.999 --> 99:59:59.999 have quite a difficult merge resolution to do. 99:59:59.999 --> 99:59:59.999 Some changes require things like moving conffiles 99:59:59.999 --> 99:59:59.999 around to different positions in the order which isn't 99:59:59.999 --> 99:59:59.999 going to be handled well by dpkg conffile resolution in the future. 99:59:59.999 --> 99:59:59.999 So all of this probably needs somebody, sadly, 99:59:59.999 --> 99:59:59.999 to sit down and design a third iteration of the system 99:59:59.999 --> 99:59:59.999 that uses a templating language or something. 99:59:59.999 --> 99:59:59.999 but that hasn't really been at all started yet. 99:59:59.999 --> 99:59:59.999 Ok, now onto some things that are still genuinely quite hard. 99:59:59.999 --> 99:59:59.999 The PC BIOS architecture has, well, 99:59:59.999 --> 99:59:59.999 secreted over 30 or 40 years of gradual development. 99:59:59.999 --> 99:59:59.999 It's basically pretty awful for modern purposes. 99:59:59.999 --> 99:59:59.999 The usual partition table format is called MBR, 99:59:59.999 --> 99:59:59.999 for Master Boot Record, or sometimes the msdos partition table. 99:59:59.999 --> 99:59:59.999 It doesn't offer any formal space for keeping boot code in, 99:59:59.999 --> 99:59:59.999 you can cram a trivial loader into 446 bytes, 99:59:59.999 --> 99:59:59.999 that basically gives you the debian mbr package, 99:59:59.999 --> 99:59:59.999 which jumps to a loader somewhere else. 99:59:59.999 --> 99:59:59.999 There are a few approaches to where you can keep 99:59:59.999 --> 99:59:59.999 the rest of, the real meat of, the loader. 99:59:59.999 --> 99:59:59.999 You can embed in a file system somewhere, 99:59:59.999 --> 99:59:59.999 at an offset that you know won't change. 99:59:59.999 --> 99:59:59.999 For instance you put it in a file and trust that 99:59:59.999 --> 99:59:59.999 the file system is not going to move that around. 99:59:59.999 --> 99:59:59.999 Sadly sometimes file systems do move things around, 99:59:59.999 --> 99:59:59.999 you might run fsck or there are some file systems 99:59:59.999 --> 99:59:59.999 that will do their own, I think tail packing in reiserfs is one of these, 99:59:59.999 --> 99:59:59.999 that will sometimes move blocks around for you 99:59:59.999 --> 99:59:59.999 on the fly and that doesn't play very well with a 99:59:59.999 --> 99:59:59.999 bootloader that has hardcoded addresses in the MBR. 99:59:59.999 --> 99:59:59.999 A few file systems support an embedding area at the start, 99:59:59.999 --> 99:59:59.999 I think btrfs does and one or two others, 99:59:59.999 --> 99:59:59.999 this is good but not really widespread enough 99:59:59.999 --> 99:59:59.999 that we can build the whole installation strategy around it. 99:59:59.999 --> 99:59:59.999 You can do what grub generally does now, 99:59:59.999 --> 99:59:59.999 you can skate the edge of the specification, 99:59:59.999 --> 99:59:59.999 that sort of really doesn't exist anywhere coherent, 99:59:59.999 --> 99:59:59.999 and you can use what's sometimes called the boot track 99:59:59.999 --> 99:59:59.999 or other names, that's the unallocated space between the MBR and the first partition 99:59:59.999 --> 99:59:59.999 and this makes some people itchy, it's a bit nasty. 99:59:59.999 --> 99:59:59.999 It means we have to do some alarming things to avoid conflicts 99:59:59.999 --> 99:59:59.999 with other software that put things in the boot track. 99:59:59.999 --> 99:59:59.999 Fortunately almost all of that software is evil. 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 There are a few bits of Windows software, 99:59:59.999 --> 99:59:59.999 well some of them are backup utilities which I'm not sure 99:59:59.999 --> 99:59:59.999 why they put things there, but they probably have a reason. 99:59:59.999 --> 99:59:59.999 There are a few things on Windows that like to put a note in a 99:59:59.999 --> 99:59:59.999 sector in the boot track to say that they have been installed, 99:59:59.999 --> 99:59:59.999 so that you cannot simply uninstall them to work around their 30 day license trial terms, 99:59:59.999 --> 99:59:59.999 you need to know to wipe that bit of the boot track aswell. 99:59:59.999 --> 99:59:59.999 And that's all terribly nasty. 99:59:59.999 --> 99:59:59.999 Grub actually now has error correction on some of it's own code in the core image. 99:59:59.999 --> 99:59:59.999 Grub can literally cope with a couple of sectors of the 99:59:59.999 --> 99:59:59.999 core image being overwritten with completely random data. 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 It's horrifying that we need this, but yes that surprisingly works. 99:59:59.999 --> 99:59:59.999 Fortunately this is becoming less of a problem, 99:59:59.999 --> 99:59:59.999 the alignment requirements on modern disks mean that for decent peformance 99:59:59.999 --> 99:59:59.999 you want to have your first partition start at 1 megabyte or sometimes even more, 99:59:59.999 --> 99:59:59.999 rather than the traditional 63 sectors. 99:59:59.999 --> 99:59:59.999 So we usually now have plenty of space on MBR, 99:59:59.999 --> 99:59:59.999 but not quite always. 99:59:59.999 --> 99:59:59.999 Things are also unpredictable when you have multiple disks, 99:59:59.999 --> 99:59:59.999 you can't tell from Linux which disk the system was actually booted from. 99:59:59.999 --> 99:59:59.999 There are some things on some BIOS versions that let you make better guesses, 99:59:59.999 --> 99:59:59.999 but it's unfortunately not universal. 99:59:59.999 --> 99:59:59.999 You also can't tell what disk it's going to be booted from next time, 99:59:59.999 --> 99:59:59.999 it's not necessarily the same. 99:59:59.999 --> 99:59:59.999 Sometimes it depends on which order the devices are enumerated 99:59:59.999 --> 99:59:59.999 by the PCI bus, even on the BIOS level. 99:59:59.999 --> 99:59:59.999 The least bad answer is usually to install to all of the fixed disks on your system, 99:59:59.999 --> 99:59:59.999 but then you annoy some people who have complicated multiboot setups, 99:59:59.999 --> 99:59:59.999 so you can't really win. 99:59:59.999 --> 99:59:59.999 The GUID Partition Table format, or GPT, is much better. 99:59:59.999 --> 99:59:59.999 It's probably the best thing to have come out of the UEFI spec. 99:59:59.999 --> 99:59:59.999 [knocks down debconf sign] 99:59:59.999 --> 99:59:59.999 Excuse me, let me not do that. Right, I'll not do that again. 99:59:59.999 --> 99:59:59.999 This gives you allocated space for 99:59:59.999 --> 99:59:59.999 the bootloader code in the UEFI system partition, 99:59:59.999 --> 99:59:59.999 in a partition type that we were able to safely allocate for 99:59:59.999 --> 99:59:59.999 ourselves because now partition types are a GUID, they are ginormous. 99:59:59.999 --> 99:59:59.999 We were able to allocate one for ourselves, 99:59:59.999 --> 99:59:59.999 even when using that on top of BIOS, which you can do. 99:59:59.999 --> 99:59:59.999 There are different intepretations of the spec, 99:59:59.999 --> 99:59:59.999 unfortunately this causes some problems. 99:59:59.999 --> 99:59:59.999 When you install a system with GPT, 99:59:59.999 --> 99:59:59.999 you're supposed to put what's called a protective MBR in place, 99:59:59.999 --> 99:59:59.999 that means that anything that trys to parse the disk as MBR 99:59:59.999 --> 99:59:59.999 knows that it should stay away 99:59:59.999 --> 99:59:59.999 and not try to scribble over the partition table. 99:59:59.999 --> 99:59:59.999 The intepretation of that part of the spec is particularly bad. 99:59:59.999 --> 99:59:59.999 Apple, at least, used to have a completely incompatible boths ways version, 99:59:59.999 --> 99:59:59.999 you were required instead of a single giant partition to do the best job of 99:59:59.999 --> 99:59:59.999 representing the GPT partitions in MBR. 99:59:59.999 --> 99:59:59.999 That's incompatible both ways, I don't know if that's still the case. 99:59:59.999 --> 99:59:59.999 If you're trying to use GPT on BIOS, 99:59:59.999 --> 99:59:59.999 some PC class systems require you to set 99:59:59.999 --> 99:59:59.999 the active flag on the protective MBR, 99:59:59.999 --> 99:59:59.999 which the GPT spec explicitly forbids for some reason. 99:59:59.999 --> 99:59:59.999 So you're stuck both ways round. 99:59:59.999 --> 99:59:59.999 UEFI is where it's at for PC firmware, apparently. 99:59:59.999 --> 99:59:59.999 Even if your new machine looks like it's BIOS, 99:59:59.999 --> 99:59:59.999 it's almost certainly a legacy layer 99:59:59.999 --> 99:59:59.999 implemented internally on top of UEFI. 99:59:59.999 --> 99:59:59.999 The economics for firmware manufacturers 99:59:59.999 --> 99:59:59.999 are much more favourable this way. 99:59:59.999 --> 99:59:59.999 Instead of having to maintain their own, 99:59:59.999 --> 99:59:59.999 I don't know however many thousand lines of code base that have accreted over the years, 99:59:59.999 --> 99:59:59.999 they can now fork Intel's TianoCore as their starting point 99:59:59.999 --> 99:59:59.999 and just do their driver layer on top of that. 99:59:59.999 --> 99:59:59.999 And this seems to have attracted essentially all of the BIOS manufacturers. 99:59:59.999 --> 99:59:59.999 We should generally expect even that legacy BIOS layer 99:59:59.999 --> 99:59:59.999 to go away soon, the direction that we hear from firmware people is all about that 99:59:59.999 --> 99:59:59.999 and we need to cope with that. 99:59:59.999 --> 99:59:59.999 Now, grub's core support for UEFI is basically fine, 99:59:59.999 --> 99:59:59.999 it has more or less the set of drivers you'd expect 99:59:59.999 --> 99:59:59.999 including relatively arcane things like serial support. 99:59:59.999 --> 99:59:59.999 Of course the big thing that's been recently 99:59:59.999 --> 99:59:59.999 is something called "Secure Boot", 99:59:59.999 --> 99:59:59.999 that's a mechanism by which firmware makes sure to 99:59:59.999 --> 99:59:59.999 only ever execute signed code, in a pre-boot context. 99:59:59.999 --> 99:59:59.999 It's obviously very possible for this to go very wrong, 99:59:59.999 --> 99:59:59.999 from the point of view of user freedom. 99:59:59.999 --> 99:59:59.999 The FSF calls Secure Boot, "Restricted Boot" for that reason. 99:59:59.999 --> 99:59:59.999 But a lot of systems now come with Secure Boot enabled by default, 99:59:59.999 --> 99:59:59.999 so we have to figure out how to work with it 99:59:59.999 --> 99:59:59.999 in the same way that we've had to figure out 99:59:59.999 --> 99:59:59.999 how to work with all kinds of devices over the years, 99:59:59.999 --> 99:59:59.999 just as a hardware enablement matter. 99:59:59.999 --> 99:59:59.999 But the important thing is to work out how to that 99:59:59.999 --> 99:59:59.999 in ways that don't end up impinging on our user's freedom. 99:59:59.999 --> 99:59:59.999 and that's been quite a difficult slog. 99:59:59.999 --> 99:59:59.999 The community has managed to figure out schemes for this 99:59:59.999 --> 99:59:59.999 that don't stop you modifying the operating system on your own computer, 99:59:59.999 --> 99:59:59.999 which is the important thing. 99:59:59.999 --> 99:59:59.999 We have this working on Ubuntu, 99:59:59.999 --> 99:59:59.999 we talked about this at DebConf last year. 99:59:59.999 --> 99:59:59.999 To get it working in Debian we need dak to sign 99:59:59.999 --> 99:59:59.999 bootloader objects that we submit to it, with the Debian key. 99:59:59.999 --> 99:59:59.999 I failed with pushing that forward so that 99:59:59.999 --> 99:59:59.999 would be a great project for somebody to take up, 99:59:59.999 --> 99:59:59.999 if I've missed it and somebody else is already working on that, brilliant! 99:59:59.999 --> 99:59:59.999 I should also mention that there are some outstanding 99:59:59.999 --> 99:59:59.999 issues with the layout of the EFI System Partition, 99:59:59.999 --> 99:59:59.999 which is UEFI's place for putting things like bootloader code. 99:59:59.999 --> 99:59:59.999 The spec proscribes how you're supposed to behave 99:59:59.999 --> 99:59:59.999 on fixed disk versus removable disks, 99:59:59.999 --> 99:59:59.999 we follow the spec but unfortunately some systems don't 99:59:59.999 --> 99:59:59.999 and essentially require the removable layout. 99:59:59.999 --> 99:59:59.999 Steve McIntyre's here at the back, 99:59:59.999 --> 99:59:59.999 we've gone back and forth a bit on that. 99:59:59.999 --> 99:59:59.999 We need some way to select the removable layout 99:59:59.999 --> 99:59:59.999 and have that actually persist and ideally detect 99:59:59.999 --> 99:59:59.999 that this is a problem in the first place, 99:59:59.999 --> 99:59:59.999 so that we don't have to ask an incomprehensible question in the installer. 99:59:59.999 --> 99:59:59.999 Now, more cheerily, a number of non-x86 architectures are in a pretty good state. 99:59:59.999 --> 99:59:59.999 they get fairly limited testing at the moment. 99:59:59.999 --> 99:59:59.999 I know that some people use them because occasionally 99:59:59.999 --> 99:59:59.999 I get some bugs when things break but generally they work OK. 99:59:59.999 --> 99:59:59.999 powerpc and sparc work fine, we sould probably consider 99:59:59.999 --> 99:59:59.999 switching the default bootloader there at some point. 99:59:59.999 --> 99:59:59.999 If you're a porter for those architectures, 99:59:59.999 --> 99:59:59.999 please get in touch with me. 99:59:59.999 --> 99:59:59.999 Some mips-type architectures would well too. 99:59:59.999 --> 99:59:59.999 Ryan ?? lent me a Yeeloong laptop a little while back, 99:59:59.999 --> 99:59:59.999 so that I could port the debian installer to it 99:59:59.999 --> 99:59:59.999 and I found that grub was basically fine. 99:59:59.999 --> 99:59:59.999 So I was able to build the installer on top of that. 99:59:59.999 --> 99:59:59.999 It was a tenth of the work it might otherwise have been. 99:59:59.999 --> 99:59:59.999 There's been a pretty pleasing trend towards having 99:59:59.999 --> 99:59:59.999 new arches include a grub port very early on. 99:59:59.999 --> 99:59:59.999 arm64, also powerpc64, has had grub working from very early in it's life. 99:59:59.999 --> 99:59:59.999 This gives them a pretty full-feature loader 99:59:59.999 --> 99:59:59.999 with not a lot of effort, so it actually 99:59:59.999 --> 99:59:59.999 works out quite well for the ports now, I think. 99:59:59.999 --> 99:59:59.999 The most recent arm64 port, I went back and looked at it, 99:59:59.999 --> 99:59:59.999 it was about 2000 lines of patches to do the initial enablement. 99:59:59.999 --> 99:59:59.999 And now that got to take the advantage of the arm port 99:59:59.999 --> 99:59:59.999 that already existed and also of UEFI code, 99:59:59.999 --> 99:59:59.999 so that helps a lot but still I think it 99:59:59.999 --> 99:59:59.999 indicates a pretty porting friendly design. 99:59:59.999 --> 99:59:59.999 OK, so if I haven't persuaded you already, 99:59:59.999 --> 99:59:59.999 why do we default to grub on debian x86? 99:59:59.999 --> 99:59:59.999 There are certainly other loaders, 99:59:59.999 --> 99:59:59.999 there's the venerable LILO, which still largely works fine. 99:59:59.999 --> 99:59:59.999 Syslinux and friends are actively maintained by some very smart people, 99:59:59.999 --> 99:59:59.999 including folks who maintain the Linux boot protocol 99:59:59.999 --> 99:59:59.999 and there's extlinux which is one of that family 99:59:59.999 --> 99:59:59.999 which you can use in the same kind of way that you might use LILO. 99:59:59.999 --> 99:59:59.999 There's yaboot and powerpc and so on and so forth. 99:59:59.999 --> 99:59:59.999 You can even boot a kernel and an initramfs directly 99:59:59.999 --> 99:59:59.999 from the UEFI environment, if you want to. 99:59:59.999 --> 99:59:59.999 So some of those are some pretty good bootloaders, 99:59:59.999 --> 99:59:59.999 they're almost all smaller and simpler than grub 99:59:59.999 --> 99:59:59.999 and that appeals to a lot of people. 99:59:59.999 --> 99:59:59.999 I do get the simplicity argument. 99:59:59.999 --> 99:59:59.999 But I tend to argue that the result of that 99:59:59.999 --> 99:59:59.999 is moving a good deal of complexity elsewhere. 99:59:59.999 --> 99:59:59.999 If you have a bootloader that can only handle some setups, 99:59:59.999 --> 99:59:59.999 the installer needs to know what those setups are. 99:59:59.999 --> 99:59:59.999 It needs to forbid you from doing anything, that you won't be 99:59:59.999 --> 99:59:59.999 able to boot, in your partitioner. 99:59:59.999 --> 99:59:59.999 You end up with static /boot partitions scattered everywhere. 99:59:59.999 --> 99:59:59.999 Having a bootloader that you can trust to handle almost anything 99:59:59.999 --> 99:59:59.999 means that you don't have to think too hard when you are moving things around. 99:59:59.999 --> 99:59:59.999 You can easily do things like having the bootloader notice 99:59:59.999 --> 99:59:59.999 when the last boot failed and behave differently. 99:59:59.999 --> 99:59:59.999 All that sort of thing is useful for a general purpose distro to be able to assume. 99:59:59.999 --> 99:59:59.999 [Ben]: We used to support some bootloaders on mips, 99:59:59.999 --> 99:59:59.999 but they couldn't load an initramfs, they could only load a kernel. 99:59:59.999 --> 99:59:59.999 So for that reason the kernel configurations we used on mips 99:59:59.999 --> 99:59:59.999 had to be restricted, they don't assume there is an initramfs 99:59:59.999 --> 99:59:59.999 and you can't boot off filesystems other than ext2, 3 and 4. 99:59:59.999 --> 99:59:59.999 [Colin]: That's CoLo, is it? Or cibyl? 99:59:59.999 --> 99:59:59.999 [Ben]: I don't know. We have recently switched over, 99:59:59.999 --> 99:59:59.999 but that has been a restriction up to and including wheezy. 99:59:59.999 --> 99:59:59.999 [Colin]: Thanks. That's useful ammunition. [laughs] 99:59:59.999 --> 99:59:59.999 I think that we do not yet have grub working on 99:59:59.999 --> 99:59:59.999 all of the mips big endian architectures, 99:59:59.999 --> 99:59:59.999 it's working on little endian, 99:59:59.999 --> 99:59:59.999 but I think there are still some problems with big endian. 99:59:59.999 --> 99:59:59.999 So we maybe can't save the world just yet, 99:59:59.999 --> 99:59:59.999 but it might not be too far off. But thanks. 99:59:59.999 --> 99:59:59.999 Plus Debian runs on lots of architectures, 99:59:59.999 --> 99:59:59.999 generally we try as an architectural thing 99:59:59.999 --> 99:59:59.999 to run the same software across different architectures so 99:59:59.999 --> 99:59:59.999 that it all works kind of the same way. 99:59:59.999 --> 99:59:59.999 We have the same interfaces, the same tools are available and so on. 99:59:59.999 --> 99:59:59.999 And it makes a lot of sense to have that be the case with the bootloader too. 99:59:59.999 --> 99:59:59.999 The installer would certainly be simplified by being able to 99:59:59.999 --> 99:59:59.999 assume grub-mount is available on all architectures, for instance. 99:59:59.999 --> 99:59:59.999 OK, so the important bit, this. 99:59:59.999 --> 99:59:59.999 We've had a lot of people involved over the years 99:59:59.999 --> 99:59:59.999 Too many to name, I've almost certainly missed some 99:59:59.999 --> 99:59:59.999 Robert Millan, Felix Zielcke and Jordi Mallach 99:59:59.999 --> 99:59:59.999 have done excellent jobs in Debian a few years ago. 99:59:59.999 --> 99:59:59.999 And Vladimir Serbinenko does an excellent job as the upstream maintainer. 99:59:59.999 --> 99:59:59.999 But as far as Debian maintenance goes, 99:59:59.999 --> 99:59:59.999 it's mostly just me at the moment and has been 99:59:59.999 --> 99:59:59.999 for the last couple of years and I can't do it all. 99:59:59.999 --> 99:59:59.999 We'd probably have had grub 2 in 2.00 99:59:59.999 --> 99:59:59.999 rather than 1.99 in wheezy, 99:59:59.999 --> 99:59:59.999 if we had a bit more redundancy there. 99:59:59.999 --> 99:59:59.999 I was kind of demotivated by the Secure Boot stuff at the time, 99:59:59.999 --> 99:59:59.999 so ended up putting it off until too late 99:59:59.999 --> 99:59:59.999 and that's exactly the kind of thing 99:59:59.999 --> 99:59:59.999 where having more people involved is more helpful 99:59:59.999 --> 99:59:59.999 there are lots of bugs, everybody's problem is critical for them, 99:59:59.999 --> 99:59:59.999 quite understandably. 99:59:59.999 --> 99:59:59.999 So keeping on top of the release critical list is absolute murder. 99:59:59.999 --> 99:59:59.999 But there are a few specific problems I'd like to highlight, 99:59:59.999 --> 99:59:59.999 that are more substantial and could really do with 99:59:59.999 --> 99:59:59.999 somebody sitting down and thinking quite hard. 99:59:59.999 --> 99:59:59.999 I won't go into too much detail, 99:59:59.999 --> 99:59:59.999 as I've just had a sign from the video team that I'm low on time. 99:59:59.999 --> 99:59:59.999 But you can ask me about any of them if you're interested. 99:59:59.999 --> 99:59:59.999 I mentioned earlier that the system for generating 99:59:59.999 --> 99:59:59.999 the config file has a bad case of second system effect 99:59:59.999 --> 99:59:59.999 and really needs a third system 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 to make it easier to customise things. 99:59:59.999 --> 99:59:59.999 All problems can be solved by another rewrite, right? 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 On MBR systems, we often have robustness problems 99:59:59.999 --> 99:59:59.999 with grub not installing the boot code to the place 99:59:59.999 --> 99:59:59.999 that your system actually boots from, 99:59:59.999 --> 99:59:59.999 particularly when multiple disks are involved. 99:59:59.999 --> 99:59:59.999 This tends to manifest as module loading failures 99:59:59.999 --> 99:59:59.999 because what happens is, over time 99:59:59.999 --> 99:59:59.999 the core image that you're actually using 99:59:59.999 --> 99:59:59.999 that you didn't know about, becomes incompatible with new grub modules 99:59:59.999 --> 99:59:59.999 because the ABI changes over time 99:59:59.999 --> 99:59:59.999 and eventually becomes incompatible and can't boot anything 99:59:59.999 --> 99:59:59.999 I think that needs some kind of overhaul somewhere, 99:59:59.999 --> 99:59:59.999 though I'm not quite sure where. 99:59:59.999 --> 99:59:59.999 There are the remaining bit of UEFI, 99:59:59.999 --> 99:59:59.999 which I mentioned earlier. 99:59:59.999 --> 99:59:59.999 There's getting signing sorted out in Debian, 99:59:59.999 --> 99:59:59.999 getting a thing called MokManager, 99:59:59.999 --> 99:59:59.999 which is part of a seperate project called shim, 99:59:59.999 --> 99:59:59.999 nicely integrated to give users control over the whole signing process 99:59:59.999 --> 99:59:59.999 which will let us require signed kernels under Secure Boot 99:59:59.999 --> 99:59:59.999 and thus make Matthew Garrett want to kill me a little bit less 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 and there's the issues with EFI System Partition layout 99:59:59.999 --> 99:59:59.999 There's some work to be done with upstream Xen 99:59:59.999 --> 99:59:59.999 to layout effectively a new boot protocol for PV Grub 2 99:59:59.999 --> 99:59:59.999 so we can have Xen guests install the bootloader 99:59:59.999 --> 99:59:59.999 in a consistent place in the file system. 99:59:59.999 --> 99:59:59.999 In fact, Ian Campbell recently sent me something that 99:59:59.999 --> 99:59:59.999 might be useful for this, I didn't have time to read it. 99:59:59.999 --> 99:59:59.999 and hopefully this will eventually let us kill off all 99:59:59.999 --> 99:59:59.999 of the old strange things that live in Xen, to do this. 99:59:59.999 --> 99:59:59.999 Finally, we should take much better advantage of grub's ports 99:59:59.999 --> 99:59:59.999 and switch over to them on many more architectures than we have done to date. 99:59:59.999 --> 99:59:59.999 I've just got onto questions, so excellent. 99:59:59.999 --> 99:59:59.999 [Ian]: One of things I've found difficult getting to grips with grub 1 99:59:59.999 --> 99:59:59.999 and grub 2 is that the documentation hasn't always been as good. 99:59:59.999 --> 99:59:59.999 LILO has always been exceptional in that it came with 99:59:59.999 --> 99:59:59.999 an extremely good document that described 99:59:59.999 --> 99:59:59.999 its principles of operation and how to get it to do, 99:59:59.999 --> 99:59:59.999 well, almost anything you might want it to. 99:59:59.999 --> 99:59:59.999 Is that also something you're looking for help with? 99:59:59.999 --> 99:59:59.999 [Colin]: Yes. 99:59:59.999 --> 99:59:59.999 About a year or two ago, 99:59:59.999 --> 99:59:59.999 I went through and tried to systematically documenting 99:59:59.999 --> 99:59:59.999 all of the grub commands and overhauling some other things. 99:59:59.999 --> 99:59:59.999 I got about half way through that, 99:59:59.999 --> 99:59:59.999 before I ran out of time. 99:59:59.999 --> 99:59:59.999 but one of the problems has been with grub 2 99:59:59.999 --> 99:59:59.999 that the documentation got out of date during the rewrite 99:59:59.999 --> 99:59:59.999 so nobody has an incentive to do it properly for their incremental changes. 99:59:59.999 --> 99:59:59.999 So I think getting that into shape 99:59:59.999 --> 99:59:59.999 would make it much easier to keep it in shape 99:59:59.999 --> 99:59:59.999 It's better than it was in 2010, it's not where it should be. 99:59:59.999 --> 99:59:59.999 So, yes. 99:59:59.999 --> 99:59:59.999 Anybody else, anything from irc aswell? 99:59:59.999 --> 99:59:59.999 [John]: This was a related question to your comment earlier 99:59:59.999 --> 99:59:59.999 about how BIOSes are widely inconsistent about 99:59:59.999 --> 99:59:59.999 how they choose which disk they're going to boot from. 99:59:59.999 --> 99:59:59.999 Is there anything that grub could do to help us 99:59:59.999 --> 99:59:59.999 with best practices for failover configurations? 99:59:59.999 --> 99:59:59.999 So for example, let's say I have a server 99:59:59.999 --> 99:59:59.999 and I want to install two disk that are mirrored, 99:59:59.999 --> 99:59:59.999 using lvm or something else. 99:59:59.999 --> 99:59:59.999 One of the challenges is, how do I configure grub 99:59:59.999 --> 99:59:59.999 on both disk to be configured the right 99:59:59.999 --> 99:59:59.999 way, despite all of the weirdness with BIOSes? 99:59:59.999 --> 99:59:59.999 To have a reasonable easy to maintain setup, 99:59:59.999 --> 99:59:59.999 so that where if one disk actually goes completely offline, 99:59:59.999 --> 99:59:59.999 the system could automatically reboot. 99:59:59.999 --> 99:59:59.999 [Colin]: One of the things I tried to do, 99:59:59.999 --> 99:59:59.999 the first time I systematically 99:59:59.999 --> 99:59:59.999 attacked this in Debian grub, 99:59:59.999 --> 99:59:59.999 the thing I tried to do was to arrange that we 99:59:59.999 --> 99:59:59.999 did as by-uuid install to disks, possibly as by-path, I forget. 99:59:59.999 --> 99:59:59.999 So we encourage people to install to all of the disks on a system. 99:59:59.999 --> 99:59:59.999 Also the grub-pc.postinst notices when a disk has gone away since it last checked, 99:59:59.999 --> 99:59:59.999 and it's supposed to offer you the chance to update this. 99:59:59.999 --> 99:59:59.999 But you're right it's a little bit fiddly to manage 99:59:59.999 --> 99:59:59.999 particularly if you're in a system that's large enough 99:59:59.999 --> 99:59:59.999 so that statistically disks are failing quite often 99:59:59.999 --> 99:59:59.999 Then it's a problem. 99:59:59.999 --> 99:59:59.999 I think what I'd probably like to do is to make sure that 99:59:59.999 --> 99:59:59.999 we install to the disks that are associated with a RAID device or something 99:59:59.999 --> 99:59:59.999 so you could just bootstrap off the RAID management. 99:59:59.999 --> 99:59:59.999 I think some of that works but isn't hooked up very well in Debian. 99:59:59.999 --> 99:59:59.999 But there are several things to do there. 99:59:59.999 --> 99:59:59.999 I think there's also a bug in d-i where it doesn't install 99:59:59.999 --> 99:59:59.999 to all of the fixed disks in your system but only one. 99:59:59.999 --> 99:59:59.999 So there are several things to fix there. 99:59:59.999 --> 99:59:59.999 There's a question over there. 99:59:59.999 --> 99:59:59.999 [??]: I think my question could be summed up as: 99:59:59.999 --> 99:59:59.999 "How tall do you have to be to ride this ride?" 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 It seems like getting started, 99:59:59.999 --> 99:59:59.999 introducing regressions would be a really scary thing. 99:59:59.999 --> 99:59:59.999 For someone who wanted to get started with the project 99:59:59.999 --> 99:59:59.999 and I'm just wondering what kind of facilities do you have 99:59:59.999 --> 99:59:59.999 for before we actually upload a version to 99:59:59.999 --> 99:59:59.999 test across the broad range of architectures. 99:59:59.999 --> 99:59:59.999 [Colin]: Right well, one of the points of this talk 99:59:59.999 --> 99:59:59.999 was to try to emphasise that you don't actually have 99:59:59.999 --> 99:59:59.999 to be as tall as people think you do. 99:59:59.999 --> 99:59:59.999 But you're right about the need for regression testing. 99:59:59.999 --> 99:59:59.999 One thing that really helps is that the different parts 99:59:59.999 --> 99:59:59.999 of grub are much more isolated than they used to be. 99:59:59.999 --> 99:59:59.999 So you can quite easily change one module, 99:59:59.999 --> 99:59:59.999 test changes with that, without having to worry about 99:59:59.999 --> 99:59:59.999 strange leakage all over the place. 99:59:59.999 --> 99:59:59.999 But there are things that that doesn't cover, 99:59:59.999 --> 99:59:59.999 there is some boot testing arrangement in the build system, 99:59:59.999 --> 99:59:59.999 so there are some make targets that go off 99:59:59.999 --> 99:59:59.999 and do boot checks of real systems against the current version of the code. 99:59:59.999 --> 99:59:59.999 And that sort of thing helps a lot, 99:59:59.999 --> 99:59:59.999 I would like to have grub hooked up better to some sort of autopkgtest arrangement 99:59:59.999 --> 99:59:59.999 so that it does emulated boots across various types 99:59:59.999 --> 99:59:59.999 of systems on ci.debian.net, I'd love help with that. 99:59:59.999 --> 99:59:59.999 [Steve]: Now of course UEFI, there's a whole slew of things. Tom was asking 99:59:59.999 --> 99:59:59.999 about MBR and a reliable fallback, of course. I'm not aware of any UEFI 99:59:59.999 --> 99:59:59.999 implementation that actually does fallback for the system partition either. 99:59:59.999 --> 99:59:59.999 So that is an utter trainwreck. 99:59:59.999 --> 99:59:59.999 [Colin]: I suspect what you're, 99:59:59.999 --> 99:59:59.999 maybe you're meant to use hardware RAID or something? 99:59:59.999 --> 99:59:59.999 [Steve]: Clearly that's what you're expected to use and it's a mess. 99:59:59.999 --> 99:59:59.999 We should definitely get together 99:59:59.999 --> 99:59:59.999 and talk about the removable media path. 99:59:59.999 --> 99:59:59.999 Again there isn't a good answer but let's try 99:59:59.999 --> 99:59:59.999 and make it as not crap as we can. 99:59:59.999 --> 99:59:59.999 [Colin]: Do you know of any way to detect that 99:59:59.999 --> 99:59:59.999 system even if it's something like a blacklist of broken DMI decoders? 99:59:59.999 --> 99:59:59.999 [Steve]: I think that's all we can do. 99:59:59.999 --> 99:59:59.999 We had a system, as an example 99:59:59.999 --> 99:59:59.999 for the people that may not have come across this, 99:59:59.999 --> 99:59:59.999 there are firmware implementiatons out there 99:59:59.999 --> 99:59:59.999 that straight after install may not actually recognise 99:59:59.999 --> 99:59:59.999 that you've put grub into the path where you're meant to put it. 99:59:59.999 --> 99:59:59.999 Once you've booted off a rescue system, however, 99:59:59.999 --> 99:59:59.999 and done some not particularly well understood sequence of things, 99:59:59.999 --> 99:59:59.999 including boot off a removable media path, reboot a couple more times 99:59:59.999 --> 99:59:59.999 shuffle things around, spin twice clockwise... 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 suddenly they then will recognise that you've put grub there. 99:59:59.999 --> 99:59:59.999 But god forbid, you ever want to update it again. 99:59:59.999 --> 99:59:59.999 It's painful. 99:59:59.999 --> 99:59:59.999 [Colin]: There are some exciting problems on Apple Macs, 99:59:59.999 --> 99:59:59.999 that are a shadow of the same class of problem. 99:59:59.999 --> 99:59:59.999 [Steve]: And of course Secure Boot, 99:59:59.999 --> 99:59:59.999 kibi asked Steven about this a few days ago. 99:59:59.999 --> 99:59:59.999 Of course, we've been talking about this for like 2 years. 99:59:59.999 --> 99:59:59.999 I feel rubbish as well, that we haven't really got any further with it. 99:59:59.999 --> 99:59:59.999 If someone really really wants to help get involved with that, oh absolutely, 99:59:59.999 --> 99:59:59.999 there's a whole load of people who would love to see you help. 99:59:59.999 --> 99:59:59.999 [Colin]: Not only are there are a whole load of people who would like to see you help, 99:59:59.999 --> 99:59:59.999 but there's actually roughly a roadmap of what needs to be done. 99:59:59.999 --> 99:59:59.999 You don't have to blaze a trail. 99:59:59.999 --> 99:59:59.999 [??]: Scrape the names off the roadmap and put your name on it. 99:59:59.999 --> 99:59:59.999 [laughter] 99:59:59.999 --> 99:59:59.999 [Colin]: It's fine, we don't really care, we just want to see it done! 99:59:59.999 --> 99:59:59.999 The things that need to be done are, as I say, 99:59:59.999 --> 99:59:59.999 the next step I think is in dak and then 99:59:59.999 --> 99:59:59.999 somebody needs to chase up getting shim into debian 99:59:59.999 --> 99:59:59.999 with the right kind of signatures on it and so on 99:59:59.999 --> 99:59:59.999 but none of it is actually fundamentally hard, 99:59:59.999 --> 99:59:59.999 it just requires time to push all of that forward. 99:59:59.999 --> 99:59:59.999 [Steve]: Exactly. I'll pass onto anybody else. 99:59:59.999 --> 99:59:59.999 [video team]: Actually, we're out of time. 99:59:59.999 --> 99:59:59.999 [Colin]: OK, thank you all very much. 99:59:59.999 --> 99:59:59.999 [applause] 99:59:59.999 --> 99:59:59.999