WEBVTT 00:00:00.000 --> 00:00:03.650 maintains still some of the flat files in our system 00:00:03.650 --> 00:00:05.320 that should increase speed 00:00:06.140 --> 00:00:07.490 the flat files of our system 00:00:09.600 --> 00:00:11.090 that my primary motivation 00:00:11.100 --> 00:00:12.190 for giving a talk was 00:00:12.310 --> 00:00:13.800 to try to force myself 00:00:13.800 --> 00:00:13.920 into actually doing to try to force myself 00:00:13.920 --> 00:00:15.500 into actually doing 00:00:15.500 --> 00:00:16.560 the work that I'm going 00:00:16.560 --> 00:00:18.340 talk about in this talk [laughts] 00:00:18.340 --> 00:00:19.980 ???? 00:00:19.980 --> 00:00:20.830 lightly successful 00:00:20.830 --> 00:00:22.830 so I may introduce you some of them, but??? 00:00:23.930 --> 00:00:25.570 become too much 00:00:25.570 --> 00:00:27.570 of the professional procrastinator 00:00:27.570 --> 00:00:30.230 to being influenced by even giving a talk 00:00:32.229 --> 00:00:34.230 Let's see how this work out 00:00:35.110 --> 00:00:37.820 The first couple of things I'm going to talk about 00:00:37.820 --> 00:00:39.990 are just some general bugs statistics 00:00:39.990 --> 00:00:43.520 I'll enjoy showing some plots 00:00:43.520 --> 00:00:47.290 I'm going to introduce the basic arquitecture of debbugs 00:00:47.290 --> 00:00:50.300 just to backup, debbug is 00:00:50.300 --> 00:00:53.260 the system behind bugs are debian debug 00:00:53.260 --> 00:00:55.600 so if you file the bug 00:00:55.600 --> 00:00:57.970 you fix the bug 00:00:57.970 --> 00:01:00.640 you wonder where bugs exists 00:01:00.640 --> 00:01:02.340 you interact with that bugs 00:01:02.340 --> 00:01:05.430 ???? 00:01:05.430 --> 00:01:10.290 I'm going to talk about some of the new features 00:01:20.630 --> 00:01:23.330 I'm going to talk about some planned features 00:01:23.330 --> 00:01:26.110 specially features that won't happen If I suddenly 00:01:26.110 --> 00:01:28.630 have two or three more people would help 00:01:28.630 --> 00:01:31.520 and I hopefully would pleased with 00:01:31.520 --> 00:01:33.720 of you or people who are missing 00:01:33.720 --> 00:01:35.080 this talk online 00:01:35.080 --> 00:01:38.020 or who may watch this talk recording later 00:01:38.020 --> 00:01:40.730 to assist me in implementing some of these features 00:01:40.730 --> 00:01:42.230 I'm a nice person 00:01:42.230 --> 00:01:44.840 we are nice people 00:01:44.840 --> 00:01:47.470 and I'd like to see any pro hackers 00:01:47.470 --> 00:01:50.130 to see a css hacker or javascript hackers 00:01:50.130 --> 00:01:52.130 or people who writes documentation helps 00:01:56.790 --> 00:01:59.710 the goal of the BTS is to 00:01:59.710 --> 00:02:01.310 report bugs 00:02:01.310 --> 00:02:03.310 track the evolution of bugs 00:02:03.310 --> 00:02:04.330 fix bugs 00:02:04.330 --> 00:02:06.920 and hopefully reduce the impact of bugs 00:02:06.920 --> 00:02:09.479 ??? 00:02:11.220 --> 00:02:16.240 this is how many bugs do we have versus time 00:02:16.240 --> 00:02:19.500 as you can see our bug growth is 00:02:19.500 --> 00:02:21.490 roughly linear over time 00:02:21.490 --> 00:02:23.490 and actually decreasing slowly but 00:02:23.490 --> 00:02:26.100 that a huge number of bugs 00:02:26.100 --> 00:02:29.940 people like to track exactly how many bugs we have 00:02:29.940 --> 00:02:31.940 ??? 00:02:31.940 --> 00:02:33.800 fun contest 00:02:33.940 --> 00:02:38.180 guessing when the ticket bug will be filed fo ar example 00:02:38.180 --> 00:02:41.940 the 760 000 00:02:41.940 --> 00:02:44.270 bug will be filed 00:02:44.270 --> 00:02:46.270 at I think 00:02:46.270 --> 00:02:48.010 on September 2 00:02:48.010 --> 00:02:53.150 and the 800 000 will be filed almost a year from now 00:02:53.150 --> 00:02:54.700 on September 13 00:02:54.700 --> 00:02:56.700 observing the linear progression 00:02:56.700 --> 00:02:58.700 maintains itself 00:02:58.700 --> 00:03:01.450 ??? enjoys that but he is not here 00:03:01.450 --> 00:03:03.250 ?? 00:03:03.260 --> 00:03:05.450 Anyway that's to show the bug reporting rate 00:03:05.450 --> 00:03:08.740 We average roughly have a 42 bugs found by day 00:03:10.190 --> 00:03:13.300 as you see, that's a huge number bugs 00:03:13.300 --> 00:03:18.930 This is the bug closing graph 00:03:18.930 --> 00:03:23.870 It's actually taking not bug closures. It's actually bugs being archetyped ¿? 00:03:23.870 --> 00:03:27.250 but for the most part it's aproximated 00:03:27.250 --> 00:03:30.870 bugs closure rate ??? 00:03:30.870 --> 00:03:35.700 So, we close roughly 95 bugs per day 00:03:35.700 --> 00:03:39.950 So from that you could imagine that the bug system is gaining 00:03:39.950 --> 00:03:44.370 50 or so much bugs everyday that are not going to be fixed 00:03:44.370 --> 00:03:48.750 Further than fortunately in this graph you can see that 00:03:48.750 --> 00:03:52.610 the bug closing rate is decreasing 00:03:52.610 --> 00:03:57.610 in contrast with the bug reporting rate awesome increasing 00:03:57.610 --> 00:03:59.020 something that i've seen 00:03:59.020 --> 00:04:03.820 in previous post I'v made on my blog 00:04:03.820 --> 00:04:05.680 this is actually kind of disturbing 00:04:05.680 --> 00:04:08.340 I'm not sure what that means for Debian as a whole 00:04:08.340 --> 00:04:10.660 what means anything but 00:04:10.660 --> 00:04:14.240 I'd much rather seen the overall rating increasing 00:04:14.240 --> 00:04:17.519 than decreasing 00:04:18.560 --> 00:04:21.440 In this graph you are familiar with 00:04:21.440 --> 00:04:24.290 this is RC bugs from the bug ?? 00:04:24.290 --> 00:04:30.210 lukely the RC bugs are those that matter for the next relase are decreasing 00:04:32.980 --> 00:04:36.420 getting in line for a new release there 00:04:36.420 --> 00:04:38.420 ?? 00:04:39.260 --> 00:04:42.400 Ok. So those is enough on graphs. 00:04:42.400 --> 00:04:45.960 Now I'm going to talk about 00:04:45.960 --> 00:04:47.960 debbugs system and how it works 00:04:49.510 --> 00:04:52.140 debbugs two main components 00:04:52.140 --> 00:04:54.140 there is a mail backend 00:04:54.140 --> 00:04:59.900 wich is what interact with any email 00:04:59.900 --> 00:05:01.900 submits bugs ?? 00:05:01.900 --> 00:05:04.540 or a bug number in bugs at ?? 00:05:04.540 --> 00:05:08.020 Next system on deb ?? 00:05:08.020 --> 00:05:12.850 wich has and old files and process you email ¿? 00:05:14.630 --> 00:05:17.580 The other aspect in debbugs is a web project 00:05:17.580 --> 00:05:24.900 That is what displays information on what bugs are on wich packets 00:05:24.900 --> 00:05:27.440 and the bug ?? 00:05:27.440 --> 00:05:29.440 and it's also here 00:05:29.440 --> 00:05:31.500 on to another machine 00:05:31.500 --> 00:05:36.690 on beach so that ?? faster ?? 00:05:38.000 --> 00:05:40.550 that bugs interact with dak 00:05:40.550 --> 00:05:44.280 wich is a software wich is responsable from maintaining the ?? 00:05:44.280 --> 00:05:46.500 so duk tells debbugs 00:05:46.500 --> 00:05:49.140 wich who maintains wich packages 00:05:51.000 --> 00:05:53.140 so debbugs know who is ?? 00:05:53.140 --> 00:05:54.790 ?? 00:05:54.790 --> 00:05:58.770 It also tells debbugs wich packages are in wich suite 00:05:58.770 --> 00:06:01.110 and wich arquitecture 00:06:01.110 --> 00:06:07.880 so debbugs can calculate wether a bug is present in a particular suite 00:06:07.880 --> 00:06:14.720 for example if the bug is fixed on stable or whether it's fixed in testing oin r stable 00:06:14.720 --> 00:06:17.690 what was calculated by previous ?? 00:06:19.690 --> 00:06:22.010 britney also 00:06:22.010 --> 00:06:25.010 is the testing migration 00:06:25.010 --> 00:06:28.150 software that migrates software from unstable 00:06:28.150 --> 00:06:29.340 to testing 00:06:29.340 --> 00:06:32.180 it uses information from the ¿buxtehude? as well 00:06:32.180 --> 00:06:36.900 it regards to wether a package is becoming more buggy 00:06:36.900 --> 00:06:38.060 or less buggy 00:06:38.060 --> 00:06:40.060 by operating 00:06:41.030 --> 00:06:43.440 the actual ?? that does that is 00:06:43.440 --> 00:06:46.340 ??? 00:06:46.340 --> 00:06:48.340 and it provides a list of bugs 00:06:48.340 --> 00:06:50.690 and also does the RC bug graphs 00:06:50.690 --> 00:06:53.640 but ir provides ???? 00:06:58.030 --> 00:07:00.590 debbugs ?? is like this 00:07:00.590 --> 00:07:02.940 mail comes in 00:07:02.940 --> 00:07:06.470 there is spam processing that happens on the first step ?? 00:07:06.470 --> 00:07:07.940 ?? 00:07:07.940 --> 00:07:12.380 ??? 00:07:13.810 --> 00:07:18.960 is primary responsable from keeping bugtrack system free of spam 00:07:18.960 --> 00:07:21.250 and these spamming 00:07:25.190 --> 00:07:29.270 ?? the little bit spam that actually make its way ?? 00:07:29.270 --> 00:07:40.770 ?? 00:07:40.770 --> 00:07:43.240 is not a great job 00:07:43.240 --> 00:07:46.130 after that mails been despammed 00:07:46.130 --> 00:07:48.670 then, it goes to processall 00:07:48.670 --> 00:07:50.870 and sub process 00:07:50.870 --> 00:07:53.580 is responsable from handleing 00:07:53.580 --> 00:07:55.580 emails that ? submit 00:07:55.580 --> 00:07:57.950 and emails that ?? bug numbers 00:07:57.950 --> 00:08:03.140 like this ??? 00:08:03.140 --> 00:08:07.110 service is responsable from handleing ?? 00:08:07.110 --> 00:08:10.930 so any email you send ?? 00:08:10.930 --> 00:08:14.760 now with the advent of 00:08:14.760 --> 00:08:16.990 control at submit¿? 00:08:16.990 --> 00:08:19.350 ???? 00:08:19.350 --> 00:08:25.110 this diagram is getting a little bit bluried. It's actually an abstraction ?? 00:08:25.110 --> 00:08:28.610 ??? 00:08:28.610 --> 00:08:33.659 then all of the information is stored in flat files 00:08:33.659 --> 00:08:37.419 and a db-h ? wich has 00:08:37.419 --> 00:08:40.330 small hash functions split ?? 00:08:40.330 --> 00:08:44.890 and is indexed with a couple of flat files indexes 00:08:44.890 --> 00:08:53.470 and then a cgi scripts use both the indexes and the flat files system to display bugs 00:08:57.500 --> 00:08:59.840 so that's how 00:09:01.230 --> 00:09:04.890 debbugs worked before I started working on it 00:09:04.890 --> 00:09:10.770 The current plan is to add on and basically replace this indexes 00:09:10.770 --> 00:09:12.770 with a databse layer 00:09:12.770 --> 00:09:19.120 and so I'm going to keep part of the flat files just because that's a well tested system 00:09:19.120 --> 00:09:22.150 there are lots of things already parts of flat files ¿? 00:09:22.150 --> 00:09:23.720 and I don't know how to deal with them ¿? 00:09:23.720 --> 00:09:27.820 and add on top a postgresql based database 00:09:27.990 --> 00:09:34.420 that, the cgi scripts ?? in order to display information ?? 00:09:34.420 --> 00:09:41.060 this will help both, increase the speed you get results back 00:09:41.060 --> 00:09:43.980 you are looking and ?? 00:09:43.980 --> 00:09:47.850 and also enable you to do more complicated things like 00:09:47.850 --> 00:09:51.960 ?? bugs that actually affects a particular version 00:09:51.960 --> 00:09:56.010 without waiting for huge amounts of time 00:09:56.010 --> 00:09:57.810 for a query to complete 00:09:57.810 --> 00:10:01.140 so for example, you want to look at all security bugs 00:10:01.140 --> 00:10:03.390 wich affects unstable 00:10:03.390 --> 00:10:06.040 well that's actually a really hard query to do 00:10:06.040 --> 00:10:08.310 without a database layer 00:10:08.310 --> 00:10:11.840 so that's one of the major things a database to do ?? 00:10:11.840 --> 00:10:16.870 so the script that actually handles loading things in the database is called dbugloader.sql 00:10:18.900 --> 00:10:20.630 Debbugs is run in parallel 00:10:26.290 --> 00:10:29.860 but perl has recently come quite a wise 00:10:30.080 --> 00:10:33.000 in handling databases 00:10:33.000 --> 00:10:35.930 Most everybody has adopted 00:10:35.930 --> 00:10:38.820 the perl ?? by using DBIx 00:10:38.820 --> 00:10:41.000 in order to talk to databases 00:10:41.000 --> 00:10:43.490 very successful database abstraction 00:10:43.760 --> 00:10:47.140 Anybody who has ever written code in DBIx 00:10:47.140 --> 00:10:49.140 knows that it is extremely tedious 00:10:49.140 --> 00:10:52.770 to do joing and complicated statements 00:10:52.770 --> 00:10:58.110 ?? writing sql and scaping and etc. 00:10:58.110 --> 00:11:00.110 using place orders 00:11:00.110 --> 00:11:02.110 ?? you have to keep track them 00:11:02.500 --> 00:11:05.600 DBA has classes and extensions 00:11:05.600 --> 00:11:09.000 that bloms together a huge number of 00:11:09.000 --> 00:11:14.310 ?? into a really concurrent database abstraction service 00:11:14.310 --> 00:11:17.120 where if you give it your schema 00:11:17.120 --> 00:11:22.770 It would build classes and enable you to talk to ?? result groups 00:11:22.770 --> 00:11:25.790 from your database 00:11:28.210 --> 00:11:29.460 It's a complete system 00:11:29.460 --> 00:11:33.300 you can actually write a schema entirely in DBIx class 00:11:33.300 --> 00:11:35.820 that you can then to SQL alike 00:11:35.820 --> 00:11:39.800 You can convert into a PostgreSQL, mySQL, etc 00:11:41.780 --> 00:11:45.140 In this case I'm interested in 00:11:45.140 --> 00:11:46.670 In write for PostgreSQL 00:11:46.670 --> 00:11:49.740 that actually will be the database package? 00:11:50.410 --> 00:11:53.910 there may be an option eventually to use SQL Lite for testing but 00:11:53.910 --> 00:11:57.880 my primary goal is to deploy PostgreSQL 00:11:58.390 --> 00:12:00.200 I'm also in using 00:12:00.200 --> 00:12:03.360 a bunch of classes that are specific to Debian 00:12:03.360 --> 00:12:07.950 for example the Debian Version Extension to PostgreSQL 00:12:07.950 --> 00:12:10.700 that enables you to sort by Debian's version 00:12:10.700 --> 00:12:13.770 That's extremely important to DBI ¿? 00:12:13.770 --> 00:12:17.610 And that is something that handles very well in PostgreSQL 00:12:17.610 --> 00:12:21.430 What I've actually done is rewrite ?? directing SQL 00:12:21.430 --> 00:12:24.880 and DBIx class has an extension called Schema Loader 00:12:24.880 --> 00:12:27.470 which handles converting the SQL schema 00:12:27.470 --> 00:12:32.660 into the class decorations for DBIx class automatically 00:12:32.660 --> 00:12:35.850 so you just write plain old SQL like you used to 00:12:35.850 --> 00:12:41.070 and it automatically creates all the database related 00:12:41.070 --> 00:12:45.040 Plurar classes that you used to talk results from database 00:12:45.790 --> 00:12:49.000 There is another module which handles deployment 00:12:49.000 --> 00:12:52.690 it can do automated upgrades from 00:12:54.040 --> 00:12:55.960 different schema revisions 00:12:55.960 --> 00:12:58.540 so as you change your schema 00:12:58.540 --> 00:13:01.080 it handles doing both upgrades 00:13:01.080 --> 00:13:03.200 and new installs of the new schema 00:13:03.200 --> 00:13:08.160 wich you can also do downgrades 00:13:08.160 --> 00:13:09.770 and yo can do others things 00:13:10.350 --> 00:13:14.100 in addition to just execute SQL alter statements you can also run 00:13:14.100 --> 00:13:17.310 ¿pro code? or anything else you want at the database 00:13:17.310 --> 00:13:19.000 IH operations¿? 00:13:19.190 --> 00:13:26.690 so that enables much easier changes to the schema in the future 00:13:27.150 --> 00:13:33.240 and finally the actual module in Debbugs ??? 00:13:33.240 --> 00:13:34.930 is DebBugs DB 00:13:34.930 --> 00:13:40.990 and so all of the database interactions classes in Debbugs ¿? 00:13:43.090 --> 00:13:46.020 and so this is the ..... 00:13:46.020 --> 00:13:49.810 It's kind of complicated but this basically tracks 00:13:49.810 --> 00:13:51.810 all of the bugs relationships 00:13:51.810 --> 00:13:55.030 It tracks who correspond with the DBs 00:13:55.030 --> 00:13:59.810 has all this source package versions imaginary packae versions 00:13:59.810 --> 00:14:05.290 version depences show for example when you uploaded version from a previous version 00:14:05.290 --> 00:14:10.120 this ables all that to be tracked 00:14:11.060 --> 00:14:14.830 Taken the DQL schema as 00:14:16.260 --> 00:14:18.600 as inspiration but unfortunately 00:14:18.600 --> 00:14:24.450 the debbugs ???? 00:14:24.450 --> 00:14:27.880 classes but I think it makes sense so 00:14:27.880 --> 00:14:31.390 Anyway if somebody uses PostgreSQL genius 00:14:31.390 --> 00:14:34.410 or an SQL hacker and is interested in 00:14:34.410 --> 00:14:39.720 maybe offering suggestions right ??? 00:14:39.720 --> 00:14:41.720 ¿talk about? 00:14:45.740 --> 00:14:47.500 Is actually pretty easy 00:14:47.500 --> 00:14:49.750 just called debbugs SQL bugs 00:14:49.750 --> 00:14:50.880 uploads them 00:14:51.070 --> 00:14:53.980 There are two different part of bugs in debbugs 00:14:53.980 --> 00:14:56.390 There is the one that you can actually modify 00:14:56.390 --> 00:14:58.390 and there are the archive bugs 00:14:58.390 --> 00:15:03.180 so this handles join both with both sets of bugs 00:15:03.180 --> 00:15:05.840 can also load versioning information 00:15:05.840 --> 00:15:09.030 this logs wich packages are depending on the ones 00:15:09.030 --> 00:15:12.910 and de Debian ¿? the architecture 00:15:12.910 --> 00:15:14.910 and source version 00:15:15.880 --> 00:15:19.120 So, the SQL is actually working 00:15:19.120 --> 00:15:22.960 ?? sql query 00:15:22.960 --> 00:15:25.550 which you also write this easy in a DBIx class 00:15:25.930 --> 00:15:28.880 let me show you that this actually works 00:15:45.530 --> 00:15:49.280 ??? 00:15:49.350 --> 00:15:54.370 I can run the SELECT statement wich is selecting the count of bugs WHERE 00:15:54.370 --> 00:16:00.720 which has been modify since June or July 00:16:00.720 --> 00:16:03.020 which are not done and 00:16:03.020 --> 00:16:05.510 and which are 00:16:07.710 --> 00:16:10.190 which are done and which there are an outer set 00:16:10.190 --> 00:16:12.190 and the answer is 00:16:13.210 --> 00:16:15.290 521 00:16:15.290 --> 00:16:19.880 as you can see that is a full load of all the bugs in Debian 00:16:19.880 --> 00:16:24.550 The actual SQL query executes very quick 00:16:24.550 --> 00:16:31.530 ?? 00:16:37.430 --> 00:16:41.340 I hopped to have more of this done by the time of this talk but 00:16:41.340 --> 00:16:43.340 There still is a lot of work that's needed 00:16:44.700 --> 00:16:47.480 The bugs file currently are not loaded 00:16:47.480 --> 00:16:51.270 the bugs files are ?? 00:16:51.890 --> 00:16:56.240 That's needed to enable full text searching index 00:16:56.240 --> 00:16:59.840 It also currently doesn't do ?? 00:16:59.840 --> 00:17:05.210 thats ?? faster loading ??