Chapter 21. 16h00 - Keynote Address

Keynote Address

Dave Jones (with introduction from Andrew Morton)

...so there I was, all set in my seat... comfy, ready for the talk... and then they kicked us all out :-(

-mm

As is customary, last year's keynote speaker (i.e. Andrew Morton) introduced this year's keynote speaker. Andrew started out by thanking the distributions, especially the big ones like Redhat. Of course this drew a bit of silence from the audience and bit of shuffling-around-in-chairs. He then pointed out that thanks to them, they employ dozens and dozens of developers and sends them cheques which allows them to work on Linux projects full-time.

Andrew then pointed out that, according to the best numbers we have, roughly 50% of Linux users use a distribution that is, or is based on, Redhat. David is in charge of the quality of the Redhat kernel; therefore David is in charge of the quality of the Linux experience for somewhere around 50% of all Linux users. Andrew basically wanted to take a step back and thank David for all his hard work:

• his forward porting of features from 2.4.x to 2.5.x in the early days of the 2.5.x series,

• as well as all his current kernel work for Redhat.

Andrew then made us all aware of how close we came to losing David and his fine work. He brought with him an email that David received... from someone named "root". The essence of the letter went something like this:

Dear Sir, You have no right to call me a ******. Please apologize or I will write your employer and have you fired.

In the words of Andrew: thankfully Redhat could see through this dilemma and decided to keep David on. With that Andrew removed his tongue from his cheek and it was David's turn.

-dj

David started by thanking Andrew. He talked about how, on the flight over, he was thinking about all the things he was going to talk about during the keynote. Then he attended Jonathan Corbet's talk on the first day... "I was going to talk about that... hmm, inotify, I was going to talk about that... hmm, realtime, I was going to talk about that...". So he ended up having to slap something together through the week.

Then David reminisced about how much has changed in the last 12 months, and how, 12 months ago, he never thought he'd be in this position. Then he ended up moving to the States... (he put up a picture of barren ice fields for as far as the eye can see). He then took this opportunity to relieve a grudge he's been carrying around with him for the last 15 years when one of his professors told him he had "no future in computer science". On the subject of "people being wrong" he also took the opportunity to quote Linus as having once said: "...all interesting stuff takes place in userspace, the kernel is [or will soon be?] stable and therefore boring"

David started talking about the early days of 2.5.x and how the development process started causing the kernel to fall apart. There were problems with:

• module loader screwing up

• many interface changes without backward compatibility

• many userspace programs needed modification to keep in sync due to changes in the kernel

Throughout all this process there was nowhere near enough testing (regression or otherwise) being done. Why?:

• too many drivers, not enough maintainers

• an explosion of hardware combinations (architectures and hardware combinations)

• many people have pointed out how wonderful it now is that Linus has started using tools to help him do his job (Bitkeeper and git) but interestingly enough David pointed out the fact that now that Linus had a new system, it allowed him to accept and merge a lot more code without him actually having to look very closely at the submitted code

• too many lieutenants not being diligent and doing their jobs completely

• this new accelerated patch submission system ensured that the kernel was changing way too fast for anyone to hope to do any serious testing with each patch, one at a time

David pointed out that thanks to people like the LTP folks we do have a fair amount of testing being done such as:

• stress testing and

• performance measurement.

What's not being done, however, are:

• regression tests

• error paths and

• code coverage.

He then pointed out the situation we find ourselves in today where kernels are only tested after they're released. In summary for this section he pointed out how the new development model works better, the need for more regression testing, and the need for faster and better feedback from customers.

Using that last point he segued into his next topic: The Redhat Bugzilla. It's not perfect, but it's really good. On the one hand it's sometimes hard to get users to be interested enough to fill out bug reports, on the other hand, the vast majority of the bug reports don't have anything to do with Redhat per se, but are problems with software that should be moved further upstream (i.e. KDE, GNOME, etc...).

One of the reasons why the Redhat bugzilla isn't perfect is because there is no mechanism available for automatic forwarding of bugs to upstream bugzillas. What would work well would be either:

1. a user selecting a component would actually enter the bug in the upstream bugzilla, or

2. there would be an easy way for a user to move a bug to another bugzilla (i.e. after it has already been entered into the current bugzilla).

As it stands now this is an entirely manual process. Fortunately there are some drones around who help David out with this effort. David speculated that perhaps xmlrpc might be the way to go on this one. In any case, this is another example of where and why better tools are needed.

Reporting bugs has to be a near zero-effort task on the part of the user. If they find a lot of related small bugs, asking them to fill out a unique bug report for each one quickly becomes a waste of the user's time. Another impediment for some users is the fact that different projects use different bug tracking systems, meaning a user potentially has to learn how to use several different tools. On the one hand it would be nice if users didn't need accounts, but that would too quickly be abused; not all users are going to want to create accounts on several different websites, using different bug-tracking tools, in order to report a couple of bugs. Work needs to be done in this area.

Although grateful for the bugs that are submitted, just because a user is motivated enough to submit a valid bug, doesn't mean the process is painless from then on:

• Sometimes users fib. Users have come to learn that nobody will help them or take them seriously if their kernel is tainted (and with good reason!). Therefore users have started editing the bug tracing output before submitting it. To counter this activity Redhat kernels have started adding hard-to-decipher hash values in their output, values which tell investigators the true nature of what's going on.

• Bugzilla allows a user to set the priority of the bug, but guess what? Users always consider their bug to be, no matter what it is, a high priority. A user-assignable priority field on a bug is useless.

• Users aren't interested in upstream. You can tell a user the bug is really an upstream issue, you can ask them to move it upstream, you can plead, but for the most part users won't bother sending the report upstream.

• There are a number of "hit and run" bug reports. People who take the time to register and create a bug, but are never found again. Often the person looking at a bug might need more information, if it isn't possible to get in touch with the person reporting the bug, there's probably no point in filing the bug in the first place.

• Some users provide too much information, some users don't provide enough information; both cases are bad. David pointed out that this is perhaps a place where the tools might help. [I don't think he's aware of a mechanism in bugzilla called guided bug entry. Guided bug entry leads a user though a series of pages asking specific questions along the way; it helps ensure that bugs have the right information.]

• Then there's one of David's favourites: the person who refuses to upgrade.

  customer:	I've discovered such-and-such a bug.
  david:	That's been fixed in version 2.6.x+1
  customer:	I don't wanna upgrade, can't you just fix it in my version?

At this point David went through a whole bunch of Redhat Bugzilla statistics.

One of the patterns David has noticed of late in the Redhat Bugzilla is that the number of problems with binary-only modules has been dropping off... right about the same time and the number of bugs with binary-only module helpers started to increase.

At this point David started in on his third and final major topic: Tools! Tools, tools, tools, tools.

He started off by pointing out: "how many times are you attending a session, a BOFS, or a tutorial here at OLS... someone asks a question and the answer is 'we don't know, we don't have a tool to find out'?" There are lots of changes that happen to the kernel on an hourly basis, it would be impossible for anyone to read all the changes, know how it's going to affect the kernel, and follow everything happening on LKML. We need better tools to help. Programming to the kernel interfaces has to be idiot-proof, there are roughly 40 different locking primitives in the kernel, and functions need to be type-safe.

Overall the same percentage of people that are using tools now are the same percentage that has always used them. It doesn't seem like people are using tools any more than they were in the past. Some of the tools David listed include:

• valgrind

• gprof

• j/kprobes, dprobes, systemtap

• oprofile

• readprofile

David then talked about other techniques for finding bugs, such as static analysis. The kernel's number of lines of code is going up fast, and getting faster. Static analysis tools help find bugs faster than eyeballs can. He also pointed out that although there are some tools available, not all of them work very well out-of-the-box; they need to be tweaked (e.g. lint). Static analysis tools also suffer from the fact they can produce lots of false positives; even when configured reasonably well. Additional tools he pointed out include:

• sparse

• gcc -DFORTIFY_SOURCE explaination

• http://www.spinroot.com/static/

He pointed out to the crowd that many security bugs have been found over the years thanks to tools like these.

Overall David pointed out that the quality of the Linux kernel source is down. He pointed out the Microsoft noticed this problem with their OS product years ago and in response developed, and made available to its developers, dozens of tools so that developers didn't have to make so many mistakes. Their solution was to focus on better developer tools.

The final thing David wanted to say was: "I like monkeys". That's what he said. He even had a slide with a picture of several moneys around his computer stuff.

With that David was done his talk. There were a couple interesting questions, the most interesting was that someone wanted him to explain his monkey/UFO story. So, amid some embarrassment, David told the monkey/UFO story:

One night he was sleeping and was having a bad dream. He inadvertently woke up his wife, who woke him up and asked him what the problem was. To which he replied: "My monkey crashed my UFO!"

postscript...

Oh yea, as I mentioned before... there I was all comfy when they kicked us out for the keynote. The reason they did that was because Tim Bird had some prizes to give away on behalf of the CE Linux Forum. The way they did it was they placed a sticker on the under-side of three of the chairs. At the end of David's speech Tim came up and told everyone to look under their seats. The winners got some electronics toys of some sort.