For Goodness’ Sake Apple, It’s Time to Open Up Bug Reporting
Update: While at WWDC I had the chance to talk to a few Apple engineers regarding the practice of "duping" radars as a way of upvoting certain issues. Despite my conclusion in the original post below that duplicating radars just floods the system with extra reports to process, it is in fact something that the engineers expect and encourage. I still think personally that it's a painfully inefficient approach for getting a sense of issue priorities, but it's better to use something that works poorly than to not give any feedback on priority at all! So I've updated some of the content below to reflect that.
Radar (internal name for Apple’s bug reporting and tracking system) is one of the most Orwellian names of any Apple product. The word implies a mechanism for detecting and maintaining visibility of something (defects) in a continuous and accurate manner. The reality is that it’s a jumbled mess of redundancy, fragmentation, silence and neglect. If there were a “Ministry of Quality” at Apple, its motto would likely be “Vigilance Through Obscurity” and its star product would be Radar.
There is a lot of debate over whether Apple’s software quality has been on the decline in the last couple years. I happen to have a firm conviction that it undeniably has been on the downswing and some prominent developers have publicly written on this decline already. But whether or not you believe that software quality is actually getting worse at Apple, the following facts cannot be disputed:
There are many, many bugs in iOS, OS X, their SDKs, and other Apple software. A good number of these are serious, and a lot have been around for a while now without being fixed, or have been “fixed”, but in fact still exist or were made worse.
In order to eliminate these bugs effectively, it is necessary to have them reported, and to gather the best and most information possible from the people who do find and report them. It is also necessary to track their status and to have enough people working on these bugs to fix them faster than they are created in order to make progress.
The reporting, information gathering, tracking and fixing doesn’t seem like such a hard thing for a company like Apple to do. In fact, pretty much every major software company from Google to Adobe have decent systems for managing these activities. But for reasons unknown (and almost certainly based on some misguided desire for more complete “secrecy”), Apple set up their system completely differently.
And it simply. does. not. work.
Oh Radar, how do I abhor thee? Let me count the ways:
Invisibility and Spam
It’s impossible to see if a particular bug or similar bug has already been filed by another developer. Why? Because Apple is perhaps the only major software company in the world that thinks it should not be possible for anyone outside the company to know what bugs exist or if they exist at all. If you sign into Radar, you can see the bugs you personally have filed, but you cannot see or search or be linked to any bug reported by anyone else ever. Period.
What does this mean? It means that tens(?) hundreds(?) thousands(?) of developers are all reporting the same bug over and over and over again. I can’t tell you how many times I’ve filed a bug in Radar and months later had it closed as a “duplicate of issue #xxxxxxx”. It’s actually such a high percentage of my bugs that end up this way, that I suspect that in some cases Apple employees are having to read and triage the same issue dozens of times. How on earth can the limited resource of engineers keep up with reported bugs if the system is so inefficient that a single bug has to be handled 20 times from 20 sources in the course of reviewing and fixing it? This process must literally drive Apple engineers crazy. Update: After speaking with some Apple engineers at WWDC, I got the impression that this process does not drive them crazy (even though it should! 😉) but has become part of how priority and importance for an issue are established. In other words, they would rather developers report the same bug over and over than have them not re-report it because someone else already did. This give them a sense of how many people find the issue important or problematic. I still maintain that being able to upvote and add comments to an existing issue is an order of magnitude more efficient and less frustrating for developers.
There is even a massive community attempt to workaround this horrible policy by having developers copy and past every bug they file with Apple into a big, searchable database at https://openradar.appspot.com
Unfortunately, there is no way for this database to be more than a small fraction of complete, and it is totally isolated from the actual Apple Radar system, which severely limits its ongoing accuracy and usefulness (although it is a noble effort and a heck of a lot better than the nothing which Apple provides us).
In a more perfect world, Open Radar might even allow developers to avoid spamming up the system with with duplicate bugs, but in reality many well-intentioned but misguided individuals Apple engineers themselves actively encourage developers to file duplicates of existing bugs as a workaround for horrible thing number two.
No One Gets A Vote
Using Apple's bug reporting tool, there is no way to vote for, “like" or otherwise indicate that you are also experiencing a specific issue. With a sane issue tracking system (you can literally buy a better one right off the shelf, Apple), users and developers can vote on reported bugs to indicate those which are the most pervasive, or in most urgent need of fixing.
A normal tracking system might have a 2000 issues, with vote counts on each ranging from 1 to 500, with perhaps an average of 15 votes per issue. In Apple’s system, people have resorted to“voting” by filing the same bug again and again. So instead of 2000 issues with an average of 15 votes each, Apple’s system would represent the same information as 30,000 separate issues, with about 93% of those issues being duplicate spam that have to be manually weeded through.
In reality, Apple’s number are much much bigger. There are minimally TENS of thousands of issues (Open Radar currently lists over 13,000 and this is just a fraction). And some of those issues are intentionally duplicated tens or hundreds of times. There is a Radar bug report filed about Radar itself being almost unusable, and according to this site, that single Radar has been duplicated at least 680 times alone.
So Apple’s bug system is likely awash in tens or hundreds of thousands of spam duplicate reports that have to be managed by their engineers, because someone thinks that way is better than letting reported bugs be visible and letting users and developers simply upvote the most important ones.
Single Source of Truth? How About Hundreds
In Radar, relevant information on a bug is scattered all over the place. Because there isn’t any way I can see an already reported bug and then add more information to it, my information is reported blindly as a partial duplicate. And who knows how many other developers are doing the same thing? So for any given issue, all the known observations, steps to reproduce, and accompanying logs are spread over many many separate and partially overlapping reports.
What’s even worse is that Apple handles these overlapping reports by closing all of them except one, which means that all the information and insight provided by multiple developers (much of it likely to be more useful than the information in the first version of the report which happened to be filed) is thrown away and never used to fix or improve the issue.
And I have a sneaking suspicion that in the process of triaging tens of thousands of duplicate reports, some of them may look similar to another at first glance, but in fact are different and yet are “closed as duplicate” anyway. Part of the reason I have this suspicion is because, again, there is no way at all for me to see the other bug reports that mine were supposedly duplicates of.
Slightly less frequent than the “bug closed because it’s a duplicate of another bug” response, is the “please provide a sample project demonstrating this bug” response. I’m sure that the vast majority of these sample projects and the time invested by developers in creating them are simply thrown away as part of the “duplicate of another bug” workflow. But what about the cases where a developer simply doesn’t have the time or inclination to create such a sample project? Well, nobody knows, but assumedly the bug remains unaddressed or gets closed in that case. However, if the bug was publicly visible to other developers, someone else interested in getting the issue fixed could step in and upload a sample project to move it forward, another advantage of allowing all relevant information to be collected in a single place. But alas, this is not allowed by Radar, which favors 50 blips all over a screen to indicate an enemy issue instead of just one blip (thank goodness Apple doesn’t do military contracting, right?).
A Goldmine of Silence
Perhaps my most hated thing of all about Radar is the most common response I receive to the issues I file reports for: complete and utter silence. For months, for years, forever. The largest percentage of my reported issues simply lie unaddressed and unacknowledged. I have no way of knowing if they were seen or not. I can’t even take comfort in knowing that other developers may be able to see them and benefit in some way. Instead, they gradually settle into a state that matches that of the entire Radar system in which they reside: total uselessness.
The problems with Radar are so bad, that I find myself failing to report the majority of bugs that I encounter because:
a. I assume (but have no idea) that some other developer probably hopefully already filed it
b. I don’t believe that the bug report will ever be seen or acted on anyway, and I would have no way of knowing if it ever was.
How could Apple be seriously committed to fixing buggy software when the very system to do so is in such a disgraceful and dysfunctional state?
The Good News
The only good news in this sad tale is that it would be so easy for Apple to start fixing this.
Open up Radar so that bug reports can be searched by anyone. If there is sensitive information, instruct bug reporters to place that information in secure fields that won’t be part of the public search results. It is absolutely crucial that existing bug reports be discoverable in order to reduce spam and duplicates, and to encourage developers to file bugs that don’t already have reports written for them.
Allow issues in Radar to be upvoted / liked / whatever so that developers and users can indicate prevalence and frustration levels for an issue in a clear way without having to create spammy duplicates or failing to give input at all.
Enable all users of Radar to add additional information and comments to existing issues, so that there can be a single location for all relevant data about a bug, and Apple’s own engineers can also easily see all of it in one place at one time.
With all the tens of thousands of engineering hours you are now saving by not processing massive volumes of duplicate spam reports, not throwing out important information, and by allowing your users to help triage issues, ensure that people who file bugs are given a way of knowing that the bug was seen and what priority / status / etc. it is in now as a result. This will work wonders for your developer relations and mobilize an army of outside engineers who will do whatever they can to assist in the effort to continually improve Apple software.
For goodness’ sake Apple, it’s time to open up bug reporting.
Posted in: applebug reportingbugsradar