Tuesday, 13 October 2015

Survey results: bug tracking software expectations

G'day:
This survey was so devoid of interest even I forgot about it! It took a bloody age for me to gather enough responses to bother writing up, but it seems to have maxed-out at 94 replies, so I'll run with that. Cheers to everyone who replied. And if you read this blog and did not fill it out: you suck. ;-)

As ever, I will preface this "analysis" with observing it's statistically meaningless because the sample size is too small, and is only really measuring the position of people a) reading this blog; b) likely to reply to a survey I create. That's pretty niche. However I got the answer I was after (yes, as I often do, I asked specific questions to demonstrate a specific point).

The questions were as follows:

  • What bug tracking software do you use?
  • When reading comments that have been added to an issue, in which order would you expect them to be listed?
  • When voting for an issue would you prefer (out of the listed options)
  • How important to you is it to be able to (perform a list of common bug-tracking tasks)
  • If you are a user of the Adobe ColdFusion bug tracker, what are your thoughts on it?
As I said in the original article (Survey: bug tracking software UX expectations), "The last question kinda gives away my motivation for asking these questions: I'm hoping to lean on Adobe to get some stuff in their bug tracker to be changed."

I had lobbied Adobe to switch the way they list comments in their bug tracker (from last first so one needs to read them upside down), to first first ("Comments displayed in reverse order"). Like reading and writing usually work. Adobe have come back with:

Currently the logic behind showing the notes in reverse (latest at top) is that:

- If user wants to go through the comment history they can go all the way to bottom, keep reading notes one by one (could be hard if any note is really huge) upwards. Once gone through all, they reach the point where they can add their comments in the note box.

- If user wants to read latest comment and respond to it, it's right in the top next to the box where new note can be added.

If we change the order then it would only make sense to move "add new note" box also to the bottom which may be confusing as the landing view will not show this box and user may think there's no way to add new comment.

I still believe the way it is shown (if we don't change design of page) is right.

Let me know if you still feel it should be other way around.

This is actually very good feedback from Kapil, and he is definitely one of the good 'uns at Adobe (like Anit). I don't agree with what he's saying, but it's a well-reasoned and well-articulated case. If it was one of the ColdFusion Team members, I'd expect this ticket to be closed "NotABug/ByDesign" without further comment. So cheers for the feedback Kapil.

This got me thinking... is my expectation actually as ubiquitous as I think it is? I am used to using Jira, and it lists the comments in reading order, not upside down order, and this is where I level my expectations. Not exactly scientific research there. Hence the survey (also not scientific research, but at least it's not just basing my position on my own person proclivities).

So what did other people think?

Q1: What bug tracking software do you use?

The sole "hard-coded" option I offered was "Jira". I was aware of a few others, but didn't know if people took them seriously: Bugzilla, FogzBugz (or whateverTF it is), Trac (chortle: I feel sorry for you if you are stuck with that). It did not occur to me to list GitHub, which would have been another sensible option. Still: I had an "other" option for you to DIY.

The responses were interesting. Not least of all because I had not even heard of a lot of the options. Some people put more than one option, and I've included them all (with how many votes each got, in parentheses).

Jira was head and shoulders the winner. My way of asking the question might have skewed this, but I will note that some people who used different systems as well as Jira answered "other", and listed the lot.

One observation. Five of you said "None"? What sort of way of conducting your work is that? Blimey. Perhaps you don't actually code in a professional environment.

Q2: When reading comments that have been added to an issue, in which order would you expect them to be listed?


Oldest first: 57
Newest first: 32 (you're all weirdos ;-)
Other: 5

The "other" options were:

I'm fine either way. If Bugbase becomes oldest-first, it's cool by me. I also now query my local copy of the Bugbase.
Oh I do so love people who sit on the fence. You have no preference for which order you like to read things? None? I'm surprised by that.
I actually prefer interfaces that thread comments, from oldest to newest.
Good call! I didn't think about threading as an option. Do some issue trackers out there permit this? But - bottom line - this is a vote for reading top-to-bottom.
I like the idea of being able to see the most recent comment first, but if reading through a substantial list of comments, oldest first (chronological order) makes more sense to me.
So - if I'm getting you right - you don't care if there's not much to read, but once it because an exercise in reading, then it becomes an issue to deal with, and you prefer top to bottom. That's a good pragmatic way of looking at it.
Newest first. Best if done in a threaded fashion. Also sortable by date
Good idea regarding being able to toggle the ordering.
Configurable at RunTime
This does not really answer the question. The answer would be based on when you see a list of comments in (some order), then which order would you use the toggling to order them in.

So the bulk of people - UX tweaks aside - prefer reading in chronological order. Not reverse chronological order. And I like the idea about being able to thread and re-sort the list too, so as to accommodate better discussion, and each person's ordering preference is preserved. Tell me though, people who suggested threadability... how would you expect the re-sort toggle to work within a thread which has more than one comment at the same level? Would that re-sort too? That seem weird to me.



Q3: When voting for an issue would you prefer:


  • Both up-vote and down-vote buttons (43)
  • A single button that you can toggle to vote or remove the vote again (33)
  • Other (10) (see below)
  • A single button that you can click once to vote, but the vote is permanent (8)
  • A text input with a minimum length wherein you must justify your vote (0)

So that's none Adobe. Not one person thinks your approach is a good one. Do you know why that is? Because your approach is bloody stupid.

Here's the "Other" answers:

A single button that you can toggle to vote or unvote, but coupled with a textarea that only accepts the letter "a" (enforced via javascript event handlers), which you must repeat 25 times.
Heh. Perfect.
A single button that you can toggle to vote or remove the vote again Both up-vote and down-vote buttons (with above)
OK, so like StackOverflow?
honestly a vote up or vote down button with a comment to be included.
I'd like an "On a scale of 1-5" type of selector that isn't permanent.
This is a good idea!
Single toggle or up and down vote options would be acceptable to me.
Either a single toggle vote button (option 2 above) or vote up/down buttons (option 3)
Poorly written question.
Not compared to the quality of your answer, I think.
Sorry, very non UX rant to follow. It all depends on how (*if*) the CF team intends to use the vote information. I think the users should be able to influence the importance of bugs, perhaps a single vote which is accompanied by a importance level. Accompany this with a limit to the number of active bugs a given user can assign to any one importance level. Ability to change these votes should be mandatory
Nice one, cheers for the thorough answer.
A button that you can toggle to vote or remove with an optional text input.
a file upload where I can provide photo evidence of the requisite virginal/animal sacrifice
Be careful what you wish for.


Q4: How important to you is it to be able to:


Not important Hardly important Average importance Rather important Absolutely required functionality
Re-edit the detail of a bug you raised 4 4 9 37 40
Delete a bug you raised 14 12 25 24 19
Re-edit a comment you have added 3 5 24 39 23
Delete a comment you have added 11 16 29 20 18

The general gist here seems to be it's pretty important to be able to edit existing stuff, but not so important to be able to delete it. One shortfall of the Adobe BugBase is that nothing is editable once it's gone in. Well except for Adobe themselves sometimes delete comments, I've noticed.

Q5: If you are a user of the Adobe ColdFusion bug tracker, what are your thoughts on it?


I think it is ok. It would be nice if they improved the design of the pages, reliability and reduced the request execution time of the pages. What a positive person you are.
Terrible
I would love to see the ability to either up- or down-vote a bug and not have to enter text for a vote. That's what comments are for. And as we've all seen, requiring "justification" for a vote does more to detract from the actual bug.
Adobe's bug tracker is a complete joke. The search interface is clunky at best. The fact that there is an internal only view that can be hidden from us is UNACCEPTABLE!!!!
should be a bug fix log, not a tracker (aka tracking bugs that never get fixed)
MySpace 1.0 was more user friendly
Yes I am a user of the bug tracker =P I'd like a filterable Feed view that shows all the activity (status changes, fixed-in-build changes, votes, comments) that have taken place. The Twitter feeds have been very nice and helpful. But sometimes I just want to see a list of ticket numbers that have had any activity in a certain date range (i.e. show me what tickets had activity last week). There should be a download option we can use to download the Bugbase to a spreadsheet so we can do more customized searches thru the data (it could be cached on Adobe's side and updated every couple of hours). Thanks!, -Aaron
Cheers dude.
It's crap.
I think I share the general distaste for lack of feature/ poor execution of half-arsed "features" as many in the community do. Having to leave a comment, let alone one of 25+ chars, just to vote up a bug is absurd. CAPTCHA submissions after already logging into a secured area of the bug tracker is, well... Absurd. Lack of editing abilities to your own bugs and comments is mildly annoying. Mistakes happen. It's just not a smooth process to do anything.
It's been through a lot of changes since I first used it. Right now it is better, but certainly not a good as it should be. You have to consider the audience; it is used by Adobe developers, and by the public, and probably has reports for executives. They can't just change it without changing a hundred different things. Enterprise software is big and complex, and doesn't just go change when blogs and twitter say it should.
Very true. However it could be improved over time, and it simply hasn't been. In reality it should bever have existed in the first place: it is not fit for purpose, and makes Adobe look a bit crap. And this is even before we get to how poorly the ColdFusion Team actually manage and interact with the thing. And their clients.
Could do a lot better. Not so much the bug tracker be nice to have a more modern one, but more the interaction on the tickets with adobe. A bug tracker is only as good as the people who use it in the end. Be nice to include an area for test cases on the bugs / features that all parties use too
nope.
You must please enter 25 characters... *(@&#(*$&*@(#$&(@*&*(!&@&*(#^!*&@#^
My eyes bleed.
Not the greatest, but it works
does such a thing exist ?
I've used better, I've used worse. Some parts of it are extremely annoying to the end user experience.
I don't use it much, but it's not very good.
it's terrible...
I've tried to use it but really don't like it's interface. I also have done more comments in support of bugs raised and I don't think I've ever raised a bug myself.
don't use it on a regular bases to have an educated opinion
Oh, come now: that's never stopped anyone else!
It's turd
I am guilty of not being more involved in the Adobe ColdFusion bug tracker. That said, the CFML Slack group has been a real eye opener for me and is really enthusing me to become a more active member of the community which will hopefully include becoming more active in the bug tracker.
Cool!
Best bug tracker ever (more text to hit the character limit)
[chuckle]
Honestly, it's not a polished or full featured experience. Too much data is kept private from the public, large ticket descriptions with code lack proper formatting for good readability.
I don't have anything to do.
I don't use it very often, but I find it is not very intuitive. I'd really prefer something with a dashboard like this: http://cfn.cfuser.com/ (Stephen Walker comment on your blog). With the ability to see the really active bugs helps to highlight areas which are impacting the community.
Don't us it!
Painful.
"A text input with a minimum length wherein you must justify your vote." Look at the bugtracker, look at how many votes have sadhsadj or any other combination of forehead+keyboard. Just don't do it.
it's shit
It's terrible and I hate it.
Hate it. The 25 char limit on votes is ludicrous. No audit trail is also shite.
Atrocious. Adds little value. Unclear how used by Adobe
Is "sucks" good enough? Appalling that a software company as big as Adobe can't build a decent bug tracker. But I guess we all know that Adobe doesn't use its own software..
It's a complete crock of utter shite. One of the worst bug trackers I've ever had to use.
Lacks feedback from Adobe. If they disagree with the bug or delay fixing it they should explain their reasons clearly. Lack of communication is very frustrating for customers who have paid $$$$ for each licence or developers who have to find workarounds because of a bug. Meaningful and respectful feedback is important too.
I don't mind it, i just wish it was easier to have a list of issues that i raised and be able to link to it easily.
Abysmal
Haven't used it in a while other than for voting on existing issues. Captcha seems like an incredibly strange requirement for successfully authenticated users. Don't care for notes/votes listed in descending order. Seems opposite to how it should be displayed.
It is probably the worst I've used and the silly 25 characters to vote is utterly pointless, but it is all pointless if you raise a bug in version X and they fix it in version X+1, X+2, etc... and expect you to pay for the fix, or just don't fix it at all...
It is a very out-dated, poorly maintained, out of touch system that might have been good enough in 1999, but does not even slightly measure up to *any* other bug/ticket tracking system out there today. The project should be scrapped and replaced with one that is *not* developed by Adobe.
Overall I think it's okay. I'd like a 'dashboard' feel to the tracker where it was possible to search on additional metrics. In addition I think an audit log is an absolute must. If Adobe closes a bug, I want to see when it was closed, who closed it, with a required text field justifying the close.
It sucks
Its the worst ever. Very slow in India.My suggestion to Adobe is that please get rid of that as soon as possible.
Ready for update.
in some ways, sufficient. in most ways, shite.
It's crap. 25 characters in a comment to be able to vote is ridiculous. Search is a joke. They need to figure out they are not in the business of building issue tracking software and get a solution from someone who is in that business.
It's the worst bug tracking / ticket management system I have ever encountered. And I have used uncustomised Bugzilla in the past.
That's my own comment. Quite understated compared to some!

And that's it. Admittedly not the most scintillating survey I've done, but I just wanted to get some material and reaction to share with Adobe. Job done.

Thanks for the input, everyone. Now let's see what Adobe do with the information...

--
Adam
at