Rankings
OK, this post is only tangentially related to astronomy so feel free to stop reading now if you want to. For some time I have been worrying about ranking algorithms, mainly in the sphere of podcasting but in a wider context too. This is partly because I use some in the Astronomy Media Player and I'm worried that they aren't fair. However, as the AMP has relatively few people using it I'm aware that I may be suffering from problems with low-number statistics. So, I'll illustrate the problem with something that has more users.
For the last few months I've been monitoring the "Top Podcasts" under the Science & Nature and Natural Sciences sections in the iTunes UK store and wondering how it gets calculated. I haven't been doing this systematically (because it seems impossible to get data from iTunes without typing by hand), but I have noticed a distinct trend. Whatever gets "featured", does well in the rankings. If you've never used iTunes, you probably don't know what I'm talking about (it doesn't seem to be web-accessible), so here is a screenshot to illustrate the point.
Screenshot of iTunes UK Natural Sciences section on 19/5/2007 CREDIT: iTunes is created by Apple Computers.
It would appear that to be featured (with an image) above the fold has a very strong affect on your ranking in iTunes. Of the top 10 podcasts, only one (Brain Food) is not currently 'featured' above the fold. In fact, of the top 25 podcasts, only seven are not featured on the page at all (highlighted by red boxes). Also, one one featured podcast (Science Update) does not appear in the top 25. Of course, it could be that being in the top 25 helps to get you featured but that does not appear to be the case. Over the past week I've been watching the Jodcast's rankings and they definitely lag behind being featured (the Jodcast lept up from around 26th place two days ago to 12th after being featured). It would appear that people, understandably, generally just click on the pictures put in front of them.
So having a prominent placement on something like iTunes boosts the number of people listening to a podcast. What worries me is that because people tend to choose the 'featured' podcasts it means that the folks at Apple, who decide what gets featured, have a lot of influence over the success of podcasts. With little turnover on the featured list, it can be difficult for new podcasts to build up listeners. Obviously, Astronomy Cast managed it but many others have not. This problem isn't just limited to iTunes though, with features such as Amazon's "people who bought this also bought..." tending to become boring if there are few eclectic people in the mix.
Now back to my problem. When I first put small podcast pictures on the front-page of the Astronomy Media Player, I noticed that those near the start of the alphabet started doing much better than they had done before. I then changed the listing to be sorted by popularity, and this appears to have had a positive feedback effect; those at the top reinforce their popularity and therefore remain at the top. I don't feel too happy about this and am trying to think of a solution. Obviously, popular podcasts should generally be near the top of the list but some randomness should probably be thrown in. The randomness would add a bit of churn to keep it interesting. The only problem is finding the right level between the positive feedback and the randomness. Any comments or suggestions?
Comments: Rankings
We get about 15,000 downloads within 2 weeks, and about 100,000 downloads overall every month. The problem with putting the shows totally at random, is that people don't know which ones are the best ones to listen to.
Perhaps you could have some kind of system where people provide reviews and ratings, and then they're ranked based on that. ;-)
Posted by Fraser Cain on Sunday 20th May 2007 (15:24 UTC)
"The problem with putting the shows totally at random, is that people don't know which ones are the best ones to listen to."
Fair point. I do have a rating system (5 stars) in place which is getting some (not much) use. That is used, along with numbers of plays, to work out the "Top of the Pods" list for individual episodes.
I'm currently trying to think of a way to add small comments (and possibly tags) to individual episodes. My main concern about allowing comments/reviews is the need to implement spam filtering and flag offensive comments. Perhaps I am worrying too much.
Posted by Stuart on Sunday 20th May 2007 (16:07 UTC)
...and I meant to add, WOW! 100,000 downloads a month!?! Great job.
Posted by Stuart on Sunday 20th May 2007 (16:11 UTC)
I'd like to point out that your AMP website does not render properly in the Safari web browser. It might help your traffic if it worked on the default mac web browser.
Posted by Sean on Sunday 20th May 2007 (22:51 UTC)
Sean, I don't have Safari, or a Mac, so I have no way to test it. Any advice?
Posted by Stuart on Sunday 20th May 2007 (23:28 UTC)
hey Stuart, you can view your AMP website on this site http://www.browsrcamp.com/
and see how it looks like in Safari
Posted by Mundhir on Monday 21st May 2007 (14:03 UTC)
Thanks for that link Mundhir. I think I've fixed the problem with the front page by hard-coding the width of the main element. Sean, could you check it with a real copy of Safari and let me know if it seems to be OK?
Posted by Stuart on Monday 21st May 2007 (16:54 UTC)
I remember when Slacker Astronomy was always in the top 10 and often being featured like AstronomyCast now is (including those big blocks at the very top of the page). Interestingly, those spots did affect our iTunes ranking - as you concluded. However, they did *not* affect our subscribership! It's an interesting result I can't explain, but numerous podcasts have also seen and reported the effect over in the podcast alley forums.
I am conservative when it comes to reporting stats. I define a listener as being a unique IP address that visits our XML feed at least every 7 days. That way the stats don't get bloated by one-off visits or visits by people with dynamic IPs. Based on that criteria, we have about 10K subscribers to the slackerpedia galactica feed (and about 4K more who visit monthly). That number was in a linear decline from a peak of 15K starting last April until this February, and has been stable since then. When we got slashdotted last Thursday we got 35,000 downloads of that show, it remains to be seen how much of that will translate into new subscribers.
BTW, when I first heard that iTunes was gonna add a podcast directory, I e-mailed the astronomy dude at Apple (whom I met at an AAS meeting) and asked to be included in the initial directory. He said he sent it to the podcast editors and sure enough, we were there when they launched. So I think some of it is simple networking and ass kissing! :)
Posted by Aaron on Friday 25th May 2007 (02:08 UTC)
Aaron, :-)
Posted by Stuart on Friday 25th May 2007 (09:16 UTC)
OK, UK iTunes have just changed the way they display the Natural Sciences section. It is now in alphabetical order!?! That means that the shows that have been deliberately putting punctuation at the start of their feed name ( - Naked Scientists ) are now ranked top.
Posted by Stuart on Friday 25th May 2007 (16:53 UTC)