Someone asked me recently about the Webjay popularity metric. It was a good metric — simple and reliable — so I thought I’d pass it along here. I do this with confidence that Yahoo doesn’t mind because its metrics are much more sophisticated.
The metric was based on playlist plays, so if somebody played a playlist this was used as input. A “play” was defined as fetching a playlist in a playable format like XSPF, M3U, SMIL, or ASX.
Who it was that fetched the playlist was recorded, so that we could filter out plays according to various reputation metrics. The reputation metric that ended up the winner was how old an account was. I tried others that were more exclusive, but they ended up filtering out so much data that there wasn’t enough left for the statistics to be reliable. By sticking to plays by people who had been around a while, we got rid of plays by people who were just looking around for the first time. New people invariable play the most popular items, so filtering out their activity fixed a feedback loop. (Note to old Webjay hands: feedback loops like this were the reason why the same few playlists would get stuck to the top of the listings).
At this point we had a set of play data that covered the entire lifespan of the project. If we counted each play as a single point, the sum would give the relative popularity of a playlist within all playlists. It would be a hall of fame with playlists from different time periods competing. (Though the point scores would have had to be normalized against the changing popularity of the site by dividing against the total points within a given time period). Given the amount of data and the number of competing playlists for such a large time period, the results would probably have been an accurate indicator of playlist quality.
However, we needed a sense of freshness, because regular visitors want to know what’s happening on an ongoing basis. To make this work the timestamps of the plays were recorded, and plays were given more value if they were more recent. Timestamps were used because they happen to ascend perfectly, which makes them monotonic. The ranking of a playlist was the sum of the timestamps.
However, there was again a feedback loop. The most popular playlists of all time still had an advantage in the popularity listing on the home page, and thus still got stuck to the top of the listing. There was a need to allow playlists to compete within different time windows, so that they could be on even footing. New candidates should be competing with other new candidates.
To set the time window of the ranking, the plays were filtered according to different time periods. I think the time periods were a day, a week, two weeks and a month. This gave us popularity contests among peers. The best playlist today, the best playlist this month, etc. Note that the filtering didn’t rely on when a playlist was created, so sometimes an old one would be rediscovered and rise to the top.
So which time window to use? There could have been pages dedicated to each one, but traffic off the home page was always going to dominate. Also, it is inconvenient to make users click around. The solution was for the different popularity contests to share the home page. This was done by choosing a random time window within the four possible time windows each time the popularity rankings were calculated. On a user level what this meant was that the home page would be showing one of four different rankings depending on when you viewed it.
This constantly shifting ranking set worked to sift playlists up through the ranks. A promising new playlist would get exposure by appearing on the home page in the “today’s most popular” set. It would compete with the other brand new playlists for enough popularity to advance to the “this week’s most popular” set. If it made the cut, it would then be on a footing to advance to the two-week set, and from there to the 1-month set. At each step a bit of popularity gave the playlist opportunity to compete for more.
A bit of good luck was that this metric captured the attention span of the community. A good playlist would be discovered, rise to the top, be tried out by all the regulars, and sink down as the regulars got bored with it.
A deliberate strength of this metric was that it was based on actual behavior rather than on vote counts, so was not as gameable as systems using the Digg approach. This also provided more input data, which improves the quality of statistics.
A weakness of this method was that it relied on a single home page, and a single ranking can never be representative of all the different interest groups. A project that I never got to do was to filter according to similarity with an input set of playlists or playlisters, so that you’d have the world according to Jim (who likes punk and modern classical) or according to Chromegat (who likes hip hop).
So that’s the metric. It developed over many sessions of trying to manage feedback loops and turn user behaviors into meaningful data, and took a lot of tweaking to get right. I hope this is useful to others.
It is very useful, thanks Lucas :)
I was going to write exactly the same thing Peter did— yes, very useful!
As I was reading your post, I kept thinking about the “weakness” that you eventually described in the penultimate paragraph. The decisions about the boundaries that define “what we think is popular” are a really interesting factor—it’s almost more about who you (the site / online community) are then about what’s the right way to define popularity. E.g., do you see yourself as one cohesive group, or as many groups, or just as many individuals?
In a way just being willing to have a single default view of the entire community on the homepage, or anywhere else for that matter, is the answer to the question. It supposes that there is one community.
To do otherwise you have to really refrain from any sort of centralized view of the system. Be an aggregator or be one of many decentralized publishers, but if you try to be both you’ll have limited success.
Don’t have an f’n idee what you talkin’ ’bout, brother. Would love to hear from you again though, and am planning a trip after I leave Emma’s ( wacked my back in work last week, it’s a definite sign I’m on the other side of the curve. and it certainly don’t loop back ). L.A. during X-mas sounds nice, but maybe later, especially if you have any gigs lined up. Checked some site and your still giggin’. This is good. Anyway, I won’t take up anymore of your valuable cyberspace ( god help me. ) and hope to hear from you. Luv Ya, Man!
brother! Very excellent to hear from you. Lots o’ gigging for me lately — five this month. Let’s talk offline about hanging out.
This is really interesting. I think I had always hoped that the metric might include something other than (or in addition to) pure plays to reflect quality and engagement — like number of tracks copied from a playlist or maybe quantifying the “originality factor” of a list by measuring how high popular tracks were on the “uploaded-first-by” list.
When the MySpacers arrived with their 200-comment emo playlists that always seemed to sit at the top of the popular list, I wondered whether comments had something to do with it. But honestly, it often seemed totally arbitrary, with odd four-dead-track playlists inexplicably bubbling up to the top. I always surprised myself by having even the slightest interest in the list…I suppose that even though it was sometimes a mess, I did find lots of good music through it, exposing things I might not have otherwise found through the other mechanisms in the site.
When everything was operational the metric did a good job of capturing the wisdom of the community, in the form of the listening habits of the people who’d been around for a while. At these times the odd four-dead-track playlists stayed out the way.
The breakage happened because of persistent problems with system administration (a weak spot of mine) that caused the calculation of the metric to fail. When that occurred the list would be in whatever order MySQL happened to return.