How to aggregate all of the answers to your StackOverflow questions into a single feed using Yahoo! pipes
21st April 2009
Feed
Here is the feed I create in this article. Add your own user id to the URL to track your own answers by RSS:
StackOverflow
I am a big fan of StackOverflow. The team there, Jeff Atwood, Joel Spolsky (and friends) have created an extremely valuable resource which goes some way to solving the many coding frustrations programmers face everyday.
The site is a unique and modern take on the question and answer site. It incorporates elements as diverse as Reddit-style voting on answers, Wiki-like editing of content and an Xbox Live-style achievements system. The end-result is a site which provides quality answers to any programming related question you dare ask.
StackOverflow feeds
As with most modern web sites, a great way to keep track of content is through RSS feeds. Each question on StackOverflow has its own feed which allows you to follow any answers posted in your RSS reader, rather than having to visit the site regularly to find updates. Unfortunately, you currently need to follow a new feed for every question you wish to see in your reader.
Of course, one subset of questions on the site will have particular relevance to you, the questions you personally have asked. Currently there is no way to follow all the answers to your questions.
Building a pipe
Recently I listened to a Podcast from HerdingCode which name checked Yahoo pipes a couple of times. Their discussion reminded me what a great tool Pipes is and it is something I have not used for a while. This gave me the incentive to get my hands dirty and build something useful.
I started out by examining the RSS URL’s in use on StackOverflow. A ‘user’ RSS feed is structured like this:
A ‘question’ RSS feed is present on every question page and looks like this:
The content of the ‘user’ feed contains both questions and answers submitted by the user. This means I need to filter the content as I am only concerned with a user’s questions.
I will then be able to route the questions back into a Pipes module to find the answers to each of the questions and aggregate them together.
Adding the user feed
My first step was to add my own user feed to the canvas. I dropped a ‘Fetch Feed’ module on added pasted my user feed URL, which I had copied from my profile page on the website.
Connecting this to ‘Pipe Output’ showed, as expected, that both questions and answers are displayed.
Filtering out my answers, leaving only my questions
Filtering out answers was not difficult as each answer started with the text “answer by”. I just added a filter block and configured it to block any items where the item.title contained “answer by”.
Exposing question URL’s with sub-element
I then needed to somehow extract the URL from the Feed which this block produced. The URL was actually buried in a sub-node, so I used a Sub-element module to expose it. I exposed item.link
I then used a second sub-element module to expose the content node of the element. I know had the question’s raw URL’s to feed into a new module.
Loop through my questions, fetching the feed for each
Up next was a loop. The loop module is very powerful and allows you to perform an action for every item in the source you provide it with. In my case I was providing a list of question URL’s which I wanted to then work with.
The loop module works by adding a second module to its centre. The loop module then performs this second module on each of the items fed into it.
In my Pipe, I added an instance of the ‘Fetch site feed’ module. This meant my loop grabbed the feed from each of the question pages.
Filtering out everything but answers
Looking at the output now, there are items for each of the answers, as required. However there is also an item for each question too. Here I did another filter using ‘answer by’ in the title, but this time it was the other way round, permitting only items with ‘answer by’.
Moving the answer text into the description field
My output is now what I expected, however it looks a little bare. The description is currently empty. The logical element to show in the description field is the answer itself.
To move this element to description I will do another loop, this time looping through the output and moving ‘item.summary.content’ into ‘item.description’.
Adding a URL input so it can be used by anyone
My last change is to add a URL input module to enable anyone to add their user id onto the feed URL. This means the same feed can be used by any StackOverflow user to track the answers to their questions.
There we have it. A feed which displays the answers to questions posted on StackOverflow. You’ll never miss a new answer again.
The finished pipe
The feed URL is here, add your own user id number to try it out.