Aggregating myself with Yahoo! Pipes

Filed under: Developer, Internet, Blogging, Web services, Yahoo!, How-Tos, Social Software

Aggregating myself with Yahoo! Pipes

by Grant Robertson Feb 20th 2007

Yesterday I wrote a bit about taming the RSS feeds generated by the myriad Web 2.0 services to which you're subscribed. It occurred to me that Emily Chang's original quest to pull together all of the feeds she creates into a single meta-feed was a perfect project for Yahoo! Pipes.

I sat down to take all the data I spew at places like my own blog, del.icio.us, Twitter, and Flickr into one aggregated bowl of RSS love. The result of my feed union isn't an hour by hour diary of my day, but does create an ephemeral stream of things that are undeniably me. The same techniques could easily be tweaked to document your day to near pinpoint perfection, if you're so inclined.

The Pipes components you'll need are Fetch, Union, and Sort. Drag your first Fetch onto the sheet to get started.

Paste the URL of your first victim feed into the Fetch box. If the feed is valid, you should see the universal RSS orange logo pop up next to the URL.

Fetch doesn't work exactly the way I'd guessed, and because of that it took a little experimentation to make this work. For some reason, if you add all your URLs into one Fetch, the sort doesn't process them correctly and they will never meld into something that makes chronological sense. Drag a new Fetch to the sheet and repeat until you have all of your feeds entered.

Next, we'll drag in Union. Union allows you to put the feeds together in a way that retains any extended specific data contained in only one feed. For RSS newbies, think about the Union step like shuffling together a few decks of cards at once; All the cards will end up in one stack, but not in the specific order you want them.

Attach the Fetch blocks we created earlier to the top of the Union. A Union will allow you to connect up to 5 feeds. I haven't tried to connect more (yet) but I'm guessing that multiple layers of Unions could be used.

Now that we've unified the feeds, we'll use Sort to straighten them out. Drag in Sort and connect it to Union. You should also connect the bottom of Sort (the output) to Pipes Output.

The drop-down box to the left of Sort will show "Updating" for a few seconds. This is Pipes attempting to catalog all of the feed elements which you could use to perform your sort. Once the drop-down fills in, we'll select PubDate, the date and time which the element was published. We should also set the order to descending, so that the resulting feed contains every item from our originating feeds, ordered from new to old.

That's it! If we click on Pipe Output, Pipes should grab and massage all the data and display it in the preview pane.

My resulting feed takes information from my blog, Flickr, Twitter and del.icio.us and creates Aggregated Grant, a meta feed my friends and family can watch to get a better idea of what I do everyday.

Tags: Emily Chang, EmilyChang, Pipes, Yahoo! Pipes, Yahoo!Pipes

Reader Comments (Page 1 of 1)

robotrock@gmail.com said 1:48PM on 2-20-2007

Or you could do this much easier with aim...

http://buddyupdates.aim.com/settings

Grant Robertson said 2:07PM on 2-20-2007

Similar idea but, Pipes is way more flexible that that AIM buddy notifications system.

e-tat said 4:38PM on 2-20-2007

Pipes looks like fun - but if all that work is just so you can combine a few incoming feeds, then RSS Owl does that with a single click. http://www.rssowl.org/

On the other hand, if you are spewing a load of feeds, then the pipes method looks pretty simple.

Grant Robertson said 4:57PM on 2-20-2007

Pipes is quite a bit of fun. My main goal writing this tutorial was to give a foundation from which to jump off when writing more in depth examples of how to use Pipes. We'll be doing more cool stuff with Yahoo!'s new baby, soon!

Carl said 5:00PM on 2-20-2007

Or, you could just use google reader, organize feeds into folders/tags, and click on the folder and walla...combined feed.

hubs said 6:08PM on 2-20-2007

a couple of things i don't like about pipes: no links. pipes strips the link from your entries. this means any links referenced in your text will be taken out in you pipes feed. also pipes only aggregates excerpts from your feeds. i believe anything above 200 words is eliminated from your aggregated pipe feed.

other options i found that are better:
xfruits
feeddigest
feedjumbler

however, none of these options allow you to show the aggregated feed on your own website like lifestreams (www.artifacting.com/blog/life) do.

Dave Chartier said 6:00AM on 2-21-2007

Pipes is capable of a *lot* more than simply combining feeds. Some of Yahoo's example feeds demonstrate its power best: one of them looks at lead stories from New York Times.com, then filters Flickr for pictures that relate to, or are about, those lead stories from the NYT. So it isn't just combining two feeds - it's actually performing quite a bit of logic to filter and combine content from two sources into a new RSS feed.

Hot.

Peter Cooper said 3:51PM on 2-23-2007

hubs: FeedDigest lets you put together things like that artifacting page you linked to. The templating is totally dynamic. In fact, I made something like that for myself combining my Twitter, del.icio.us, and blog feeds into a single "life log" :)

Featured Time Waster

Civiballs is a beautiful, soothing physics puzzle Time Waster

I have an absolute weakness for physics games, and while Civiballs isn't the strongest physics-based game, what it lacks in the physics department it makes up for a few times over in style and fun.

In Civiballs, you are presented with a few colored balls, and your goal is to get those balls into the same-colored urn on the level. The "civi" part of Civiballs is that there are 3 sets of levels to play, each representing a different civilization. While the civilization doesn't affect gameplay, the artwork for each level is beautifully themed to it's appropriate era.

To play the game, you are given only one tool - a sword with which to cut the chains that are holding the balls. The puzzle part of the game is in figuring out what order, and with what timing to cut each chain. Do it right, and all the right balls end up in the right urns, with no stray balls entering an urn (a no-no). Do it wrong, and you get to start over again.

Civiballs is not terribly deep on gameplay; the entire game can be completed in about 15 minutes. But if you enjoy this type of game, it will be a very enjoyable 15 minutes.

View more Time Wasters