Skip to main content

Data Mashups Made Easy: Yahoo! Pipes

Posted by davidvc on February 9, 2007 at 2:52 PM PST

I have been thinking for a while about how you data mashups: the ability to query and combine data source across the web into new data sources. I've looked at the Semantic Web, I've looked at Google Data and Amazon S3. To me what you want is a very very simple way to query web services in a way that a program can understand.

Now there is a new kid on the block for data mashups, and a seriously good one too: Yahoo! Pipes. What's very cool about this is not only does it let you combine and sort and filter RSS feeds in various interesting ways, but it lets you do this in a very easy to use, graphical fashion . This really is data mashups for the masses. It's also something that Mr. O'Reilly says he's been waiting ten years for

In the spirit of UNIX pipes, anyone can take a pipe I created and pipe it into theirs, or modify it to meet their needs. Over time I can imagine some very rich pipes built on pipes. I remember having fun with this in UNIX-land, and now here I am doing it for web feeds. Pretty cool.

Just for kicks to see how hard/easy it was, I created a pipe that combined searches for references to me from , Google Blog Search and Technorati (I'm nothing if not narcissistic :)).

I was able to sort by date in descending order, filter out duplicates, and filter out hits that were the blogs that I wrote (Google Blog Search finds my blogs as well as others').

One thing I couldn't figure out was how to parameterize the actual URL so I could pass in anyone's name, not just mine. But I'm sure somebody smarter than me can figure that out.

I ran into a couple of issues. In particular, I kept getting errors saying "Error processing your request" with absolutely no helpful information to tell me what was wrong. Also, even though I tried to filter out my own blogs, they still were showing up in the results (update: I finally figured out how to fix this -- but references to other members of my family are still showing up. Ah, well, good enough, and I have to move on...)

I also tried adding the resulting feed to my Google Reader using the "Add to Google" button -- oops, for some reason Google Reader says it has no items, when I know it does. When I used "Get as RSS," however, it worked great.

So, I guess there are still some bugs to fix. But the concept and in general the execution is excellent. I hope they are prepared for some pretty heavy load :)

What I really want to see happen is making it easy to do visual data binding from a NetBeans web app project onto a pipe like this. Just as you can today with the Visual Web Pack for databases, you should be able to visually bind this data source to a table in your web app. Now that would be cool. I guess I better get busy :)