© 2023

Yahoo Pipesopener.rtf

Someone told Daniel [Raffel] tonight that if they ever wanted to work at Yahoo! they would want to work on Pipes. I think that is the ultimate compliment.

Pasha Sadri, co-creator of Yahoo Pipes, in 2007

Want to know whether the latest logged earthquakes were near you? Aggregate 100 top news sites, but only see items that mention cats? Get a steady stream of sport scores, scraped from sites that don’t offer an RSS feed? Find a rental apartment amidst those posted on Craigslist and other online apartment listings that fits your price range and is near a park? Exclude stories on topics you’re not interested in from publications you already follow?

Yahoo Pipes—or, officially, Pipes by Yahoo!, a rare switcheroo of the company’s name—was a service that offered all that and more in a single platform. Individual “Pipes,” as it were, were both personal and public—the service seemed like a portent of the future that techno-utopians were then predicting. But like many great products and services of its time, it never quite came to pass.

Also like many great ideas in the mid-’90s to mid-’00s—that nostalgic first decade of the commercial internet—Pipes started with one person who built a small team to knock out some code that they thought might just change the world… or at least kick a dent in it.

Chapter 1

The early morning knocking of the Pipes

This openness that encouraged—really, required—knowledge-sharing embodied some of the internet’s best qualities that were on the verge of being lost at that time.

It was 4 a.m. when Pasha Sadri got the call. The Pipes were busted.

Just months before he’d come up with the idea that underpinned Pipes: a visual programming tool that could produce a series of structured entries as RSS, JSON objects, and other commonly used newsreader and data-processing formats. Pipes was an authorized skunkworks effort within Yahoo, and part of a host of cutting-edge ideas funded by the company. A finished version would be intended for end users and could also be plugged into website programming workflows.

Alarm Clock, reading 4:00 AM

Sadri built Pipes as a cloud-hosted service alongside just four other core team members. Users dragged and dropped elements in a web browser to roll their own information feeds, taking published items from websites in many formats—news, database outputs, and blog posts among them. Pipes’ users could then merge the results, filter and process them, and produce a regularly updated stream that could be subscribed to in a desktop app—or texted to a phone. Geotagged results could even pass through a Yahoo Maps filter to restrict results based on location and proximity to features like parks or city neighborhoods.

Every Pipes’ mapping was public; every Pipe could be copied and modified. “That was design principle one for us, and I think that had a lot to do with making it accessible to non-developers,” Sadri said. This openness that encouraged—really, required—knowledge-sharing embodied some of the internet’s best qualities that were on the verge of being lost at that time.

After roughly six months of development from mid-2006 to early 2007, Pipes was slated to be quietly released into public beta in February 2007. But instead of the expected thousand or so initial users at launch, Pipes had hundreds of thousands—not to mention the many others who failed to get through. Unoptimized code on a handful of servers struggled to execute the newsfeed workflows and serve up pages; the system was completely overloaded.

Pipes offered a vision of cloud-based processing engines yet to come—and its launch, a preview of the demand for them. But the service’s journey was paved with good ideas and unclear expectations.

While any single Pipe could have run on personal and workplace computers then, the Pipes service in aggregate as initially designed—not necessarily for large-scale use—required significant data-center resources to keep running. Bandwidth was scarce and often expensive—and a single Pipe could pull down tens or even hundreds of megabytes for each uncached workflow execution.

“We had to spend the next six months just scaling what we had built to [...] keep up with the demand,” Sadri said. That 4 a.m. call in February 2007 would become just one of many pre-dawn alerts as Pipes’ explosive popularity far outpaced the expectations of the team that built it, and demanded much of a system largely built as a prototype.

The interactive, web-based visual programming of Pipes—inspired by earlier interfaces—became a primary influence on a generation of UI/UX designers and products that followed. Its remix ethos matched the time, and led people to dream bigger. It seemed like it held some version of the possible—but with no business model behind it and no internal commitment to advance the product, it became a poster child for unrealized potential, like so much else at Yahoo.

Chapter 2

The genesis and genius of Yahoo Pipes

Yahoo! mug

Sadri spent most of 2000 to 2007 at Yahoo, minus a brief detour to Google in 2005 where he worked on Google Maps. Accepting a job back at Yahoo in 2006, he joined the Technology Development Group (TechDev), created as a vehicle for Flickr co-founder Caterina Fake (Yahoo acquired the company in 2005) as part of Yahoo’s Advanced Development Division, headed up by Bradley Horowitz.

Horowitz’s domain was a set of sometimes overlapping groups. Fake’s TechDev was a key element, tasked with testing out possible new directions for Yahoo that weren’t bound up with Search and bringing entrepreneurial employees into it.

Search paid the bills: ads that appeared on Yahoo results pages and search-licensee sites brought in the vast majority of the company’s revenue. The most innovative product at Yahoo after its founding hierarchical list of sites was a browser toolbar designed to keep people connected to Yahoo products even when they weren’t on a Yahoo domain.

Like many companies, Yahoo diversified through a combination of buying and licensing promising outside technology and fostering new ideas internally.

Most of Yahoo’s employees worked in California’s South Bay, in the Silicon Valley cities of Sunnyvale (Yahoo’s HQ town) and nearby Santa Clara. Some employees living in San Francisco dreaded the commute out of the city to farther-flung offices, a common complaint among local tech workers then and now.

San Francisco, an hour north on a good traffic day, hadn’t fully emerged in 2006 as the darling of new tech startups and established firms that wanted more edgy cred, but it was on the verge. Fake had earlier developed a plan for a quick-footed incubator called Brickhouse. That group would build on and become in charge of some of what was happening in TechDev further south. Yahoo created the division in San Francisco.

Yahoo’s entrepreneurial internal divisions were sometimes hard to separate, and left a thin contemporary record behind about who was in charge of what. This led to some confusion even today about the finer details of the timelines. Fake shared that TechDev was started for her to “gather the most talented crew from around the company to develop new products.” Pipes came out of that initiative, and Brickhouse was founded later, bringing Pipes under its umbrella.

Fake explained, “I was given the mandate by Yahoo co-founder Jerry Yang to ‘make Yahoo more like Flickr’…being a startup product person who loves working with small engineering teams, I had come up with several proposals of how to create small, effective teams within Yahoo, get the org out of the way, and create a groundswell of Web 2.0 energy.” Pipes was the first of many initiatives.

Chad Dickerson, hired by Fake as the head of Yahoo Development Network, and in that role at the time of Pipes’ launch, agreed that Pipes “became the prototypical project of what Brickhouse aspired to be.”

TechDev/Brickhouse worked to keep some of Yahoo“s most creative and restless employees from leaving. Kevin Cheng, who joined the Pipes team as a designer a few months after the project“s boot up, said, “We had a saying that Brickhouse was the last stop on the way out for people at Yahoo.

TechDev served a similar purpose, but Brickhouse’s San Francisco location was more appealing to some, particularly those who lived in or near that city.

Sadri’s return to Yahoo after a year at Google might seem baffling in 2023. But times were different, and there was intrigue. While Google was rapidly churning out successful projects, Yahoo held a different sort of possibility, always seeming to be on the precipice of what could be some next big thing.

And Yahoo drove massive amounts of traffic. Kent Brewster, then part of Yahoo’s DevNet, said that the home page had 1.2 billion visitors a month when Pipes launched; its sites together received over a billion page views a day. It was substantial enough that after he suggested linking to Al Gore’s An Inconvenient Truth on Yahoo’s home page, the former vice president arranged a screening of the film near Yahoo HQ.

After the showing, Brewster said Gore strode down the aisle to find him and said, “I understand we have you to thank for getting us on the front page of Yahoo.”

So it goes that one might have come to Yahoo with an appetite. After working on Google Maps, Sadri wanted to build something that leveraged new capabilities in browsers to allow real-time interactivity with otherwise complicated-to-manage data and services. “My experience at Google showed me, wow, we are in this new era of what is possible to do in browsers,” Sadri said.

Horowitz gave him a clear mandate: “You want to work on some interesting projects? Take some time to come up with something to work on.” Though he had several ideas, it was the mélange of RSS news feeds, cloud-hosted software agents, visual programming, and in-browser interaction that would become Pipes that got Horowitz’s attention.

(“Pipes,” as it were, stemmed from the pipeline concept in Unix, in which one kind of program could straightforwardly feed, or pipe with a |, its output to the next, and so on.) 

Yahoo Pipesinconvenient-truth-DVD.jpg

inconvenient-truth-DVD

Chapter 3

Automator for the people

According to Sadri, Pipes’ graphical conception was heavily influenced by what Apple’s Automator had already demonstrated as an end-user tool for people with programming experience. To write a program in Automator, a user browses a list of actions connected to apps or to the system.

By dragging actions into a linear workflow, a user could construct an automation with as many parts as needed. For instance, take a folder of images, resize them all to the same width, and save them as JPEGs into a specific folder. As one “programmed,” Automator wrote real code.

Apple released Automator in 2004 for a similar reason as the Pipes team, said Sal Soghoian, then the product manager at Apple in charge of the Macintosh operating system’s programming language AppleScript. It was a way of taking something that processed operations that normally required coding skills and making it accessible to all users.

In particular, the AppleScript team wanted to make the language easier for non-programmers to use, and Soghoian said that Apple engineers Eric Peyton and Tim Bumgarner had each experimented with visual pipelining. With eventual sign-off from Steve Jobs, Automator was featured in a Worldwide Developer Conference keynote that year. It remains part of macOS, alongside the newer graphical programming Shortcuts system.

automator app

Pipes would more fully develop many ideas cribbed from Automator. However, it wouldn’t be a native app—it would live in the browser. Also unlike Automator, which let programmers expand on its prefab actions by writing scripts or code in most any language available on a Mac, Pipes would be limited to a constrained—albeit still powerful—set of options.

As a cloud-hosted system in that era, had Pipes allowed arbitrary code to run, it would have been disastrous, chewing up CPU cycles and crowding out resources. (Later cloud-based engines and hosting created sandboxes, computational throttles and price tags, and virtual machines to limit impacts.)

Sadri recruited Jonathan Trevor and Ed Ho from within Yahoo to join the team as its only programmers. Ho noted that before Pipes, he and Trevor had worked on another highly interactive project that let a user drag a Yahoo Maps view around and see all the events happening in the area.

Although he and Trevor were used to developing their own ideas with Yahoo, Ho said “Pasha was kind of well known at Yahoo for being this super creative guy [...] He was given kind of carte blanche to make something cool.” That sold them both.

Lanyard

Ho built much of the back-end plumbing, while Trevor focused mostly on the user interface/user interaction portions, writing JavaScript to allow drag-and-drop interaction and live graphical feedback. Trevor had his work cut out for him, given the wide variety of partially feature-complete browsers in 2006, like Firefox, Opera, and Safari; Internet Explorer had its own oddball coding and HTML requirements.

Daniel Raffel was brought in as the product manager. His practical work became a combination of sketching ideas, encouraging the team, managing workflows, staving off questions, and getting upper management to provide more resources. Kevin Cheng joined a few months later to focus on design.

As they churned away on features, code, and design, Raffel said the team had never been given a timeline for a formal launch. They moved—deliberately—toward sometimes amorphous goals, not knowing whether they had days or months or years to finish what they’d started.

Every member of the team agreed that they had remarkably little anxiety about their unknown deadline. They accepted that one day, a countdown clock would start ticking away the seconds until launch; they accepted, too, the possibility that Pipes might never see the light of day.

Chapter 4

When plumbing seemed exciting

Yahoo Pipessupercuts.mov

00:00:00

00:00:00

Pipes arose from the zeitgeist. During the mid-aughties, a vibrant internet culture embraced the trend of mash-ups: the art of extracting diverse materials from various sources, skillfully deconstructing, intertwining, re-editing, and ultimately merging them to create novel, invigorating works. Companies, in turn, joined the fun to (hopefully) capture the audience, page views, and money.

A perfect example were supercuts, rapid-transition video collations that began appearing around 2005. Andy Baio—a Yahoo employee from 2005 to 2007 through Yahoo’s purchase of Upcoming.org—is credited with the name, describing supercuts in 2008 as when a “superfan collects every phrase/action/cliche from an episode (or entire series) of their favorite show/film/game into a single massive video montage.” (As they got more popular, media outlets that had initially scoffed at supercuts—and at the creators that spliced them together—sometimes took to creating their own.)

A similar spirit of the time encouraged open-source coding coupled with freely licensed or accessible information that could be aggregated, filtered, or analyzed by anyone anywhere.

Information wanted to be free—or at least, freed from constraints—and for a brief experimental period, it was.

Pipes built on these trends by letting users control information flow, offering them access to new stuff appearing on any site. Often, the material arrived as elements within an RSS feed, a standard developed in the ’90s that was in its heyday by 2006.

It offered a (really) simple and machine-readable approach for disseminating blog updates, publishing podcast episodes, refreshing search results, and granting access to other quanta of information. Usually, every entry contained a title, publication date, unique identifier, and author. The main body carried the essential content, while starting in the early 2000s, an optional attachment entry began to be used to reference audio files, almost entirely to syndicate podcast episodes.

Yahoo PipesRSS feed-pipes.jpg

RSS feed-pipes

Jon Udell, who in 2007 had recently joined Microsoft after transitioning from his role as an analyst and writer at the corporate technology magazine InfoWorld, highlighted how RSS “gave us this normalization of access such that something like a Pipes could come in and rest on top of that because it wasn’t struggling to figure out how to talk to a million different sources.”

With Pipes, any user could create entries for any number of data sources. While RSS was key, Pipes also let users grab other data and feed formats, like Atom, RDF (released as RSS 1.0), and CSV (comma-separated values, a simple database and spreadsheet export format).

Pipes gave visibility into internal Yahoo sites and services, particularly Flickr, and eventually supported Yahoo Query Language (YQL), a project designed for interoperable in-house and third-party scripting across all of the company’s offerings. With a little elbow grease, you could even retrieve a web page and filter it for specific data.

footnote

YQL was a company-wide mandate to produce services with common endpoints to allow queries across internal products (and beyond), both for internal and external developers. While not inspired directly by Pipes, YQL allowed for Yahoo-wide mashups, and ultimately let developers outside the company even store the results of queries in SQL tables at Yahoo. Effectively, it tried to turn Yahoo into a platform for other websites, while leveraging the advantage for new internal offerings and combinations. You could see this as a flicker (not Flickr) of what Amazon Web Services would ultimately offer: a way to offload certain kinds of programming structures and resources to a specialty provider instead of building out the tools and services on one’s own data center servers.

Yahoo PipesRSS feed-pipes-2.jpg

RSS feed-pipes-2

Yahoo PipesScreenshot 2007-04-11.jpg

Screenshot 2007-04-11

But unlike a newsreader, which aggregated the results into a list or groups, Pipes’ power came from the ability to connect and filter the output of all the feeds through intermediate steps. Compared to an aggregator, Pipes allowed for more advanced processing and customization of collected data. It struck a balance between being versatile enough for experimentation and specific enough that most workflows focused on filtering news, blog posts, and listings at online auctions, Craigslist, and ecommerce sites.

The Pipes team’s canonical example was the scenario of a renter seeking an apartment within a given distance from a park. (The Pipe workflow took a feed of Craigslist rentals, processed the addresses against Yahoo Maps’ query language to look for parks, and then produced a list that fit within a given radius from any park.)  Via Mastodon, former user Nicolas Hoizey pointed me to a Pipe he used from 2007 until the service’s end in 2015 that let him add a bookmark to the Pinboard bookmarking service and process it into the form he wanted before handing off to IFTTT and Buffer to post to Twitter.

Pipes output could be embedded into a page or formatted as HTML, JSON, KML, RSS, or XML. There was even a way for a non-Yahoo website to include JavaScript that queried a Pipe workflow and received the results back without any front-end browser or back-end server involvement. Pipes, in effect, could power other sites.

Sometimes, though, it could also drag them down. Now and again, major sites gave banning Pipes a go. One of those sites was Craigslist, which blocked Pipes in 2009 after a non-Yahoo developer used it to create a map of Craigslist rental listings. The company’s CEO defended the action by claiming that Pipes was devouring too many resources.

The ban left mashups across the internet missing data; some developers left Craigslist behind as a data source, but opted to keep their Pipes. (The blockade was ultimately short-lived, reversed after about two weeks.) This sophisticated—occasionally mischievous—filtering, mixing, and repurposing, despite being astonishingly valuable in a data-saturated world, wasn’t even the truly revolutionary part. No, the revolutionary part was the graphical interface.

Chapter 5

A placid surface masked an ocean of complexity

We would play Mario Kart every day after lunch and then just keep working.

Since the dawn of modern computers, coders have been largely constrained to 80-character lines, whether they were literally knocking out chads, typing at dumb terminals, or clattering away in modern word-processor-like code-development system windows. As Udell put it, “In practice, we all feel like it’s kind of bizarre that we’re operating in this world where, effectively, we’re dealing with punch cards.”

Sadri threw this out the window as part of the effort to make Pipes broadly accessible. The proposition was that Pipes would not only require no formal programming—it wouldn’t allow any. Instead, every task would be accomplished by dragging and dropping nodes within a browser window, and drawing links between them to indicate control and data flow.

Yahoo Pipesgoogle-maps-2005.jpg

google-maps-2005

It was only in 2005 that Google Maps launched, which became a shining example of allowing people to use drag and zoom within an interface, interacting with a map that updated live instead of with page reloads. Improvements in browser technology let CSS and JavaScript interact more fluently and efficiently with users, while also allowing JavaScript to have continuous live interactions with remote servers (a technology initially labeled AJAX).

Building a structure of nodes and wires within Pipes, and representing them within a visual coding environment, occupied more time and effort than any other part of the project before launch, according to team members’ recollections. It required a fair amount of custom HTML, CSS, and JavaScript to allow Pipes to work in that era. Sadri said getting their boxes and animated, curvy lines “to actually work across all the browsers that we supported was one of our major technical achievements.”

Fortunately, the team was up to the task. Raffel said Trevor’s rapid development skills and his ability to quickly address the team’s needs with innovative solutions, especially, played a crucial role.

“Just to give you a sense of how amazing Jonathan was at programming: I would mock stuff up, and before I could complete writing everything out, he’d have a prototype working,” Raffel said. “I’d start using it, and I’d be like, ‘Well, that’s neat, but we need a way to know which primitives can accept an input.’ He’d, minutes later, have something working that was a visual cue that would show you, ‘This highlights this thing.’”

Turning those prototypes into more fully-fledged and colorfully designed features was set to a backdrop of Mario Kart. Sadri recalled Ho getting the team hooked on the game, which could be played in close proximity using individual Nintendo DS handheld consoles.

“A lot of times [people] would find us feverishly playing” he said. Ho noted, “We would play Mario Kart every day after lunch and then just keep working.” A whiteboard in their workspace effectively obscured their work—and play—from passersby.

Raffel recalls these gaming sessions as critical to Pipes product development. “It was so much easier to get feedback and to have a discussion while people were playing games and having fun,” he said. After a game ended, “We would usually go into a conversation mode and whiteboard out these ideas.”

Repeated playing of Mario Kart kept front of mind sets of bright colors in high contrast with one another as Pipes was designed. And then the Nintendo Wii came out in November 2006. “We basically copied the early Nintendo Wii color schemes [in Pipes],” Raffel said. Ho recalled the influence of the Wii, and the “soft glowy colors [that] were part of their aesthetic.”

The largely blue-and-white scheme was found elsewhere at the time, too. Apple’s reboot of its operating system into the Unix-derived Mac OS X in 2000 introduced the Aqua interface. Aqua became increasingly bulbous and shaded as Apple emphasized its graphical prowess, perhaps reaching its peak level of wateriness around 2003 before settling down in subsequent releases and then sliding temporarily into skeuomorphism. Cheng recalls Apple and its multi-color iMacs, too, being a heavy visual influence.

Chapter 6

Craft works

It was really hard to write the wrong program in Pipes ... As soon as you dragged something on screen, it would immediately start working and produce some output.

As Pipes evolved from one person’s idea into a collaborative team project, the interface progressively grew in complexity and drew inspiration from more sources. Expanding on cues from Automator, which adhered to a strictly linear program flow (an initial version of Pipes was linear too, according to Sadri), Pipes embraced a more dynamic approach, enabling the execution of scripts and code for each node in its program sequence.

Yahoo Pipes encompassed an array of control structures and programming constructs to allow it to work more like a general-purpose computing language, despite its limitations. You could use if…then conditionals, create loops to iterate through items, use splits and joins, and, yes, pipe the output to other web services. Pipes even used link syntax coloring, with control structure lines marked in gray and data flows in blue. While Automator facilitated program execution, Pipes went a step further by crafting mini-platforms.

Raffel, an electronic musician, was no stranger to this concept. Prior to his involvement with Pipes, he had extensively explored diverse computer-based visual programming environments for music control. Raffel recalled Kyma, a graphical sound-creation system that enabled users to connect audio synthesis and sound processors, as well as a dynamic video and audio editing combination still around that’s called Max/MSP/Jitter.

Coincidentally, Sal Soghoian, also a musician, was familiar with diagrammatic graphical interfaces. He mentioned that there was an unreleased version of Automator featuring modular blocks with detachable wires that could be connected and disconnected at will. None of these were the first node-graph style UIs, but they were some of the closest in proximity to Pipes.

Spreadsheets effectively popularized the concept of reactive programming with the wider world—an early demonstration of the power of automatic updates based on input changes. Sadri shared that the team looked to them as a gold standard for coding for non-programmers. “[They’re] actually one of the most popular programming environments in the world,” he said, describing them as “the only programming model where you can actually see the data.” Indeed, if something goes wrong, a spreadsheet provides immediate feedback—no compilation and log-based debugging necessary.

Pipes offered reactive data pipelines and dynamic workflows designed to provide that same real-time visibility: a Pipe started running as soon as any element was connected, using visual indicators to show action. “We wanted the data that is flowing through the ‘pipes’ to be very visible,” Sadri said. An optional text debugger at the bottom of the screen provided additional feedback by displaying the current results as individual items.

“It was really hard to write the wrong program in Pipes,” Sadri noted. “As soon as you dragged something on screen, it would immediately start working and produce some output.”

But developing Pipes as an early visual programming tool didn’t take visual programming—it took code. And the code was developed rapidly and iteratively, the result of hurtling towards an unspecified deadline. The team sometimes resorted to shortcuts, banking on the expectation that if the project were turned into a complete product in the future, they or others would refine it accordingly.

Ed Ho said most of the underlying code was in Perl, which was popular as a scripting language for server-side systems with extensive text handling requirements. However, it fell short when it came to meeting the production needs Pipes ultimately demanded.

“The Perl was horribly inefficient; it didn’t scale very well,” Trevor noted. “It was designed for what Pipes was built to be, which was a fairly quick v.1 to proof of concept.”

Chapter 7

Judgment day and an O’Reilly encomium

media

In early 2007, after months of development with no fixed end point, despite presenting progress and demoing features internally—the hammer abruptly came down. Raffel recalled the day when a higher-level boss gave them two weeks to complete Pipes.

“It was so ridiculous,” he said. “We were just not two weeks away.”  But launch they did.

It’s little surprise that the subsequent launch came as more of an explosion than a graceful liftoff. If it wasn’t due to the lack of clarity on timelines and Yahoo’s expectations punctuated by a sudden time crunch, it might be chalked up to modesty. None of the team, including management, expected Pipes would see heavy usage. The team had been sketching, albeit in rather rich detail.

Trevor recalled a woefully low-server setup: “We had an old decommissioned mail machine that we were running everything on.” Others suggested they had a bit more computational power than that at launch—Raffel recalled being allocated some number of servers—but today, the exact details are fuzzy. What’s clear is that they weren’t prepared for the number of users to come.

“Our internal estimates were more like, hey, maybe we will have like a thousand users when we launch,” said Sadri.

A few days before the beta launch, Caterina Fake, then heading up Brickhouse, brought Sadri to show Tim O’Reilly a demo. O’Reilly, founder of the eponymous O’Reilly media, had enough sway to literally break the Internet—and this time would be no exception.

Yahoo Pipeso-reilly-books.jpg

o-reilly-books

He wrote something of a paean to Pipes in the O’Reilly Radar on February 7, 2007, posted as the service went live.

“Yahoo!’s new Pipes service is a milestone in the history of the internet. [...] It democratizes web programming, making it easier for people to have more control over the internet information services they consume, and providing a general-purpose platform for interacting with sites that is more powerful than the browser or feed-reader alone, but without requiring full programming skills.”

“He was very kind to write this very nice review of it,” Sadri reflected, “and that exploded interest, and caused our servers all to melt down.”

As launch day went on, usage didn’t taper off. The demand for Pipes required vastly more resources than it had available, and Yahoo put all hands on deck to support keeping the nascent system running.

Ho remembers “just sitting in my chair the entire [launch] day until late in the evening, just bringing up machines.” Yahoo’s data-center team started shifting servers that they had already prepped for deployment for other groups into service for Pipes.

“Every time it came up, everybody jumped on it and it crashed again, and everybody cycled back into a waiting state,” said Trevor, the lead engineer. “It felt bad at the time, but in retrospect, it was awesome.”

Raffel said Yahoo co-founder Jerry Yang told him to use his name to get whatever he needed to keep Pipes running then: “I’d never done that before. I certainly haven’t done that since.”

The beta launch’s popularity caught the team so off guard that they hadn’t even prepared a crash page. (Though that soon changed: Trevor said that the first thing they created after the beta launch was a page that read something like “our pipes are clogged.”)

In the days and weeks following, the team started to see more of the loose threads they hadn’t been able to stitch up in the sudden sprint to an unexpectedly momentous launch. Among a variety of issues, once someone created a Pipe, it ran forever.

There was no end-time built in or a need for someone to login to refresh or request a feed. (Even if there had been, RSS newsreaders polled for updates automatically.) Raffel said the team didn’t consider “just how many zombie things would get generated. [...] We kind of put ourselves in a situation where, fairly quickly, we had a lot of usage from things that may or may not actually [have been] real users.”

Still, many others agreed with O’Reilly’s sentiments. The InfoWorld writer turned Microsoft evangelist Jon Udell blogged about a week after the beta launch, “The dominant way in which most people will ‘program’ the web is by writing metadata, not code, and we’ll need an interface as friendly and powerful as Pipes to help them do that.”

Yahoo threw more servers at Pipes, the programmers optimized the least inefficient code, and within a few months, Pipes’ reliability stabilized. But just as it was rising it was also—already—nearly fixed in amber.

Chapter 8

Clogged

Pipes’s downfall was seeded in its launch. Expanding the service from a mere concept to a fully developed product, whether free or not, would have required a significantly larger team and a well-defined mission.

Yahoo DevNet’s Kent Brewster said,

“We would go to all these conferences and we’d have a booth…and people would go, wow, this is super impressive. But when they asked how much they could pay for service, we had to reply, ‘We’re not really at that point yet.’” He explained, “You could not purchase some number of guaranteed-to-work Pipes calls per month. It absolutely wasn’t a thing. Without that, no professional in the world, let alone a large company that would actually put some money in—nobody was going to invest in that.”

(Brewster noted this was true of everything built in these blue-sky development projects.)

Yahoo’s trajectory was soon to change, too. Just after laying off about 1,000 employees in January 2008, it rejected a $44.6 billion offer from Microsoft in cash and stock. Later that year, it laid off another 10% of its staff, about 1,500 people. The company failed to experience any notable improvements, even after the appointment of Marissa Mayer from Google as CEO.

The highlight of Yahoo’s financial performance from 2008 until its sale and effective dissolution in 2017, came in 2012 when Mayer agreed to sell their stake in Chinese e-commerce company Alibaba. The value of that Alibaba stock, which Yahoo had acquired in 2005 for $1 billion, had soared to an impressive $7.6 billion at the time of the sale.

Tangled mouse

Yahoo Pipeswhiteboard.jpg

whiteboard

The quasi-skunkworks operations at Brickhouse had already produced several projects lionized in the technical press in 2007. These included BravoNation, an online system for giving kudos; FireEagle, a service that mediated privacy while providing location information; KickStart, a job placement site for college students (unrelated to the Kickstarter crowdfunding platform); and Yahoo Live, an early offering for live webcam streaming.

None ever turned into successes. Fake went on extended pregnancy and maternity leave starting in April 2007, handing off some roles for that period; she left Yahoo in June 2008. The division head, Bradley Horowitz, left Yahoo for Google in February 2008; Salim Ismail, put in charge of Brickhouse during Fake’s absence, departed in March 2008. Dickerson, who took over TechDev from Fake, left around when she did in mid-2008. Brickhouse was shuttered by the end of that year.

The Pipes team sheared off, too, either leaving Yahoo or joining other teams, handing over support to caretakers within the company.

A few months after the launch of Pipes, Sadri left to establish Polyvore, a platform enabling users to share collages of images, frequently used for fashion and interior design. Polyvore was acquired by Yahoo in 2015; Sadri is now at Sutter Hill Ventures. Ho left around the same time as Sadri; he’s currently a stay-at-home dad. Cheng and Raffel left within a year of Pipes’s launch. Cheng re-engaged with Yahoo in 2014 for nearly two years, and now describes himself as a musician and producer. Raffel’s most recent role was as a product lead at Slack, and he’s currently researching his next venture.

Pipes itself—well, Pipes lingered for years.

Trevor remained at Yahoo through 2010, seeing Pipes through a transition that would otherwise have doomed it then and there. “The reason [Pipes] was even still going when I left was because I’d carried the project with me as I went between different teams,” he explained.

Trevor further elaborated that he rewrote the project from Perl into Java, ensuring compatibility with the internal YQL language, that was, for a time, required for Yahoo projects. Beyond that, “it was only a question of when it was going to be turned off.” Trevor eventually left Yahoo to join Sadri at Polyvore. (He later became the co-founder of Observe, created by Sutter Hill Ventures.)

Of the five core members that saw the project through to launch, all remember the excitement of being on a small team that produced so much. “We were in a very special place where everything was just, like, clicking, and we were really creating a lot of new things that became the norm on the Internet during that project,” said Sadri. “The magic of having these small, great teams,” Ed Ho said, has stuck with him.

Yahoo ultimately dismantled Pipes in 2015. Some of the team members didn’t see the news; others felt only a small pang of nostalgia or sadness. But its “end of life” announcement on HackerNews brought a wave of nostalgia from still-active users looking for alternatives, and those who saw the potential for what it could have been.

Sadri, who engaged with the threads, posted at the time: “I wish Pipes was launched in the age of containers (e.g., Docker). We had the idea of one-click deployment to what we today call a container.”

One poster wrote that Pipes was “one of the best demonstrations of flow based programming to date. I really loved Yahoo pipes [sic] and it influenced the way I think.” Another shared that “Pipes and Visual Basic shaped my view of how fun and easy programming can be,” an idea that was reinforced in various flavors by other posters time and again.

Yahoo PipesScreenshot 2015-06-04.jpg

Screenshot 2015-06-04

Chapter 9

The downstream result of Pipes

Yahoo PipesiPhone Workflow App.jpg

iPhone Workflow App

Pipes was inspirational to a generation of programmers, user interface designers, and product people. Just in the way the Pipes team ran with ideas they saw in Apple’s Automator, later projects seem to owe a lot to Pipes’ conceptual DNA.

One of the closest tools in some regards to Pipes was the iPhone/iPad automation app Workflow, purchased by Apple in 2017, then shut down, integrated, and redeployed as Shortcuts in iOS, iPadOS, and later macOS.

Andrew Cantino, creator of Huginn, a non-visual agent-based system that performs automated tasks for you online, wrote on Hacker News in 2021, “Yahoo! Pipes inspired me to make Huginn.” Node-RED, launched in 2013, has a distinct Pipes vibe.

The list of tools—open and closed source, popular and obscure—that seem influenced by Pipes, or that even come up as “Pipes alternatives” continues to proliferate.

More broadly, Pipes was an anomaly in its heyday in that it offered something akin to programming for the non-coding masses—while creating entry points to programming for others and enthusiasm amongst developers eager to experiment with neoteric means of building (often for fun). On its end-of-life HackerNews thread, poster juliendorra shared:

“I used Pipes as the basis of data mashup classes, with non-developers students that were mostly afraid/bored of anything related to programming. Using Pipes, in just a few hours they would make small, custom data apps.[...] I had a student using the location entity extraction to geocode his favorite street art blog that cited cities in the text but didn't use rssgeo, and then output the augmented feed to a map view. [...] It was a really effective tool for non-developers to learn about data markup standards, to think in term[s] of data flows, to get introduced to the idea of a data or web API.”

Greg Wilson, long involved in teaching coding and data science skills to researchers, said that visual-programming tools abound as profession-specific tools, with nearly every industry and discipline having one or more prominent block-connection coding apps, many of which predated Pipes.

“All high-end animation in Hollywood is done using tools that look like Yahoo Pipes—it is all wiring together a data flow set of blocks.” Further, he noted the popularity of MATLAB, a mathematical modeling platform used by engineers and scientists with an estimated four million global users, many of whom use a visual programming interface for it.

Yahoo PipesScreenshot 2007-02-07.png

Screenshot 2007-02-07

Visual programming—from no- and low-code tools to “yes code”—is growing in popularity, but to Wilson, it seems odd that graphical tools haven’t been widespread for general purposes. While there could be any number of reasons, one might relate to copyright, though it never quite seemed to crop up for Pipes. As Anil Dash, then an executive at Six Apart, makers of Movable Type and other blogging tools, wrote in a comment on O’Reilly’s benediction on Pipes:

Is the existence of a feed implicit permission for people to subscribe to it? For people to poll it frequently? For people to republish it? Amazingly, almost ten years in, we still don’t have answers on those questions, and now we’re getting into remixing, which is even more fraught.

Sixteen years later, those questions still haven’t been answered, which provides a de facto chilling effect. In the United States and many other countries, there’s no simple way to determine whether copyright is violated without going to court—either being sued or, as with the “Happy Birthday To You” lawsuit, filing for a declarative judgment that something isn’t a copyright violation and spending years and fronting huge sums to win.

That lack of certainty and risk of lawsuits could deter any firm that wanted to launch a free or fee-based service until a landmark case provided strong guidance about what’s acceptable.

Even were the copyright issues to be settled, the current Internet has clamped down on freely available information feeds and even more so on “scraping,” the act of retrieving a regular webpage that wasn’t intended to act as a feed and parsing data from it. (Scraping was a breeze in Pipes.)

Sadri noted that many legitimate APIs  “became commercial APIs that are gated behind some kind of a paid plan.” Still, the appreciation of Pipes lingers. When I asked people for Pipes’s recollection on Mastodon, I received an outpouring of memories and fondness. One poster, Pat, summarized the sentiment: “Even today, there are a lot of problems that I have solved in my day-jobs that could be better served by Yahoo Pipes.”

Who knows if shaking the dust from some of the team members’ memories could still find seedlings ready to plant? All the team members had nostalgia about the era in general and Pipes in particular.

Near the end of an interview with Sadri, he said, “Sometimes, I think maybe we should do Pipes 2. There’s just a lot of amazing infrastructure now available to do a next-level version of it.”

This story was most recently updated in December 2023.

We still <3 visual workflow automation. How about you?

[Yes]
[No]