Self-managing Internet Applications Flex Their Muscles

The European research project SELFMAN (http://www.ist-selfman.org/wiki/index.php/SELFMAN_Project) has created a programming system that makes it much easier to build high-powered distributed internet applications that manage themselves.

Distributed applications tap the internet’s power by enabling hundreds, thousands, or even millions of devices and programs to link up and work together. These are the invisible programs that let users bank, buy and play games online, search for information, share files, and network with applications like Facebook and Twitter.

SELFMAN ensures that all those moving parts stay in touch, up-to-date, and work together smoothly without the teams of experts that have been needed to tweak, patch or protect the system as components come online or drop out, or in response to communication or component breakdowns, and even deliberate attacks.

“We want to make this kind of application easy, so people can really take advantage of this huge network of computers and small devices,” says Peter Van Roy SELFMAN’s coordinator based at the Université Catholique de Louvain, Louvain-La-Neuve in Belgium.

SELFMAN’s programming architecture and set of components have already sparked new applications that utilise and showcase its capabilities.

“It’s the first time that we’ve been able to get large internet applications with many computers talking together, when all the applications can take care of themselves,” says Van Roy. “Now we are taking this to the next level.”

What’s wilier than Wikipedia?

When you pitch a question to Wikipedia, you’re likely to be one of some 2000 people trying to access it that second. Wikipedia manages those 2000 requests per second, plus the constant stream of updates to its database of articles, by using at least 16 powerful database servers arranged in a master-slave architecture.

Researchers at the Zuse Institute Berlin (ZIB), one of SELFMAN’s academic partners, decided to test the SELFMAN approach by implementing Wikipedia’s core database in a flexible, scalable and self-managing distributed system.

They based their Wikipedia on Scalaris, an open-source distributed database system that incorporates SELFMAN’s self-configuring, self-tuning and self-healing capabilities.

“Every piece of information in our Wikipedia exists four times – that’s a number that we found works well – and is spread over all the nodes,” says Van Roy. “A clever algorithm called Paxos does the heavy lifting to make sure the nodes all agree.”

They expected that their Wikipedia would have several advantages, including higher reliability, lower operating costs, and scalability – the ability to add new capacity easily in response to increasing demand. The Scalaris-based Wikipedia met all those expectations.

“With our wiki, if ten times as many people start to use it, you just add more machines,” says Van Roy. “We can handle far more users than the actual Wikipedia.”

In addition, since Scalaris takes care of itself in the unpredictable internet environment, the costs of keeping it running are significantly reduced.

ZIB’s Scalaris-based Wikipedia won first prize in the Institute of Electrical and Electronic Engineers 2008 Scalable Computing Challenge.

“It really shows the kind of things we want to do,” says Van Roy.

Streaming video on demand

The wide applicability of SELFMAN’s toolkit is evident in a completely different application – PeerTV, produced by another SELFMAN partner, Stockholm-based Peerialism.

“They had a very different problem,” says Van Roy. “PeerTV needed to manage load between flowing streams of video data coming from a few big servers but going to millions of clients.”

They solved the problem of streaming all that video reliably and efficiently by optimising peer-to-peer (P2P) networks with SELFMAN’s ability to balance loads and transfer data efficiently.

“Until now, P2P wasted enormous amounts of bandwidth because of its random routing of data,” says Van Roy. “Some service providers had to block certain P2P applications.”

In contrast, PeerTV connects components through a well-defined, constantly optimised topology. That “topologically aware” routing reduced network traffic by two-thirds.

Overall, PeerTV found that they could provide the same quality of service as established video providers, easily scalable to an unlimited number of viewers, but with a cost reduction of from 50 to 90 percent.

PeerTV has a direct commercial interest. Peerialism developed it for the Swedish company MPS Broadband AB, and Peerialism itself is currently being acquired by GGF.

Van Roy is excited by the varied applications that are starting to flow from SELFMAN. These include small applications, such as a program that lets users work together on a graphic design using their gPhones, and massive ones, such as a load-balancing system for France Telecom. But he is even more excited by SELFMAN’s implications.

“The actual smartness of the internet comes from applications like Google or Wikipedia, not from the plumbing that just shuffles data around,” says Van Roy. “SELFMAN takes those smart components and amplifies them – makes them automatically more efficient, more scalable and more robust.”

The SELFMAN project received funding from the ICT strand of the EU’s Sixth Framework Programme for research.