Group items tagged

The fundamental flaw underlying both Metcalfe's and Reed's laws is in the assignment of equal value to all connections or all groups. The underlying problem with this assumption was pointed out a century and a half ago by Henry David Thoreau in relation to the very first large telecommunications network, then being built in the United States. In his famous book Walden (1854), he wrote: "We are in great haste to construct a magnetic telegraph from Maine to Texas; but Maine and Texas, it may be, have nothing important to communicate." As it turns out, Maine did have quite a bit to communicate with Texas—but not nearly as much as with, say, Boston and New York City. In general, connections are not all used with the same intensity. In fact, in large networks, such as the Internet, with millions and millions of potential connections between individuals, most are not used at all. So assigning equal value to all of them is not justified. This is our basic objection to Metcalfe's Law, and it's not a new one: it has been noted by many observers, including Metcalfe himself.

Metcalfe's Law does not lead to conclusions as obviously counterintuitive as Reed's Law. But it does fly in the face of a great deal of the history of telecommunications: if Metcalfe's Law were true, it would create overwhelming incentives for all networks relying on the same technology to merge, or at least to interconnect. These incentives would make isolated networks hard to explain. To see this, consider two networks, each with n members. By Metcalfe's Law, each one's value is on the order of n 2, so the total value of both of these separate networks is roughly 2n 2. But suppose these two networks merge. Then we will effectively have a single network with 2n members, which, by Metcalfe's Law, will be worth (2n)2 or 4n 2—twice as much as the combined value of the two separate networks. Surely it would require a singularly obtuse management, to say nothing of stunningly inefficient financial markets, to fail to seize this obvious opportunity to double total network value by simply combining the two.

Zipf's Law is one of those empirical rules that characterize a surprising range of real-world phenomena remarkably well. It says that if we order some large collection by size or popularity, the second element in the collection will be about half the measure of the first one, the third one will be about one-third the measure of the first one, and so on. In general, in other words, the kth-ranked item will measure about 1/k of the first one. To take one example, in a typical large body of English-language text, the most popular word, "the," usually accounts for nearly 7 percent of all word occurrences. The second-place word, "of," makes up 3.5 percent of such occurrences, and the third-place word, "and," accounts for 2.8 percent. In other words, the sequence of percentages (7.0, 3.5, 2.8, and so on) corresponds closely with the 1/k sequence (1/1, 1/2, 1/3…). Although Zipf originally formulated his law to apply just to this phenomenon of word frequencies, scientists find that it describes a surprisingly wide range of statistical distributions, such as individual wealth and income, populations of cities, and even the readership of blogs.

Zipf's Law can also describe in quantitative terms a currently popular thesis called The Long Tail. Consider the items in a collection, such as the books for sale at Amazon, ranked by popularity. A popularity graph would slope downward, with the few dozen most popular books in the upper left-hand corner. The graph would trail off to the lower right, and the long tail would list the hundreds of thousands of books that sell only one or two copies each year.

"Well, the most popular form of XML in the world today is RSS, there is no standard

The idea here is that the community is defined as the relations between the members where the relations have semantical value, where that semantical value is defined by the relations. And I know it sounds like bootstrapping, but we've been doing that throughout history. People exist in relations to other people, to things, to resources, even to spaces.

What has to happen is this mass of posts has to self-organize in some way. Which means there has to be a process of filtering. But filtering that is not just random. And filtering that isn't like spam blocking. Filtering has to be a mechanism of determining what it is we want, because it's a lot easier to determine what we want than what we don't want.

The first pass at this I described in a paper a couple of years ago called "The Semantic Social Network" and the idea, very simply, is we actually attach author information to RSS about blog posts. It kills me that this hasn't happened. Because this is a huge source of information. And all you need to do is, in the 'item', in, say, the 'dc:creator' tag, put a link to a FOAF file. And all of a sudden we've connected people with resources, people with each other and therefore, resources with each other. And that gives me a mechanism for finding resources that is not based on taxonomies, is not based on existing knowledge and existing patterns, but is based on my placement within a community of like-minded individuals.

Now that semantic social network is just a first pass at this. We want to create these connections on many levels. And so what we want is metadata, not simply created by the author of a post, but created by readers of posts. This is what I call 'third party metadata'. Third party metadata -- we're beginning to see some of this out there in the blogosphere, in a small, limited and usually site-based way, right? Links, references, readings, annotations, classifications, context of use. But it can't be site-based. Because that doesn't create a network. It might as well be random.

Now the way this should work, and the way I've proposed for this to work in the educatiuonal community, is that as much of this third party metadata as possible is created through automatic means.

And so we get enormously rich descriptions through very simple mechanisms of automatic classification.

My contention is that instead of the spike-based power-law-based Instapundit-based network, that when we get something like the semantic social network, and we will get something like the semantic social network, because it's very simple to do, patterns of organization will be created. In the field of neural networks and connectionism they tyem 'clusters', you get a cluster phenomenon where we're not creating communities around a specific word, or specific concept, but the community itself emerges as being created by and defined as that particularly dense set of connections.

I've set up a system called Edu_RSS which is a very primitive first pass at this, and the idea here, Edu_RSS is an aggregator, there should be many instances of Edu_RSS, in the ideal world everybody would have something like this on their desktop, and it pulls in the link metadata, but it also pulls in rating metadata, and it doesn't pull it in from the entire world, the way Technorati does or the way Blogdex does, it pulls it in from my community, my network of friends. And if you set up the network in this way you can actually stop worrying about searching, because the network itself becomes the search where you go through layers of linking and so what comes out the other end is stuff that will be of interest to you. And if you're finely grained enough at the output end then you can get a very precise set of inputs. But the thing is, this set of inputs comes from the entire blogosphere of four million people rather than the randomly chosen top one hundred. The community is the network. There is no centralized place that constitutes community, there are only people, and resources, that are distributed, that are all acting on their own behalf and in their own interests - if you ever read Marvin Minsky's "The Society of Mind", it's like that - where the network consists of a set of self-selected relations using a variety of contextual information, that I've defined as third party metadata, to establish meaning, and where this meaning not only defines the community but emerges from the community.

ecommendations are based on the quality of items, rather than more objective properties of the items themselves

The basic idea is: The system maintains a user profile, a record of the user's interests (positive as well as negative) in specific items. It compares this profile to the profiles of other users, and weighs each profile for its degree of similarity with the user's profile. The metric used to determine similarity can vary. Finally, it considers a set of the most similar profiles, and uses information contained in them to recommend (or advise against) items to the user.

One observation is that a social information filtering system becomes more competent as the number of users in the system increases.

The system may need to reach a certain {\em critical mass} of collected data before it becomes useful.

Finally, we haven't even begun to explore the very interesting and controversial social and economical implications of social information filtering systems like Ringo.