Swine flu — there is at least one good thing about it…

We are in the midst of an information storm about an apprehended flu pandemic. Data clearly shows a spike starting 4-5 days ago, in the news as well as in the blogs, where the number of indexed documents referring to flu, of any kind, shows a ten to hundred–fold increase. That is not surprising, would you think. But what is, at least to me, is that Google trends' special flu microsite shows absolutely no sign of increasing activity.

Wikia search is closing

In my previous post I reported that the open-search index Wikia had stopped working. Earlier today I stumbled across this blog entry from Jimmy Wales, yes — Wikipedia's founder. Wales says that his foundation has stopped funding Wikia Search.

Is there such a thing as Spring fever for search engines?

On April 1st, Alexa has changed its search logic moving from documents to site-level searching. Probably a good thing actually.

Alexa provides a new and potentially interesting visibility metric

Alexa (an Amazon company) used to return the number of entries found in the top n sites. I routinely queried it to get the number of mentions for specific concepts appearing in the top 100 sites. 

The accuracy of web-based visibility metrics

Just a quick post to introduce a research note I am just about to file. It presents web-based visibility metrics harvested during the US presidential race (vote held on November 4th 2008) and the Canadian federal election (vote held on October 14th 2008).

The most striking finding is the almost super-natural precision of visibility metrics culled from the news search engines. Visibility shares on the eve of the vote were off by 0.2% and 0.3% in the Canadian and US election respectively.

Table 1: Summary of recent results

Below are radar charts of various metrics computed between 03:00 and 05:00 on the day of the election, where day-to-day indicators (essentially news and blogs) refer to one day earlier (i.e. on the morning of November 4th, news search engines were queried on content published/indexed on November 3rd).

Further below are time series showing how visibility has evolved during the races. It should be noted that indicators other than news are mere averages of a collection of ratios computed for each relevant search engine (see the list at the bottom of this post). For the news index, the metric is the arithmetic average of the returns reported by Factiva, Google News and Yahoo! News for the Canadian race. In addition, NorthernLight and Bloglines were used to compute an average score for the US race. These rules were established based on a reliability assessment made on independent data (no fishing here).

I plan to recalibrate each index before the end of the year, and publish revised timeseries.

As a final note, the last chart traces the news visibility index published by Factiva in 2004. This is perhaps what is most important — just a few years ago, media where both off and slow to react. My inclination is to think that blogging has changed the nature of news reporting, although we could also argue that indexing and searching have improved.

Fulltext is here.

Figure 1: Visibility shares, Canada, October 13th

Figure 2: Visibility shares, US, November 3rd

 

Figure 3: Visibility shares, Stephane Dion

Figure 4: Visibility shares, Barack Obama

Figure 5: Visibility and delegates shares, Howard Dean & John Kerry, 2004

Table 2: List of search engines

Weak visibility measurements require robust procedures

In my previous post I showed that visibility measures taken from News Search Engines (NSE) appeared to be very reliable, with a test-retest Mean Absolute Percent Accuracy in excess of 99% and a scale Cronbach’s Alpha in excess of 0.992. I also warned that these results had to be tested on a larger sample of concepts.

In this post I’ll focus on the robustness issue. I’ll follow-up with a detailed summary of my findings on the reliability of NSEs, in my next post.

First, recall that a reliable measure is a measure that is consistent. We can appreciate reliability by taking repeated measures of the same metrics. The consistency of test-retest is certainly desirable but clearly insufficient as a defective instrument returning constant results would be consistent yet unreliable. We must therefore take repeated measures of a metric on different subjects. A reliable instrument will be able to consistently discriminate between subjects (i.e. if you weight people and rank them from the heaviest to the lightest, a perfectly reliable scale would consistently return identical ranks). A better way is to take repeated measures using different instruments. That is the idea behind "convergent reliability".

These ideas come largely from the field of psychometrics, where the distribution of errors could reasonably be assumed to be somewhat normal. But this is not the case with web metrics, where errors can be extreme.

So let me introduce the idea of robustness and contrast it with the idea of reliability. (see a wikipedia article here). A robust instrument is an instrument that always returns reasonable values, with normal errors. A weak instrument is an instrument that occasionally breaks down and may return extreme values. Think of a bathroom scale — a robust scale always gives your weight, give or take a couple of kilos; a non-robust scale will sometimes return erratic values like zero kilo or 900 kilos. Non-robust could be conceived as "delicate", where a deliberate trade-off has been made such that usually the result is very accurate but occasionally something goes wrong, or merely as "weak" where freak events occur in addition to normal errors.

Consider the figure below, where 100 consecutive measurements are reported. Both instruments are identically reliable (99.5% accurate). The difference is that errors are random in one case, and extreme in the other. Now, let me ask: which scale would you rather use, the robust scale or the delicate scale?

My answer is that it depends — if you use a robust estimation technique, you should be able to detect extreme values and you will prefer series 2, produced by a delicate instrument which is usually right on. If, on the other hand, you cannot tell if a value is extreme or not, then you may prefer series 1, produced by a robust instrument which is never far off the mark.

Extreme values occur quite frequently on the web. In my next post I will report on a set of 9 queries about 1 000 concepts. Out of 9 000 queries, 39 were obvious outliers (unable to get a result after 6 tries). But this is the very tip of a quite large iceberg as there are hundreds of suspect values, including some really really extreme outliers.

This is both not really surprising nor trivial. Not surprising as extreme errors may have several causes, either at the source (a NSE returns an erroneous value) or during the treatment (a parsing error). Not trivial as the magnitude of these extreme values dwarfs the true correlation. Consider the scattergram below, where counts about 1 000 queries to Google News and Yahoo News are displayed.

For some reason, on October the 7th, Yahoo! reported a count of 585 000 news items for the word "Loveridge" compared to just one instance in Google News. This is not a parsing error as AllTheWeb, owned by Yahoo! returned 750 000 instances, with other NSEs returning very low counts.

At the other end of the spectrum, Google News reported a count of 72 000 news items for "Lies" compared to 1 071 at Yahoo! News. Other NLEs also reported counts in the neighborhood of 1 000.

Keeping these two extremes in the dataset will yield a very weak correlation of .02 between Google and Yahoo! News. Removing just 2 observations will bring the correlation close to .75!!

The figure below shows the same points as above, after removal of these 2 extreme outliers.

So what? First, take more than one measure. Three at least because if you are confronted with a "Loveridge case" where one NSE says "very many" and another says "very few" you’ll need a third if not a fourth data point if you want to be able to tell which is which. Second, eyeballing extreme values will not do it. There are just too many data points, many of which are not obvious calls. Since NSEs are not robust instruments, a robust estimation is a must.

Visibility metrics in the news media – a few observations

Before I present and briefly discuss the data, let me clarify a few notions.

By metric, I mean "some abstraction that can be
measured." What you weight is a simple metric. How happy you are is
not. In this post we deal with a simple metric — how visible is a
concept (say… football) in the news media.

By measure I mean applying an instrument to a metric. Up on a
scale and we can make a reading of how much you weight. In this post,
the instruments are news search engines. 

By index I mean an aggregate of measures. The main purpose of
an index is to yield a more reliable figure. In this post I report on
the following news engines : {Ask, AllTheWeb, Bloglines, Google, Factiva, Live, Topix and Yahoo!}

By reliable I mean that an instrument produces consistent
estimates of a metric for a specific concept, i.e. if you take the
measure twice, you should get the same value.

By robust I mean that an index is designed in such a way as to appropriately discount bogus values. More about this in a future post.

—

To follow on the sports example I introduced a couple of posts back, Table 1
shows the number of results returned by each news search engine for 5
sports. Whenever possible, the search specified a single day (October
1st), but no region nor language restriction.

Two observations should be made. First, there are fairly large
differences across engines. Not considering Ask, Live nor Topix’s
figures (see notes (1) and (3)), counts range from a low of 2,293 news
items to a high of 6,740 news items for football. Second, all engines
produce consistent estimates — on successive requests, counts returned
by an engine will generally not vary by much.

Consistent yet different counts need not be a matter of concern if
counts vary merely by some fixed proportion (i.e. Yahoo! always
returning more items than Factiva). If it were not the case (i.e.
sometimes, Yahoo! claims more items, sometimes Factiva does), then
there would be a problem. To answer this, we must correlate instruments
(the search engines) across concepts (the sports).

Table 2 shows how results correlate across news search
engines. We can readily see that Ask and Live are poor indicators in
our example, because counts have reached their ceilings, therefore
providing no information on relative visibility — according to Ask,
cricket is the most visible sport, dubious as all other engines but one
put it at the bottom. We can also see that Topix’s overall stock of
news items correlates as well as any other engine.

Table 3 shows the customary reliability statistics for our
array of measures (which could become an index).  The Cronbach’s alpha
value (more or less similar to the average correlation between items,
varying between zero for pure noise, up to one for totally reliable)
can be very high. As high as 0.992 if we merely remove counts returned by Ask and Live.

So far so good, but not good enough as the concepts used in this
example are in no way representative of the search universe. In my next
post I’ll present results derived from a much larger and reasonably
diverse sample of concepts.

Table 1

Notes:
(1) Notice that Ask and Live cap results. Ask will not
return counts above about 800 pages; Live will not return counts above 1100 pages or so.
(2) The results figure provided by Google News
appears to refer to the total stock of active news items (presumably a
month). If you click to list news items of the past week, day or hour,
the results count stays the same. The numbers I report are my own
estimates, based on the number of items available or the rate at which
the past 1000 items have been published.
(3) Topix doesn’t allow narrowing the search to a specific day or date. Figures refer to the stock of active news items
(4)
MAPA stands for Mean Absolute Percent Accuracy. It is computed as (1 –
MAPE) where MAPE is the well know Mean Absolute Percent Error,
routinely used to compare various forecasting methods. These figures
were computed based on 5 bursts of 50 consecutive requests made at
intervals of less than 1 second (because differences between longer
intervals might arise because real changes have affected the underlying
metric, i.e. new news coming in, old news moving out).

Table 2

Table 3

Note: (5) The 6-item index excludes Topix, Ask and Live ; the 7-item index excludes Ask and Live

Is visibility everything? My 2-bit sentiment analysis

Today’s question is whether it makes sense to rely on raw visibility metrics, without any consideration given to what is visible. As I pointed out earlier, raw visibility metrics correlate quite well with the apparent popularity of candidates in various electoral races, so why bother with content?

Well… John Edwards was highly visible in the news last August, for events that have had a major, negative, impact on his political fortunes. So any tracking system worth its salt should provide some information on content as well as on the visibility of such content, isn’t it. And several start-ups promise to do just that. Jodange, Sentimine, Corpora. And there is this amazing academic site.

But sentiment analysis — as it is often called — is not an easy task. (read this entry for a feel).

If you are to rely on automated procedures, there are syntaxic traps. For instance, would a Natural Language Processor (NLP) understand that "XYZ is not good, he (she) is unbelievable!" has positive sentiment or would it be fooled by the "not good"?

And context matters (this phone is very small / this apartment is very small). A smaller phone is preferable to a larger one, at least up to a point, and a large apartment is worth more than a small one. Yes, we can teach machines, but there are so many variables that the task is herculean.

And then we have to consider that a fact may be interpreted differently by different persons. "This politician is pro-life" (i.e. opposed to abortion) could carry a positive, neutral or negative sentiment. Depends on the writer’s views, who may support, attack or merely state the politician’s attitude. And then, suppose a pro-lifer writing (favorably) about a pro-life candidate. That doesn’t mean that the reader will interpret the statement favorably. And then consider the case of a pro-choice writing favorably about a pro-life candidate?

Then what? One way is to forget about bases for sentiment. After all, facts are not sentiments (even though several facts carry obvious sentiments — headlines such as: YXY convicted of first degree murder). Ignoring facts, we may focus on valence and care only for statements such as "[so and so] is cool, [so and so] is good, I like [so and so]"

So? My 2-bit sentiment analysis goes like this: Compute the ratio of pages including the word "good" over pages including the word "bad" in addition to the concept of interest. For instance, how many pages are returned for the query ["Microsoft Vista" +good]? How many pages for ["Microsoft Vista" +bad]? Compute the ratio (or the share if you prefer). Et voilà! Cannot be simpler.

Is this a joke or a useful metric? Well consider this comparison between Microsoft’s Vista and Apple’s OSX. Graphs below suggest that Vista is more visible than OSX whereas OSX has higher 2-bit sentiment score on all dimensions except the NEWS vector. Plausible.

And how does such 2-bit sentiment analysis fares in the political realm? Surprisingly well. Below are scattergrams of sentiment scores for leading candidates in the USA and Canada races comparing the 2-bit sentiment metric with textMap’s NLP derived scores (textMap doesn’t provide scores for Sarah Palin nor Elizabeth May, two relatively recent figures in the political realm).

We can see that both metrics agree quite well on relative sentiment. In fact, it would be difficult to decide which scale is more appropriate. On the other hand, a case-by-case examination of the 2-bit sentiment metrics is not for the faint of heart as the words "good" and "bad" are very often used in completely irrelevant contexts (a candidate is interviewed on "Good Morning America), or have diametrically opposing sentiments (a candidate was "not good").

But let’s not worry for the moment as the first order of business is to assert the general reliability of www-based metrics. If we can establish that they are — not to worry, they can be — it will certainly be interesting to explore the validity of "automated n-bit sentiment metrics" 🙂

The visibility game

I had written that I would do my best to update visibility charts for leading political candidates. So here they are: results for the Canadian race, and results for the US presidential. You may subscribe to daily updates by clicking on the xml tag at the bottom of these pages.

News visibility scores for Canadian candidates put Harper ahead of Dion at 41.8% vs 26.7%. Striking similarity with the latest Canadian Press Harris-Decima survey published this morning, which puts Harper/Dion at 41%/26%.

On the US scene, scores suggest that the Republican ticket is ahead, largely because of the Palin sensation who dominates the "social" vector (a composite of such things as backtype, twitter, delicious, digg, etc.) where McCain does not do well. Here also, results are matched by polls.

Still very "alpha", so expect several changes. In particular, results are currently nothing more than averages from several distinct sources. Still, no doubt we can learn from this.

Four reasons to be concerned about visibility metrics

Google "baseball", "basketball", "cricket", "football", "hockey" using advanced search preferences to obtain the number of results for pages located in Canada, India, the UK and the USA. Here’s what I got on September 3rd 2008.

Not so bad. Hockey is Canada’s favorite. Cricket is India’s. Football in the UK and the USA. Makes lot of sense. Visibility metrics appear to be useful.

But wait! Football doesn’t mean the same thing in North America and elsewhere. North Americans say "soccer" when everybody else say "football". So what do we have here? And then look at this. First page for cricket in Canada. Oh, yes, cricket also has several meanings (there is the sport, there is the insect, there is the Disney’s character, etc.). But these are validity issues. With hard and intelligent work it is conceivable that we could weed out irrelevant results and zero in on our target concepts. And with more hard work it is conceivable that we could identify the underlying relationship between "www visibility" and "popularity", whatever we mean with these definitions.

But wait some more! Is it really possible that Canadians prefer cricket to football (26 000 to 16 400)!? Even accounting for the Jiminy effect, something must have gone wrong somewhere. These figures may not be reliable. (Remember that reliable measures are a pre-requisite for valid inferences, and that reliability means that measures are consistently reproducible).

There are important reliability concerns :

1) the numbers are estimates. Fair enough. It would be next to impossible to return exact data as the web is in constant flux, as queries are handled by different servers, and so on. (I will illustrate later how much this can be a problem — in a nutshell, it is significant when big news stories erupt).

2) Repeating the exact same query several times in a row will often return different results. In general differences are small, but they can be surprisingly large (less than half of the previous estimate, obtained 2 minutes ago). And sometimes totally off the scale. Visibility metrics are not very robust.

3) Pages do not materialize. Try a Google search for "cricket" (no quotes) in the USA, updated within the past 24 hours. I got 67k results. But only three pages of results(!!??). Clicking to go to the last page returned results 201-211 (fair enough), but a revised estimate of the total results to match (bummer!). I am now told that that are 211 web pages about "cricket" from the USA updated within the past 24 hours, down from 67 700. This is not a trivial difference — the revised figure, no longer an estimate, is a paltry 0.3% of the initial figure. We can guess at various explanations, but the bottom line is that the results figure will dramatically inflate the visibility score of more visible concepts (whose score will not be truncated for consistency). To put it differently, if Google says that there are 67k pages yet they can show no more than 211, should I use the 67K figure? Or should I use estimates for some concepts and hard results for others, knowing that hard results may be a tiny fraction of estimates?

4) Different engines give wildly different results. AlltheWeb (now part of Yahoo!) returns no less than 1.4 million pages resulting from a search for "football" in Canada, updated during the past 24 hours. Compared to 16 400 pages from Google. A hundred-fold difference. Is it a difference caused by the engine? (i.e. does ATW systematically report much larger estimates than Google?), a sign that estimates are unreliable (i.e. for other queries, Google could return estimates that are much larger than ATW’s), or just bad luck? (i.e. for most queries, ATW and Google roughly agree).

—-

What to make of this?

Even though the number of results returned by a query contains "some" information as we can see in the sports-by-country example, these numbers vary considerably and sometimes unexpectedly. In the absence of a reliability gauge, reaching conclusions based on visibility scores is a risky business.

In my next post I will explain how I am trying to build a reliable and robust measurement scale of… let’s call this "visibility" for the moment, even though the term is misleading.