Discovering Content by Mining the Entity Web

November 18th, 2009 by Deep Dhillon

I had a blast last night presenting to CS students at the University of Washington. For those who missed the talk, the video is embedded below.

Abstract: Unstructured natural language text found in blogs, news and other web content is rich with semantic relations linking entities (people, places and things). At Evri, we are building a system which automatically reads web content similar to the way humans do. The system can be thought of as an army of 7th grade grammar students armed with a really large dictionary. The dictionary, or knowledge base, consists of relatively static information mined from structured and semi-structured publicly available information repositories like Wikipedia, Crunchbase, and Amazon. This large knowledge base is in turn used by a highly distributed search and indexing infrastructure to perform a deep linguistic analysis of many millions of documents ultimately culminating in a large set of semantic relationships expressing grammatical SVO style clause level relationships. This highly expressive, exacting, and scalable index makes possible a new generation of content discovery applications.

The full talk: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, and the Slides.

Automated Action Based Content Generation

November 9th, 2009 by Deep Dhillon

I’m excited to announce The Attack Machine, an exploratory site available via Evri’s experimental Garden, which leverages the power of the Evri API to automatically generate content oriented around an action, or verb.

Content on The Attack Machine is automatically generated leveraging Evri’s deep linguistic analysis of news, blog and other web content. The Attack Machine is an example application highlighting the power of leveraging a grammatical clause level understanding of verbs. The entire Attack Machine site could be straightforwardly mapped to other verbs, such as love, hate, punch, cherish, destroy, etc.

Content on The Attack Machine homepage is generated by an algorithm which runs every few minutes and looks for the top attackers and victims in particular categories such as animals, locations, weapons, and politicians.


Evri’s system can be thought of like an army of 7th grade grammar students armed with a really large dictionary, or knowledgebase. These 7th grade grammar student algorithms are able to scour the news, blog and other web content to break every sentence down into key grammatical clauses such as the grammatical subject, verb, and object. The attackers in The Attack Machine are, in essence, the grammatical subjects; the verb is always attack and related verbs like: kill, assault, maim, etc.; and the victims on The Attack Machine are the grammatical objects. So, for example, if a blog post contains a simple sentence or title like: “Israel attacks Hamas.”, this post will appear in the attacker page for Israel, and the victim page for Hamas.

In a blog post titled Evri’s Garden Sprouts Some Search I discuss in detail the underlying search mechanism our scientists and engineers use to power higher level API and application functionality. One of the key higher level API functionalities The Attack Machine leverages is an API resource called Get relations about an entity. This API resource is used, for example, to populate the center column in the individual attack pages such as this one on alligator attacks:

and more specifically, the following REST API call is used:

http://api.evri.com/v1/organism/alligator-0×397510/relations/verb/attack/?media=article&sort=date&includeMatchedLocations=true&appId=evri.com/blog

This API call allows us to automatically identify the key people, places, organizations and things involved in attacks, in addition to getting the articles which correspond to the latest attacks. Now if the user clicks on a specific person, place, or thing, the following API call is used:

http://api.evri.com/v1/organism/alligator-0×397510/relations/verb/attack/location/florida-0×3154d?media=article&sort=date&includeMatchedLocations=true&appId=evri.com/blog

So in this way, we can populate the data for all attacks by alligators in Florida.

Finally, The Attack Machine leverages the Evri API to generate unique natural language content which benefits the reader, as well as significantly helps page SEO. For example, the 1st and third paragraphs in the 1st column in the screen shot above are automatic formulations from API output. In addition, the Evri knowledgebase is leveraged to minimize human editorial contribution. For example, the second paragraph above is written by an editor and linked to either a specific entity, a narrow category for an entity, such as animal, or politician, and a higher level category such as person, organism, or location. Simple logic is then applied at page generation time to select the natural language content from an entity handle if it exists, if not, from a narrow category handle, and if none exists (since we have thousands), then from a higher level category (there are only a handful of these) handle.

That’s all for now. If you have any questions on how to use the Evri API for similar applications, or any other feedback, please let us know on our API forum.

Embed Evri’s Sentiment Widget

November 3rd, 2009 by Deep Dhillon

picture-2Check out our new sentiment widget. You can select your favorite topic to seed the widget. Then just select your blogging platform, or grab the code, and embed it onto your web site or blog. Your readers can then get an up to the minute assessment of how the web feels about your topic.

If your topic is Barack Obama, for example, your readers can see that 38% of the web feels positively about him, and 62% of the web is expressing negative sentiment about him. You can also see who Barack Obama’s top critics and praisers are, and then explore exactly what they are saying about the president. Play around with the drop down boxes to get different sentiment expressions found on the web and as always, please do send us your feedback.

On a final note, the Evri sentiment widget is built using our sentiment API which you can read all about in this blog post titled Sentiment API Exposes Web’s Feelings.

EvriVerse 2.0

August 18th, 2009 by alexv

Starting today, the newest version of EvriVerse, our application for iPhone, is available in the iTunes App Store. If you already have a previous version installed you should be able to upgrade for free right on your phone or through iTunes. If you own an iPhone and you don’t already have EvriVerse, what are you waiting for? Go get it!

In this release we’ve added a bunch of new features and enhancements that we think are pretty awesome. It’s free, so why not check it out and let us know what you think.

What’s New

The first change you’ll notice is that we now start the application by showing you the current top newsmakers. This is the same feed that you’ll see on our home page. We think this is a great jumping-off point to do a little exploration.

photo 4.jpg

In this view you can see top headlines and associated entities. You can either read the article or start exploring with that entity in our new graph view. So let’s say that we wanted to dig a little deeper into Les Paul:

photo 5.jpg

We’ve overhauled this view to make it clearer what you can do and making it a little more pleasing to the eye. You can still navigate from entity to entity as before. In addition you can deep-dive on the main entity by touching it.

One of the new enhancements we’ve added is the ability to browse recent images for an entity. In addition to culling article text from the web, we have the ability to track images for an entity. If we were looking at the details for Jimmy Page, we might see images like this:

photo-1.jpg

In this view, selecting one of the images will take you to the original web page that contained the image.

These are nice enhancements but the most exciting new feature we’ve added is the integration of Evri’s Collections. If you already have an account with Evri you can just sign in with your existing credentials. If you don’t already have one, you can sign up for a free one right from your iPhone.

follow-sign-in1.png

Once you’re signed-in, you can start putting your favorite entities in your collections by simply selecting the star at the top of the screen. Your list of collections will appear and you can simply “star” the ones you want to add the entity to. You can even create a new collection here if you like.

news.jpg photo.jpg

Once you’ve started some collections you can go right to them from the main view and see the latest news for all of the entities you’ve added to that collection.

photo2 photo-21

We’ve integrated this feature with the same collections you’ve created on our site so we keep them all in sync for you automatically. Magic!

So try the new version of EvriVerse. Tell your friends and family, and let us know what you think!

Introducing the Evri Cocoa API

August 18th, 2009 by alexv

Now that we’ve gotten version 2.0 of EvriVerse in the App Store, it’s time to talk about some of its internals. In this post we’re going to talk about how EvriVerse works with our public API using our open-source Cocoa library. While the library code isn’t specific to the iPhone, the Xcode project that it is part of has a definite iPhone bent to it. Eventually we’ll enhance it to build a Mac OS framework. For now it has been somewhat purpose-built as a static library for iPhone applications.

Using the EvriApi

Using the API starts with creating an instance of the EvriApi class. When you create one you need to provide an application identifier.

EvriApi *api = [[EvriApi alloc] initWithAppID:@"My-Awesome-Evri-Powered-App"];

Right now we aren’t using API keys, but we would appreciate it if you could put some short descriptive text here so we can differentiate between requests when we analyze our API logs.

You only need one instance of this class (it’s thread-safe) so you should keep a reference to it in some globally-available place such as your application delegate class.

The EvriApi is designed for maximum UI responsiveness. Instead of blocking callers until results are returned from our web API, requests are made with a target object and a specific method selector. When the request completes, the callback method will be invoked on the specified target asynchronously. This keeps your main thread of control from blocking which can lead to a lousy user-experience. All of the API-related methods on the EvriApi class implement a signature like:

// assume we have the method -handleCallback:(EvriApiResponse *)response
NSString *requestID = [api makeRequestForSomething:@"A Thing"
                                   performSelector:@selector(handleCallback:)
                                          onTarget:self];

The callback method will be handed an EvriApiResponse instance which can be queried for success and for the message payload:

- (void)handleCallback:(EvriApiResponse *)response {
  if ([response success]) {
    NSArray *entities = (NSArray *)[response responseObject];
    // do cool stuff with entities
  }
  else {
    // handle error case
  }
}

When making a request, it immediately returns a unique request identifier as a string. You can use this to cancel any outstanding requests if the user switches views before the request completes. OK, enough Cocoa details, let’s dig into the API…

Using the API

Newsmakers

So let’s dig in by starting with our new startup screen. Instead of starting with the graph view, we now give you a view of the top newsmakers as we show them on our homepage. This is done by calling the -fetchPopwhyList:withSelector:onTarget: method. The callback method will be delivered an array of PopwhyItems, which are just simple model objects. We use each entry to fill out the table cells in this view, except for the images which are retrieved asynchronously.

photo 4.jpg

The Graph View

Once an entity is selected from the previous screen, we make a call to -fetchTopTargetsForEntity:performSelector:onTarget:. This call will asynchronously deliver an array of Entity objects to the callback method. In EvriVerse we take the top five surrounding entities and render the graph view like so:

photo 5.jpg

Whenever a user touches one of the target entities, we simply re-issue the same query with a different entity URI and rebuild the graph on the callback. Because the results are delivered asynchronously it was easy to split our animations up into changes that execute concurrently with the request (shrinking down nodes we’re going to throw away) and animations that occur after we have received the query results (growing the new nodes).

Note that the Entity objects returned by this call are not fully “fleshed-out”. If you want additional details you need to query the API for them (explained below).

Entity Headlines

On the main entity in the graph view, the user can select one of four sub-views. The first one is the headlines view. Here we request top news articles for an entity like so:

NSString *reqID = [api fetchArticlesForEntity:@"/person/les-paul-0x182c4"
                              performSelector:@selector(handleNews:)
                                     onTarget:self];

The callback method will receive an array of Article objects which contain the headline, a snippet of relevant text and the original URI.

photo 3.jpg

When a user selects one of these articles, we simply push a web view into view with the original article URL as the content source.

Entity Images

One of the new features we’ve added in the 2.0 release of EvriVerse is the ability to browse recent images for the selected entity. To retrieve images for a given entity URI, make a call like this:

NSString *reqID = [api fetchImagesForEntity:@"/person/jimmy-page-0x35793"
                            performSelector:@selector(handleEntityImages:)
                                   onTarget:self];

The callback method will receive an array of ImageData model objects. Note that these objects only contain the URLs for the images, not the image data itself.

photo-1.jpg

Images are associated with their original source, so when a user touches an image we push a web view with the original web page as the source.

Entity Profile

Sometimes you just want to know who or what a particular entity is. This view provides some high-level details to answer that question such as an image, and a description.

photo.jpg

To get this kind of information you can call the API like so:

NSString *reqID = [api fetchEntityDetails:@"/person/les-paul-x182c4"
                          performSelector:@selector(handleEntityDetails:)
                                 onTarget:self];

The callback method will return a fleshed-out Entity object.

Entity Connections

The last view in our tour of the EvriVerse application is the entity connections view. This is one of the things that really sets Evri apart from other news applications. The connections listed in this view are always up to date based on what’s going on in the news.

photo 2.jpg

In this example we can see a number of other musicians related to the late, great Les Paul. These connections are culled from the vast array of sources we index on the web. Over time these connections change based on what people are writing about on the web.

To get this information, call the API like this:

NSString *reqID = [api fetchTopTargetsForEntity:@"/person/les-paul-0x182c4"
                                performSelector:@selector(handleRelatedEntities:)
                                       onTarget:self];

The callback method will receive an array of Target objects which contain a URI describing the relationship and the other Entity. We use this information to render the table view above.

photo 2-1.jpg

When a user selects one of these connected entities we want to display a list of articles constrained to those two items. For example, selecting “Jimmy Page” will return us a list of articles that contains articles with sentences connecting Les Paul with Jimmy Page. To get this information call the API like so:

NSString *reqID = [api fetchArticlesForEntity:@"/person/les-paul-0x182c4"
                                  otherEntity:@"/person/jimmy-page-0x35793"
                              performSelector:@selector(handleArticles:)
                                     onTarget:self];

The callback method will receive an array of Article model objects.

EvriVerse is just one of what will be many ways that we wrap our public API into a compelling user-experience. With the release of EvriApi under an open-source license we are excited to see what other developers do with it. So go get the library, build something cool and wow us!

Sentiment API Exposes Web’s Feelings

August 11th, 2009 by Deep Dhillon

picture-91Every minute of every day people are expressing their sentiments and writing them down in news articles, blog posts, and other web content. Many people are too famous to write down their sentiments, but journalists, bloggers and other content creators are more than willing to document their feelings. Often times a famous radio commentator will bash a politician, or a politician will thrash a Hollywood actress. And on occassion, a true act of heroism will be recognized, and all sorts of famous folk will follow up with praise. Whether depressing or uplifting, disturbing or unnerving, tapping in to the sentiments of key actors on the world stage can be highly informative and engaging.

I’m excited to announce the release of our new sentiment web API which lets you build applications around the sentiments of specific entities (i.e. people, places, or things) as well as categories, or facets. Every minute of every day, Evri’s systems are busy scouring the web, reading news content, blog posts and more so you don’t have to. Now, Evri’s system is also understanding the sentiments, or positive and negative expressions by and about entities.  Many types of applications can be built using the sentiment API in areas including, but not limited to: market intelligence, market research, sports and entertainment, brand management, product reviews and more. Specifically, Evri’s new sentiment API lets you:

  • Find the percentage of positive and negative expressions of sentiment made by an entity, or about an entity. For example, find out what percentage of things being written about the iPhone are positive and which percent are negative.
  • Discover who is criticizing and who is praising a particular person, place or thing. For example, see who is criticizing and praising Microsoft right now.
  • Read what praisers and critics are saying about an entity. For example, see what the GOP are saying about the Democrats.
  • Discover who or what your favorite entity is bashing and why. For example, see who Lance Armstrong is complaining about.
  • Discover who or what your favorite entity is praising and why. For example, see who the World Health Organization is commending and why.

Now, as an exploratory exercise, or tutorial, on how to use the API, I will walk through the calls needed to make a widget called the Vibology Meter. So, imagine the widget below is externally configured to be about the entity Barack Obama corresponding to the Evri URI: /person/barack-obama-0×16f69. Upon first load, you see something that looks like this:

picture-6

From the above screenshot, we can see that the percentage of positive sentiment and negative sentiment expressed by Barack Obama are displayed. We can also see the specific top entities being praised by Barack Obama in the left column, and the specific entities being criticized in the right column. For example, from the above screenshot, we see that Barack Obama is criticizing the GOP, Rush Limbaugh, the ACLU, Al Zawahiri, and Israel. In order to render the screenshot above, this sentiment summary information is returned by the following REST API resource call:

/v1/sentiment/summary?sentimentSource=/person/barack-obama-0×16f69&includeSummaryDetails=true&sort=date

Now, consider the use case outlined in the screenshot below, where the user clicks on [Anything] under the positive vibes sentiment. In order to get the results outlined to the right of the positive and negative sentiment columns, we execute a resource request like:

/v1/sentiment/about?sentimentSource=/person/barack-obama-0×16f69&sentimentType=positive&sort=date

From this request URI, we see that the sentimentSource references Barack Obama, meaning we are interested in vibes or sentiment expressed by Obama, as opposed to about him. Next we see the sentimentType is set to positive, meaning we are interested in positive sentiment expressions. Finally, we see sort=date meaning we are interested in the latest results.

Also from the screenshot below, we see the results of this resource request, namely, the specific snippet from the article, as well as a time stamp, the article title, and a link off to the source article. From the snippet, we see the sentence stating that “the president commended…” — the Evri system recognizes “the president” to be the source of the vibes, or sentiment, and commendation to be the prime justification for his positive sentiment expression.

picture-7

And finally, we consider the case illustrated below, where the “Receiving vibes” tab is selected, and the particular source of negative sentiment is chosen by the user to be Rush Limbaugh. In this case, by executing this resource request:

/v1/sentiment/about?entityURI=/person/barack-obama-0×16f69&sentimentType=negative&sentimentSource=/person/rush-limbaugh-0×1ebf5&sort=date

From this request URI, we see that the entityURI references Barack Obama, meaning the returned sentiment is about Barack Obama. We can also see that the sentimentType is set to negative, meaning returned sentiment expressions will be negative in nature. We also see that the sentimentSource references Rush Limbaugh. The URI referencing Limbaugh was obtained from the sentiment summary results of the request shown above in reference to the first screenshot.

picture-12

That pretty much sums up our walk through with the sentiment API. For complete documentation on this new API resource, see the Get sentiment information section of our REST API Specification. And finally, if you have any comments on how we can make this API better, questions on how to get things to work, or examples of bugs, please let us know on our developer forum.

Toolbar 1.4: Twitter, Recommendations, and More

August 3rd, 2009 by Deep Dhillon

We’ve got a few nifty features in the new Evri Toolbar version.  The Evri Toolbar is a convenient way to get great recommendations of other things to read, watch, and look at as you browse the web. In the 1.4 version, you can quickly navigate to more information about key topics on any web page by clicking on the On this page drop down in the menu bar. You are also only 1 click away from reading recommended articles to the web page you are on. Finally, if you are tired of copying and pasting URLs into URL shorteners and tweet platforms, you are now only a click away from sharing any web page with your friends via Twitter. If you haven’t tried the Evri toolbar before, or you want to try the new and improved version, download it at www.evri.com/toolbar/download and take it out for a spin.

picture-55

Check out Evri’s new JavaScript API

July 31st, 2009 by Deep Dhillon

picture-50 I’m excited to announce the launch of our new JavaScript API.  We’ve been using the JavaScript API internally for quite some time, for example, our widgets are built on this flexible and easy to use web API. It feels great to finally open it up to the Evri development community.

The Evri Javascript API utilizes JSONP to make web service requests to Evri’s RESTful API. The JSONP requests are augmented with a set of callbacks similar to those used in many cross-browser Javascript libraries for handling AJAX requests. This allows the Evri Javascript API to return first class objects back to the calling Javascript code without your needing to parse or traverse the JSON returned by Evri’s RESTful API.

Amongst other places, you can check the JavaScript API out in action on a fun new mashup called The Summarizer available at: www.thesummarizer.com. The Summarizer lets you see entities (people, places, or things) present in Twitter tweets about a topic, or by a given Tweeter. For example, you can find entities contained within the latest tweets by @cnn, or all entities in tweets about the Iran Elections. In addition, The Summarizer lets you pivot on one of these related entities, to find other tweets, images, videos, articles, etc. In short, its a great way to explore highly related information to something going on right now, in real time.

picture-51

So how does it use the Evri JavaScript API? According to Carlos, the rockin developer behind The Summarizer:

We instantiate a Session object which we use for all API calls. We get a TweetsList object by calling Tweet.findFromUsername and findForQuery as needed. The TweetsList has a list of the entities (i.e. people, places and things present in tweets) but it doesn’t have all requisite entity information such as entity descriptions. We retrieve an EntityList by using getEntityDetails. The EntityList has entity names and descriptions. The EntityList is then used to populate the left column. We bind each list item to its respective Entity object. When you click on an entity we make the following calls depending on what right side tab is selected:

  • entity.media.getImages is called to populate the Images tab
  • entity.media.getVideos is called to populate the Videos tab
  • entity.media.getArticles is called to populate the Articles tab
  • entity.getRelations to populate the ‘More About’ tab
  • entity.getTweetsAbout to populate the Tweets tab

And voila, that’s it. You can read all about the API in the JavaScript API documentation. And as always, ask any question, and please give us feedback on our developer forum.

Our new release – an abundance of features awaits you!

May 12th, 2009 by neil

We deployed our latest release, and it’s got a ton of good stuff. Coupled with yesterday’s announcement of our iPhone APP (thanks @TechFlash, for covering it!) we have a lot going on.

Here’s what’s in the release. A new home page, featuring our EvriFeed – a constantly updating stream of the latest about interesting topics in the EvriVerse. We use our unique approach to find interesting topics (grammatical subjects and objects in sentences being written on the web) and stream it to the home page. It’s always updating, so check back frequently to find something surprising.
hompage-mod

The EvriFeed shows you the Topic in question, a link to the relevant article on the web, and the most recent Evri connections for this topic.

Also on the new home page we are showing more of the breadth and depth of our content in the new Browse box on the right-hand side. You can easily find topics in our major domain — Politics, Entertainment, Business and Sports – or go directly to our full browse experience, which we talked about recently.

browse-box-hp

Continuing with the Home page, we have made your collections — and other interesting ones — much easier to find. If you haven’t yet created and account and built some collections, you really should. This is the best way to track your outside interests in real-time on the web.

collections-hp

Our Profile pages have new stuff as well. First, we now show you any collections (yours or others) that this topic is part of. This is a great way to discover the power of collections. If you are on the page for Star Trek (the new movie) for example, you can see it’s in three collections, including this one that I created.

We also now provide a way for you to find our how any topic is related to anything you want to type in. Even if it’s not in the Dig Deeper section of the page, you can type in any search term that you want to connect to the current profile and see how they are related. In this case, I was on the Microsoft profile, and wanted to see how they were currently related to Yahoo. I entered Yahoo in the ‘filter by keyword’ box and get the results right there. Cool!

keyword

Last thing I’ll mention for now is that we have added interactive stock charts to all of our public company profiles, thanks to our friends at Wikinvest. This is a highly-interactive in-page application that gives more than just a share price. Coupled with the other information we have on the page our company profiles just got a lot more valuable. (Though GM may not feel that about their page today!)

wikinvest

I’ll do follow-up posts exploring some of these features in more detail. Please let us know what you think, and what you would like to see us work on next and what we need to fix. You can always email me (neil[at]evri.com). And don’t forget that you can find us on Twitter: @evri is the company account, and you can get to me @neilr.

Welcome to the EvriVerse – Evri’s iPhone App

May 11th, 2009 by alexv
The Evri iPhone

The Evri iPhone

Here at Evri we love the iPhone. We know that its been a complete game-changer in the mobile-space. And, it seems like this whole iPhone App thing is catching on. So naturally, with so many iPhone fans in the company we thought to ourselves, “hey, we oughtta write an iPhone app!”

Starting as a “20% Time Project”, we’ve put together our first iPhone application that showcases our unique content browsing and discovery engine — and it’s all built with the same APIs that we expose to external developers. We call it “EvriVerse” and it’s available now in the iTunes App Store. We think it’s pretty nifty.

So what does it do? Well, one of the questions that we’re good at answering here at Evri is “what’s going on right now, out in the world?” With our ever-growing structured data store and deep semantic understanding of dynamic web content, we have an awful lot of connected, contextual, relevant information to play with. We do the work of finding important information about topics you care about, and let you browse them in a new way. Let’s take a look…

The initial EvriVerse screen

The initial EvriVerse screen

When you first start the app, you have two places you can go. You can either search for something specific by touching the magnifying glass, or see what’s making news by touching the rainbow. Let’s start with the hot list.

whatshot

Selecting an entry from the list will take you to our main view where you can see that topic and its top-five connections. The connections are to People, Products, Organizations or other things out in the world that the topic you are interested in is connected to.

sykes-graph

Here I’ve selected Wanda Sykes. Touching one of the related topics allows you navigate the what we call the “Entity Web”. Think of this as the ultimate “Six-Degrees of Kevin Bacon” tool.

Touching the main topic will give you a list of current articles, as well as current actions, top connections and profile information for that topic.

sykes

“Actions” and “Connections” give you a new way to browse web content about the topic you are interested in. With Actions you will see just that, a list of verbs your topic is engaged in – visiting, speaking, buying, etc.. Connections will show you what things your topic is connected to. Navigating any of these will take you to the list of articles that help you understand the connection. You can then follow this all the way through to the article — and you didn’t need to do any typing to get there!.

sykes-article

This is just the first release of this application and we have a host of features planned in the future. We hope you find it useful, informative and entertaining. Please do let us know what you think, and what features you think we should add. If you have questions or comments, please visit our iPhone Support Page and leave us a note.

Once again, you can download the app here . If you like it, please rate and review it. If you have a problem, please let us know.

Cheers!