I’m Tony Sarris and during the first half of this session I’d like to provide three things: a brief

1

retrospective on early work on semantic, agent-based technologies; a look at the current

state, and a look ahead at what still needs to be done. I’d like to raise some issues around

the impacts – good and bad – that such technologies may have on us from a sociological

standpoint. In the second half of this session, my colleague, Peter Sweeney, will discuss

and demonstrate a technology that fills some of the current gaps in fusing information

together in meaningful ways for use by both humans and intelligent agents.

Some years ago, back in the 90s, the notion of intelligent distributed software agents

2

coordinating on behalf of their human charges was all the rage. In fact, I had a tag line in

my e-mail signature block that read “Have Your [Software] Agent Message My [Software]

Agent and We’ll Do [Virtual] Lunch.” As part of the R&D community in the field of artificial

intelligence, specifically working on ontology, we envisioned a time in the not-too-distant

future when automated agents representing our needs and interests would shop for us,

book trips and appointments, do research, manage our finances and help us run that

complex, dynamic enterprise known as our lives.

We envisioned that there would be a rich set of such agents available as ‘services’. They’d

be context aware and have deep knowledge about the domain they were operating in.

They’d be easy and inexpensive to access, would be available anytime, anywhere and

would provide instantaneous results. There would be different kinds of agents employed --

some handling fairly simple and singular tasks and others handling more complex tasks --

and determining the appropriate agents would be based on the criteria and goals of user

requests. More general purpose, ‘horizontal’ type agents would contract with more vertical,

specialized agents as needed to accomplish the end user goal. That sort of integration

among distributed agents would take place automatically, and hopefully largely be

transparent to end users. Context would be preserved and adapted – again automatically –

across different domains and sets of associated tasks required to accomplish some goal.

And lastly, those agents would be woven seamlessly into our human-centric social world. In

other words, the agents would fit into our world rather than us having to fit into theirs.

That was a pretty good vision, don’t you think? But the problem was: at the time the vision

3

was missing some pretty important implementation details – so it came out looking more

like this.

That’s because unfortunately :

•We didn’t have truly mobile computing devices and ubiquitous high-speed networks

Cell phones were still clunky and service was expensive, laptops were more like desktops

and web access was still anchored to a dial-up phone line.

•Few large repositories of domain-specific, searchable content

Things were being connected into the internet – but many were in legacy formats making

them difficult to access. And search was nothing like it is today.

•There were effectively few or no reference ontologies or domain ontologies with which to

make sense of that content

•There was little or no ability to relate data/events to other data/events, apart from text

string matching or other syntactical operations and no ability to contextually situate

data/events, and to move from one context to another

•And because we didn’t have many of the fundamental building blocks, we clearly weren’t

in a position to really build intelligent agents. AI research was perhaps at its zenith, but AI

technologies, including agent-oriented applications, were almost non-existent.

Now times have changed quite a lot since then. I’ve been impressed sitting in the sessions

4

this week to see just how far we’ve come in a number of areas. So you can reasonably ask

the question: are we there yet? Well, we’re certainly much closer than we were, but we’re

still not quite there. While some gaps have been filled, a lot of gaps still remain. We’ve got

agent-amenable mobile devices and widely-available high-speed networks. But the

networks still aren’t ubiquitous and prices for the level of access required to support these

sorts of technologies are still relatively expensive. We do have large bodies of content –

some might say the problem right now is too much data, but not enough information – and

while search has improved greatly, getting the right information is still difficult. Some people

argue it’s actually been getting worse lately. We’ve got decent metadata tagging and some

conceptual models, but we’re still in the infancy stage of ontologies. Web applications still

have very limited context and situational awareness. There are some pretty good single

purpose rule-based or agent-based applications, but we certainly are not at the point where

we have collaborative, distributed agents working seamlessly together to accomplish some

goal. And for all of the technology we do have, we have very little understanding of the

social implications of its use.

I guess you can distill all that down to the fact that there has been considerable progress –

only cellular service is still expensive and AI still doesn’t exist.

But at least the fundamental pre-requisites exist to enable the sorts of things we envisioned being performed by intelligent agents, plus a number of other more socially-centered things most of us

5

performed by intelligent agents, plus a number of other more socially-centered things most of us never considered at the time.

So where could we, and where should we, go with agent-based technologies?

Let’s start by considering a scenario that I’ve heard in various forms over the past several months. It is centered on the intersection of smart mobile devices and social networks. Let’s say you’re visiting Dublin, Ireland on a business trip. Maybe you had a business dinner arranged for the first night, but the next evening you’re free, you don’t really have the time or energy to go on some grandiose exploratory journey through the city, or maybe you’ve already been there a couple times before and you’ve already seen the highlights. Since you’re kind of tired, you’re tempted to just grab a quick bite, go back to your hotel room, listen to some favorites from your playlist on your phone or some mobile music device you’ve brought along or surf the web from your smart phone or portable PC -- whether to do some more work, or for news or entertainment. As a former road warrior myself I can identify with this scenario, as I’m sure many of you can. But then it occurs to you, “Wait a minute, I’m in Dublin! Isn’t there some place not too far from the hotel here in the city centre where I could get some traditional Irish pub food, a locally-brewed stout, and maybe even be around some other people?” because hey, it can get lonely on the road and you could really use some human companionship. No, not *that* kind of companionship. We don’t need more technology dedicated to that purpose, because there seems to be plenty of that already on the internet.

Here’s where you can use technology to first find a suitable pub restaurant near your hotel

6

in the city centre. You want one that has traditional Irish food (at least according to

RUaFoodie.com’s definition of traditional Irish food and a quick historical check you did on

AllAboutIrelandsPast.org). So you discover that Irish stew is one example of a traditional

Irish food. But in addition to traditional Irish food you want the place to serve a locally-

brewed stout that is well-regarded, so you check your favorite beer-related social

networking site (brewophile.com) to find a stout maybe rated at least 4 stars out of 5 by

people who have established very good on-line pedigrees for that sort of thing. So you end

up with a hit on Blarney’s, serving traditional Irish stew and Dublin’s best locally-brewed

stout, and located in the centre-city. That’s all pretty basic technology these days.

Next comes something that’s really starting to be talked about – whether it is totally realized

7

yet or not. You are here! You want to search for someone who, per their mobile device,

has a geospatial position that is within 1 K of your hotel, and who per one or more linked

social web site(s) or public calendars indicates they are available to meet for dinner within

half an hour of your desired dinner time. Now let’s also add in the attribute of wanting, or at

least being willing and able, to discuss James Joyce’s Ulysses (based on their profile on a

site such as LibraryThing.com). And you’d prefer it be somebody local who can give you an

Irish perspective on Joyce and his work.

You get a hit. He’s registered on LibraryThing and is noted as a fan of James Joyce,

8

including Ulysses. But wait! From other data it looks like he’s born, raised and currently

lives in New York, and apparently just happens to be in Dublin at the same time you are. So

much for the local perspective! Alternatively, there is a local guy who meets all the criteria,

but he has a reputation for posting tirades in various web forums when drunk – which

apparently is almost nightly. You don’t mind a little slurred speech here and there, and a

little passion in the discussion is fine, but you’d prefer not to have fisticuffs with an angry

drunk. There’s also an Irish woman – not a Dubliner, but Irish nonetheless – someone in

the education field who is in Dublin for a conference at Trinity College. From Library Thing,

she appears to have a keen interest in James Joyce, including Ulysses – but on another

site, a dating match site, she notes that she’s a non-smoking, non-drinking divorcee. Still,

maybe she won’t mind if you have a drink, so you decide to send her an invitation to meet

for dinner, making clear for both parties that the nature of the dinner is strictly platonic. As it

turns out when you meet her, she is a professor of European literature and is somewhat of

an expert on Joyce and Ulysses. She expected that you had wanted to discuss Joyce’s

work from a more critical, literary standpoint. But you muddle through dinner okay and

move on.

How much searching would it really take to reveal all this? And wouldn’t it be great to have

an intelligent agent that quickly analyses multiple data sources about these people in some

meaningful context? Or do you simply chat or text briefly first, asking the sorts of questions

that might better determine the extent of overlap of your shared interest in the proposed

get-together, as described?

None of this would even have been an issue a few years ago, because using technology

9

this way and at this level wouldn’t even have been possible. To find your desired restaurant

or pub, one that’s likely to be populated with the type of person you might have the desired

conversation with, you would probably have asked for a recommendation from the hotel

concierge or someone on the street who has a perceptibly Irish accent and seems like a

local. They’d perhaps recommend a perfectly nice seafood restaurant just down the block.

Or perhaps you would have simply hung out for awhile in the hotel bar. Maybe in the bar

you would have seen the woman who sat next to you on the plane, with whom you had

great conversation for several hours on the way across the Atlantic. And you would have

ended up going to dinner with her at some little Indian place in an off-beat neighborhood

that she loves to eat at when he’s in Dublin, and you would have continued the enjoyable

conversation from the plane – comparing the trials and tribulations of your teens with hers

and discussing some more how growing up in the turbulent 60s and 70s continues to

influence your lives. Or maybe you would have sat at the bar next to some guy you

normally wouldn’t necessarily strike up a conversation with and yet he starts a conversation

with you. It turns out you have a shared love for trout fishing and you spend a wonderful

evening exchanging fish stories – including over dinner at a little out of way local place

you’d never find on the web, a place with great fresh local ruby red trout. You wouldn’t

achieve your goal of discussing Joyce with an Irishman – something technology might help

you to achieve – but in each of the scenarios above, you’d still have an enjoyable time, and

one you likely couldn’t have foreseen.

For me, that all begs the question: Will we lose the beauty of randomness when we use

10

technology to overtly – and perhaps overly – target an explicit goal? Or could intelligent

agents actually expand opportunities for a broader range of interactions? For example, let’s

say you had exchanged contact information with the woman on the plane and then noted in

your personal preferences that given an opportunity to talk with her again (say if you were

in the same city over the same time period, or booking seats on a plane again sometime),

you would enjoy doing so (on a 8.7 out of 10.0 scale), and an intelligent agent made use of

that information. Applications like TripIt offer very rudimentary starts down that path. Or

what if an intelligent agent presented a varied menu of the evening’s options, rank-ordered

based on some set of criteria including a low-key personality, likes beer (but no reputation

for drunkenness), is interested in literature (but as a lay person), and maybe even check to

see if they have good financial credentials with on-line payment clearinghouses (so you’re

less likely to get stuck footing the entire dinner bill), etc.?

One thing to naturally consider at this point is under what conditions people are willing to strike up a conversation with a stranger, or positively react to a conversation that a stranger attempts to start with them?

11

conversation with a stranger, or positively react to a conversation that a stranger attempts to start with them? A lot has to do with basic personality, whether you are an introvert or an extrovert. Some of it has to do with attractiveness, not just physical looks per se, but a whole set of body language-based signals, coupled perhaps with verbal queues, that each person sends and that get interpreted by other people. People quickly assess other people whether they actually meet them or are simply in some shared physical space with them. Are they a threat, friendly, weird, interesting, attractive/sexy, or whatever. Numerous studies indicate that the time it takes a person to form a basic assessment of someone else can be as rapid as a few seconds. Even with pictures or live video feeds on your mobile device of choice, it’ll be awhile before we replicate the experience of really meeting and interacting with someone face-to-face, or even being in the same general physical space with them. Similar sorts of things may take place in a virtual space, but the interaction is much different in its nature and is based on a much more limited set of inputs, perhaps even just words on a tiny screen. From my perspective, virtual worlds like Second Life are still a poor substitute for the real-world. Your mileage, of course, may vary – but there’s a complex equation behind social interactions that virtual agents probably won’t master anytime soon.

Let’s go back to our previous scenario. How do you even begin a face-to-face conversation with someone if all you know is that the other person has a desire to discuss James Joyce’s Ulysses? Is it more difficult than having a text-based interaction with that same person via some electronic device? How does it change things when you in fact know a lot about the person (from data collected), but yet you’ve still never actually met them before? It’s that whole virtual versus physical world thing coming into play, as I talked about a minute ago.

Even if you know a lot of data about someone, as assembled from the web by you or your intelligent agent, when you first meet them in-person, it is a unique, and perhaps uniquely human, experience to integrate your perceptual or sensory experience of the person face-to-face with your factual knowledge of the person. From the pre-web world, maybe the best correlate to this sort of social experience is having reviewed a prospective employee’s detailed resume, and then meeting them for a face-to-face interview. Only in that context, the data set is typically quite narrowly constrained and the expectation of the nature of the interaction is also fairly well understood from common social conventions and the previous experiences of the individuals. Or perhaps it is more similar to a ‘blind-date’? I’m curious what sorts of social conventions, if any, will arise going forward based on these new forms of technology-aided social interaction.

What about safety issues? That’s a reasonable concern, particularly if you’re traveling

12

alone at night, or for that matter for anyone who wouldn’t normally meet a ‘stranger’ for a

drink or dinner, but who might be willing to now based on a social network type of

connection. How and when does safety get factored into this mix, or does it? This is clearly

an already widely-discussed issue in the area of virtual, internet-based interaction moving

to the realm of real-world, physical interaction. Internet predators are an almost constant

media topic. If the number of meet-ups increases exponentially in volume and speed

because of new technologies, will safety, fraud and other forms of exploitation become

even bigger issues?

Can an intelligent agent actually help lower risk through producing and analyzing

composite, on-line profiles of potential meet-up partners on the fly, including factoring in

weightings based on navigating social networks (e.g., so-called ‘friend of a friend’ (FOAF)

relationships)? Scenarios that involve real-time ride-sharing may be good for global

warming, but could be hazardous for personal safety if they aren’t done safely. Or what

about a localized Twitter-like message sent at 11:00 PM: “Is there anyone near Mission and

5th right now, walking down towards 10th who would like to walk with me?” Is that a boon

to safety or a risky proposition? How much triage by you or your intelligent agent, and

based on what kind of data, would it take before you’d feel safe being accompanied by a

complete stranger for a walk down the street at night? Admittedly, that’s an extreme

example, compared to suffering through a mistake in a hotel room booking or bad dinner

company, but it’s a good illustration of the opportunities and challenges for intelligent

agents and real-world human interaction.

We’re at the point now where we have the vision – in fact it’s fundamentally the same one from the 90s I think – but what we need now are the visionaries to bring it to realization. There is still work to

13

90s I think – but what we need now are the visionaries to bring it to realization. There is still work to be done. The deeper the processing and analysis that software agents are expected to perform on data on our behalf, the more the need for richer domain ontologies coupled with horizontal ‘commonsense’ ontologies. Simple conjunctions or intersections of metadata tags aren’t the same thing as meaningful interpretation of data in situational contexts that span multiple domains. Context is critical. What’s needed is semantic fusion, similar to what’s done in the field of sensor fusion. As the linkages between the virtual world and the real-world increase and begin to meld, the relevancy and value of data for the types of activities covered by personal intelligent agents will be established by both the speed and accuracy of that data – just like internet searches today – but with a much more demanding component of situational context and data integration, and perhaps with more profound consequences if we don’t get it right.

These new forms of social interaction, as enabled by the intersection of mobile devices, the web, social networks, and the use of intelligent agent technology, could prove very beneficial to us as humans living in a world that is at once more global, but at the same time more fractured. We can perhaps transition technology from a set of tools that today enables a lot of virtual interaction to a set of tools that increases and enhances real-world interaction of a deeper social nature. Technology is making the opportunities for real-world interactions more abundant, but those opportunities are constrained (sometimes reasonably so) by limitations of technology and of our social conventions. For these interactions to be useful, meaningful, convenient and safe, there needs to be advances in the enabling technologies, as well as exploration of new social structures and patterns. Together that will help ensure that the semantic web – Web 3.0 – truly enriches our everyday lives. Let’s start by having your agent message my agent, and we’ll do lunch. But before lunch, I’d like to introduce my colleague, Peter Sweeney, from Primal Fusion.