Intelligent Applications for Web Priti Srinivas Sajja Associate Professor Department of Computer...

Intelligent Applications for Web

Priti Srinivas SajjaAssociate Professor

Department of Computer ScienceSardar Patel University

Visit priti sajja.info for detail

1Created By Priti Srinivas Sajja


Bio-inspired

Web

Web Intelligence

Searching and Retrieval

Knowledge Management on Web

Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence


• Name: Dr. Priti Srinivas Sajja• Communication:

• Email : [email protected]• Mobile : +91 9824926020• URL :http://pritisajja.info

• Academic qualifications : Ph. D in Computer Science

• Thesis title: Knowledge-Based Systems for Socio- • Economic Development (2000)• Subject area of specialization : Artificial Intelligence

• Publications : 108 in Books, Book Chapters, Journals and in Proceedings of International and National Conferences


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Introduction Natural Intelligence• Responds to situations flexibly.• Makes sense of ambiguous or erroneous messages.• Assigns relative importance to elements of a situation.• Finds similarities even though the situations might be

different.• Draws distinctions between situations even though there may

be many similarities between them.

Artificial Intelligence• According to Rich & Knight (1991) “AI is the study of how to make

computers do things, at which, at the moment, people are better”.

• A machine is regarded as intelligent if it exhibits human characteristics generated through natural intelligence.

• AI is the study of human thought processes and moving toward problem solving in a symbolic and non-algorithmic way.


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Introduction

“Artificial Intelligence(AI) is the study of how to make computers do things at which,

at the moment, people are better”

• Elaine Rich, Artificial Intelligence, McGraw Hill Publications, 1986


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Introduction

Extreme solution, either best or worst taking (infinite) time

time

Acceptable solution in acceptable time

Nature of AI solutions

where people are better

human thought process

characteristics we associate with intelligence

knowledge using symbols

heuristic methods

non-algorithmic

Constituents of artificial intelligence


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Testing IntelligenceAI Tests Turing test will fail to test for intelligence in two circumstances;

1. A machine may well be intelligent without being able to chat exactly like a human; and;

2. The test fails to capture the general properties of intelligence, such as the ability to solve difficult problems or come up with original insights. If a machine can solve a difficult problem that no person could solve, it would, in principle, fail the test.

Can you tell me what is

222222*67344?

Why Sir?

The Boss could not judge who was replying, thus the machine is as intelligent as the secretary.

The Turing test


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Can you find any test to check the given system is intelligent or not?AI Tests

If it talks like human

Translates, summarizes, and learns

Solves your problem

Reacts differently

Walks, perceives, tests,

smells, and feels like human

conceptually form a test and use it in different situationbefore accepting it.

Makes and understands joke


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Rich & Knight (1991) classified and described the different areas that Artificial Intelligence techniques have been applied to as follows:

Applications

Mundane Tasks• Perception - vision and

speech• Natural language

understanding, generation, and translation

• Commonsense reasoning• Robot control Formal Tasks

• Games - chess, backgammon, checkers, etc.

• Mathematics- geometry, logic, integral calculus, theorem proving, etc.

Expert Tasks• Engineering - design, fault

finding, manufacturing planning, etc.

• Scientific analysis • Medical diagnosis • Financial analysis


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Basic transactions by operational staff using data processing

Middle management uses reports/info. generated though analysis and acts accordingly

Higher management generates knowledge by synthesizing information

Strategy makers apply morals, principles, and experience to generate policies

Wisdom (experience)

Knowledge (synthesis)

Information (analysis)

Data (processing of raw observations )

Volume Sophistication and complexity

TPS

DSS, MIS

KBS

WBS

IS

Data pyramid

Data Pyramid


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



According to the classifications by Tuthhill & Levy (1991), five main types of KBS exists: Expert systems Linked Systems CASE based Systems Intelligent Tutoring Systems Intelligent User Interface for Database

Knowledge base

Inference engine

User interface

Explanation and

reasoning

Self-learning

General structure of KBS

Knowledge Based systems


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Knowledge Based systems

Experience

Satellite Broadcasting (Internet, TV,

and Radio)Printed Media

Experts

Sources of knowledge

Types of Knowledge • Tacit knowledge

• Explicit knowledge

• Commonsense knowledge

• Informed commonsense knowledge

• Heuristic knowledge

• Domain knowledge

• Meta knowledge


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Intelligence, explanation and reasoning Partial self learning, uncertainty handling Documentation of knowledge Proactive problem solving Cost effectiveness

Nature of knowledge Large volume of knowledge

Knowledge acquisition techniques Little support to engineer AI based systems

Shelf life of knowledge and system Development Effort


Pros and Cons


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence Bio-Inspired Computing

New approaches to AI Taking inspiration form nature and biological systems Includes models such as

Artificial Neural Network (ANN), Genetic Algorithm(GA), Swarm Intelligence(SI), etc.

Nature has virtues of self learning, evolution, emergence and immunity

The objective of bio-inspired models and techniques to take inspiration from Mother Nature and solve problems in more effective and intelligent way


Bio-inspired


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence Artificial Neural Network (ANN)

An artificial neural network (ANN) is connectionist model of programming using computers.

An ANN attempts to give computers humanlike abilities by mimicking the human brain’s functionality.

The human brain consists of a network of more than a hundred billions interconnected neurons working in a parallel fashion.


A biological neuron An artificial neuron

X2

X1

XiWi

W1

W2

… ….

y

Xn

Wn

Bio-inspired


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


A Perceptron

Multilayer Neural Network


W12X1

X2

X3

.

.

.

.

Xn

O0

O1

….Om

.

.

.

.

.

.

.

.

.

.

.

.

W1h

Input layer Hidden layers

Output layer

Bio-inspired


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Genetic Algorithms (GA)• It mimics Nature’s evolutionary approach • The algorithm is based on the process of natural selection—

Charles Darwin’s “survival of the fittest.” • GAs can be used in problem solving, function optimizing,

machine learning, and in innovative systems.


Genetic cycle

Modify withoperations

Start with initial population by randomly selected Individuals

Evaluate fitness of newindividuals

Update population with better individuals and repeat

Initial population

Selection MutationCrossover

Evaluating new individuals through fitness function

Modify the population

Bio-inspired


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence Swarm Intelligence

Inspired by the collective behavior of social insect colonies and other animal societies

Ant colony, fish school, bird flocking and honey comb are the examples


Bio-inspired


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Some more examples ….


Natural

Inspired

Bio-inspired


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web

• Internet can be defined as network of networks.

• The World Wide Web (WWW or Web) is a large scale distributed hypermedia system on the internet platform.

• The WWW is based on the HTTP-protocol for data transfer, HTML markup for content display on top of the Internet infrastructure that uses different protocols and content description schemes.

• According to Hans-Georg Stork (2002), the Web is experiencing two issues:• Not able for “semantic” access and use problem • Depends on the universality of physical access via

high-bandwidth local loops and broadband wireless channels.


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web

• Semantic Web is an extension of the current Web in which information is given well defined meaning by

associating metadata. (Berners-Lee, Hendler, & Lassila, 2001).

• Basic objective of a semantic web is “Making content machine-understandable”.

• The semantic web aims to allow Web entities (software agents, users, and programs) for interoperating, dynamically discovering and using resources, extracting knowledge, and solving complex problems.


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web Intelligence

Challenges and limitations of the current Web Lack of knowledge-based searches Lack of effective techniques to access the Web in depth Lack of mechanisms to deal with dynamic requirements of users Lack of automatically constructed directories Lack of multi-dimensional analysis and data mining support

By employing the AI techniques for web functions, it is possible to partly impart intelligence in web-based business.

The Web Intelligence (WI) is considered as employment of AI techniques for the Web.

Web Technology

• Platform of Internet • Protocols and standards• Browser • Search engine • Semantic Web• Other software

Web Technology

• Platform of Internet • Protocols and standards• Browser • Search engine • Semantic Web• Other software

AI Techniques

• Knowledge representation• Knowledge management• Expert system• Heuristic functions• New AI methods

AI Techniques

• Knowledge representation• Knowledge management• Expert system• Heuristic functions• New AI methods

Web Intelligence


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web Intelligence

Web Information Retrieval

· Information retrieval and filtering

· Performance measures

· NLP

Web Mining· Web log mining· Web structure

mining· Web content

mining· Sensor Web

mining

Web Agents

· Intelligent agents

· Multi agent systems

· Pattern discovery

Human Computer Interaction/NLP

· Personalized interface

· Multi lingual interfaces

· Usability

Semantic Web· Search Engine· Ontology

management· Meta ontology· Interoperability· Inference

Social Intelligence

· Popular tools and techniques

· Social Network Analysis

Search Engine Techniques

· Customized searches

· Meta search engine

· Search engine optimization

Web Knowledge Management

· Knowledge management architecture for Web

· Security


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



To implement a simple Web crawler following steps can be performed.

1. Start interaction with user and seek keywords and URL to start with

2. Add the URL to list to search for

3. Repeat while list is not empty

3.1 Consider the first URL and mark with appropriate flag

3.2 If the protocol of the selected URL is not HTTP then break

3.3 Follow the robot.txt file (instructions), if any

3.4 Open the URL

3.5 If the URL is not an HTML file then break else add the file into list of files found

3.6 Extract links by traversing the file

3.7 Repeat this procedure for every link within the file

Searching


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Searching

Web crawler process

SpiderSpider

ListsLists IndexIndex ProcessingProcessing StorageStorage

Web

Focused Crawler Searching relevant pages

Simple Crawler Searching all pages

Scope of focused crawler


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Information Retrieval (IR) is a science of • information finding, • acquiring, • storing and • utilizing the information for problem solving.

The formal steps are given as follows:• Indexing• Query formulation• Matching query

representation• Relevance feedback and • interactive retrieval

Information Retrieval


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement




Models of Information Retrieval

1. Boolean Model - Boolean operators like AND, OR and NOT are applied to retrieve content.

2. Vector space model - represents the documents and queries as vectors (defined by keywords) in a space having more than one dimensions.

3. Probabilistic model - considers the retrieved content according to some rank based on some probability.

4. Latent semantic model - considers associations among terms and documents to retrieve required content.


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



NLP for IR

Generic NLP architecture

UserUser Web

Interpreted Query

Grammar Lexicon Token Templates

Terminology

Tokenizer Recognizer ParserPreprocessing Interpreter

Natural Query

Search Result

Web Search Request

Filtered Result

Dialog ProcessorDialog Processor Search MechanismSearch Mechanism

Search Result

Dialog Analyzer and

Generator

Dialog Analyzer and

Generator

Context Model and Domain Terminology

Context Model and Domain Terminology

User Profile and Local informationUser Profile and

Local information


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Research Trend in IR

Research Trend in IR

• Heuristic filtering

• Semantic Information

• Multimedia Data

• Opinion Retrieval

• Information retrieval and translation

• Fuzzy Boolean model of information retrieval


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Knowledge Management

DiscoverDiscover

UseUse

DocumentDocument

ShareShare

Knowledge Base

Knowledge Base

Knowledge Sources

Knowledge Sources

Knowledge Engineer

Knowledge Engineer

Organizational Requirements Organizational Requirements

Standards, Protocols, and

Services

Standards, Protocols, and

Services

Typical Knowledge Management Process

The Web follows document-centric approach, which lacks efficient representation and access of

the content on Web.


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement




Knowledge Management Architecture on the Web

Local Documents

Domain Ontology

Knowledge repository

Editor

Crawler Inference mechanism

Metadata

Knowledge Presentation

User Profiles

Knowledge Processing

Knowledge Discovery

Experts

Administrator

Web

Users

Service Standards


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement




• Autonomous agents for knowledge discovery

• Protocols for knowledge share and use

• Ontology editors

• K-Commerce

• Knowledge management models

• Virtual world

• Wisdom Web



Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web Mining

Data Mining The data mining techniques are dedicated techniques that

extract patterns and useful information from the

existing known sources of data.

Text Mining Text mining techniques are used to find, organize and discover

information from the textual resources.

Web Mining Web mining techniques are used to find, organize and discover

information from the huge unstructured platform such as Web.


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web Mining

Challenges of Web Mining Structure highly unstructured Size tremendous Nature dynamic Accessibility global by anybody Redundant similar information in

many formats Noise virus, malware and adware


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web Mining Web Mining

Web Content Mining

Web Log Mining

Web Structure

Mining

Data Type /Sources

Purpose

Any data Textual data Web data

Finding relevant

data

Finding pattern and knowledge

Data Retrieval


Web Retrieval

Data Mining

Text Mining

Web Mining

Web Mining and Other Related Activities


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence Web Content Mining

It attempts to mine content of the Web to discover useful patterns through hyperlinks.

The content may be text, images, audio, video, and structured data like tables and graphs.

The web content mining goes beyond keyword extraction and requires advanced techniques such as NLP and AI.

Web content mining strategies are of two groups one that directly mine the content of documents and second that improves on the content search of other tools

like search engines.


Web Mining


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Classification as Web Content Mining Techniques Classification : deals with classification of the content into various

groups as accurate as possible. The training sets and test (validation) sets are provided to the classification algorithm to build and to test the classification model respectively.

Typical classification techniques include:

Decision tree based methods; Rule base classification; Supervised learning through artificial neural network; Evolutionary techniques; and Support vector machines;


Web Mining


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Clustering as Web Content Mining Techniques Clustering : deals with finding groups of similar objects based on

the content characteristics itself in unsupervised approach.


Web Mining

Partition and hierarchical clustering

Initial Points

Partition Clustering1, 2 and 3 are independent

clusters.

1

2

3

Hierarchical Clustering

Here cluster 3 is subset of 2; and 2

is subset of 1.

1

2

3


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Classification and Clustering as Web Content Mining Techniques Association Mining : deals with discovering interesting

relations between variables in large databases. This technique find rules that will predict the occurrence of an entity based on general pattern exists in the given data sets.

Consider following example.


Web Mining

Transaction ID Bread Cheese Sauce

1 Yes Yes Yes

2 No No Yes

3 No Yes Yes


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Classification as Web Content Mining Techniques Opinion Mining: deals with extraction of opinion of users

learn attitude of the content, person or product. Opinion mining plays an important role in mining applications for customer relationship management, consumer attitude detection, brand and product positioning, product reviews, and market research.

Feature based opinion mining mines the Web content by given features of a specified product/entity.

Once the opinions are collected, they are further grouped and analyzed.


Web Mining


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence Some other Web Content Mining Techniques

Structured data extraction: Structured data extraction deals with

extraction of important information about product, services

and data records that are available in structured form on host pages.

Unstructured content extraction: It deals with extraction of

content that is not available in structured form. Web information integration: It extracts content form multiple

site, checks for redundancy, and integrates information. Vice versa, the content mining can be used for web site classification/clustering also.

Detecting noise: The malware, adware and virus from multiple site can be identified and blocked.

Opinion mining: The customer surveys, opinion, sentiments and product review information etc can be extracted here.


Web Mining


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence Web Usage Mining

The Web usage mining provides the collection of information accessed so far to its users.

Web usage mining highlights the behavior of users on the Web and understands access patterns and trends.

The web usage mining deals with web log and accumulated data on web servers in order to understand the user behavior and the web structure.

There are two main purposes for web usage mining. The first one is to track general access pattern and second is customized usage tracking.


Web Mining


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web MiningActivities for web usage mining

Log Data

Registration Data

Other Information

Analysis and Use

Integration & Analysis of Patterns Discovery

Cleaning Noise

Cleaning Malware

Pattern discovery

and Analysis

Retrieval

Cleaning

IdentificationIntegration

Use


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence Web Structure Mining

The Web behaves like a hypertext document information system. The Web objects such as pages and sites are generally exist between the numbers of links.

Web structure mining focuses with structure of such hyperlinks on the Web.

There are two basic techniques to analyze the network of links on the Web. These methods are

(i) Hyperlinked Induced Topic Search (HITS) concept and

(ii) Page Rank method. The Web may be represented as a huge directed graph

structure.


Web Mining


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence Sensor Web Mining

Data are collected from different sensors placed at remote places.

Provides opportunity of efficient geo-referencing in remote fashion.


Web MiningSolar panelSolar panel

MicrocontrollerMicrocontroller RadioRadio

MemoryMemory

Sensor SuitSensor Suit

Architecture of a Pod

Sensor Web consists number of sensor platforms called pods.

Each pod senses some dynamic environmental data in real time fashion.

Radio is used to connect the pod with its local neighborhood.

Applications are weather forecasting, costal area monitoring, communication and education, and eco-system information and management.


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Web Mining


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement

Artificial Intelligence AI for Web Mining

Mining pro-active agents

ANN for finding/ analyzing patterns

Fuzzy partitions and clustering

Evolution of patterns from Web

Heuristic based filtering functions for mining

Sentiment mining using NLP on social network platform


Web Mining


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Types of Agents Collaborative Agent

Interface Agent Mobile Agent

Information Agent Intelligent Agent Hybrid Agent


Agent Agent

Sensors to acquire environmental information and user’s

requirement

Action interface

AutonomyAutonomy

CooperationCooperation

LearningLearning

MobilityMobility

ProactivityProactivity

An Agent

Agent Based Web


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Example

Web/ Semantic

WebInternet Internet

BrowsersBrowsers

Filtering Agent Filtering Agent Interface

Agent Interface

Agent

URL Management

URL Management

Ontology AgentOntology Agent

Search Engine Agent

Search Engine Agent

Social Networking Agent

Social Networking Agent

Agent based web

Core Services

Customized Services

Ontology ToolsOntology Tools

Query Agent Query Agent

Document Management

Agent

Document Management

Agent

Protocols and Standards

Protocols and Standards

Agent Based Web


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



WebWeb

ClientClient ClientClientClientClient

Web BowserWeb Bowser

Knowledge BaseOntology

Local Databases Base Domain Databases Base

Query Manager Search Engine Visualization

Figure 11.10 Information retrieval agentAgent Based Web

Example


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement


Agent for semantic analysis Verification and validation (V&V) agent Finding suitable web services agent Crawler agent Explanation and reasoning Agent Natural language interface agent Communication agent Network traffic management agent Mobile agent for personalized content

representation 51

Created By Priti Srinivas Sajja

Agent Based Web

Other

Examples


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



Major References Knowledge-based systems, Akerkar RA and Priti Srinivas Sajja, Jones & Bartlett

Publishers, Sudbury, MA, USA (2009) Intelligent technologies for web applications”, Priti Srinivas Sajja, Rajendra

Akerkar; CRC Press (Taylor & Francis Group), Boka Raton, FL, USA (2012)

Other References llustrationsOf.com coders-view.blogspot.com info.ideal.com http://businessintelligencetalk.blogspot.in www.gadgetcage.com Engadget.com scenicreflections.com lih.univ-lehavre.fr business2press.com globalswarminghoneybees.blogspot.com pritisajja.info

http://www.jbpub.com/Catalog/9780763776473/Overview/


Bio-inspired

Web

Web Intelligence



Web Mining

Agent Based Web

Acknowledgement



To the participants and authority of the AICTE sponsored Staff Development Programme on Data

Mining, 16-28 April, 2012at the

L. J. Institute of Engineering & Technology, Ahmedabad.

Intelligent Applications for Web Priti Srinivas Sajja Associate Professor Department of Computer...

Documents

Transcript of Intelligent Applications for Web Priti Srinivas Sajja Associate Professor Department of Computer...