1. Ontology Engineering:
O t l E i i
Design and Practices
g
2009년 03월 21일
한 성 국
의미기술 연구소 / 원광대학교
2009-03-20 skhan@wku.ac.kr 1
2. Agenda
Review of Ontology
Ontology
gy Development Methods
p
Ontology Building
Ontology
gy Building Summary
g y
2009-03-20 skhan@wku.ac.kr page 2
4. A Plethora of 'Ontology-Like Things’
Glossaries / Controlled Vocabularies Data and Document Metamodels
ad hoc XML Restricted Logics
Hierarchies (OWL, F-logic)
structured Schema
(Yahoo!) Glossaries formal
Taxonomies
Terms XML DTDs
Thesauri
‘ordinary’ Principled, Data Models
Glossaries informal (UML, STEP)
taxonomies
Data Frames General
Dictionaries Logic
DB (OKBC)
(EDI) Schema
Informal Taxonomies and Thesauri Formal Knowledge Bases & Inference
Many Different Ways of Expressing Meaning
M Diff tW fE i M i
2009-03-20 skhan@wku.ac.kr page 4
6. Summary: Comparing Ontology‐Like Things
Ctld. Taxonomy Thesaurus Ontology Data Models Object Models
Vocab
Defined Controlled Controlled vocab. specification of a Specification of Specification of
Definition terms, vocab. in a in a network. conceptualization DB structure a software
controlled hierarchy. application
domain
Free text,
text Strict: tree Broader/narrower Logics,
Logics e g ER
e.g. Hierarchy of
Notation Definition (maybe taxonomy) Taxonomy as diagrams classes, rel's
structure Or: multi- backbone + atts. Entities & attributes &
varies. parent Gnl. association; & relations. Relations methods
Nrl lang Nrl lang Nrl lang def's + Logics w/ fml
fml. Precise,
Precise not Increasingly
def's def's + meaning of links. semantics. logic-based. formal.
meaning of
link B/N: various mng's Isa hierarchy; Focus on data, Isa hierarchy,
Meaning Gnl Assoc n: no
Assoc'n: Dom/Range not meaning Aggregation /
Dictionary; Strictness & specific meaning constraints; (e.g. toss rel'n Composition,
common Precision cardinality. names). Dom/Range
usage varies. constraints;
Isa, partOf, Nrl. language Data dictionary cardinality.
similarTo … comments in the separate.
ontology.
Human HC + HC + Structure Union of all the HC + Structure HC + Structure
Purpose communi- Structure digital libraries;
g ; others & more. (
(and validate)
) software
cation info. base; indexing, databases. systems.
(HC) browsing browsing & search
2009-03-20 skhan@wku.ac.kr page 6
7. Ontology
an formal, explicit specification of a shared conceptualization of a domain.
개념화 방법
개념화 수준 a shared conceptualization of a domain.
형식화 수준 formal
자연언어 > 시소러스 >…형식논리
표현 언어
현
XML > Metadata >…온톨로지 언어
explicit specification
2009-03-20 skhan@wku.ac.kr page 7
8. Ontology in a nutshell
Domain model: a formal, explicit specification of a shared conceptual model
Shared formal conceptualizations of particular domains.
Provide a common interpretation of topics that can be communicated between people and
applications.
A formal vocabulary for information exchange.
exchange
Typically contain hierarchies of concepts and their relations within the domains and
describe each concept’s crucial properties through an attribute-value mechanism.
Also allow definition of axioms and constraints on particular concepts and properties.
p p p p
Ontological Commitment: General agreement between Ontologies
Ontologies are social contracts. 도메인 핵심 개념어
• Agreed, explicit semantics Concept
p
• Understandable to outsiders
• (Often) derived in a community process 구성 데이터 집합
Instance Relation
핵심 개념어간
의미적 관계
개념 관계값
Function Axiom
도메인 지식 규칙
2009-03-20 skhan@wonkwang.ac.kr page 8
11. An Ontology Building Life‐cycle
Investigation I NVESTI GATI ON
- Identify problem and opportunity
- Identify potential solutions
- Feasibility study
CONSTRUCTI ON
ANALSYS
Analysis
A l i
- Knowledge elicitation process
- Capture requirement specification` develop a seed ontology
` domain and goal of ontology ` modify and extend from initial semi-
` design guidelines formal description of the ontology
Construction
` knowledge source
EVALUATI ON ` users and usage scenarios
` competency question
- Technologh-focussed evaluation framework - Support for collaboration through REFI NEM ENT
` Language conformity / Consistency brainstorming
` Interoperability / Turn around ability - Id tif representation llanguages/tools
Identify t ti /t l - FormalizationOntology
phase gy
` Performance / Memory allocation Refinement ` transfer into the target ontology
Evolution
` Scalability / Integration into frameworks ?express in formal representation
` Connectivity language
- User-focussed evaluation
Evaluation
` requirements specification document
` competency questions
M AI NTENACE
` prototype
` Feedback from beta user
` usage patterns Ontology Development -CCentralized and distributed strategy
- Quality and time
2009-03-20 skhan@wku.ac.kr page 11
12. Methodologies to develop Ontology
OTK (On-To-Knowledge) Methodology
Univ. of Kharlsrhue
Methontology
Univ. Politecnica de Madrid
Cyc methodology
Manual codification of common sense knowledge extracted by hand, machine learning tools
for new knowledge acquisition
Uschold and King
Identify the purpose, build, evaluate, document
Gruniger and Fox
Identify the i
Id tif th main scenarios, identify the competency questions, extract relevant concepts
i id tif th t ti t t l t t
and relations, formalize in FOL
KACTUS methodology
Ontology built on the basis of an application KB, by abstraction
KB
2009-03-20 skhan@wku.ac.kr page 12
13. OTK Methodology
Baseline Target O-based
GO!
ontology
gy ontology
gy Application
pp
ONTOLOGY
Maintenance
Feasibility Ontology Refinement Evaluation &
study Kickoff Evolution
Identify people Requirement Knowledge Check Manage
Focus domain specification elicitation with requirements organizational
Select tools Analyze domain Test in target maintenance
from OTK tool knowledge experts application
i i process (Who is
suite sources Develop and Analyze
responsible?
Develop refine target usage How is it done?)
GO / No GO
decision baseline ontology p
patterns
ontology Deployment
2009-03-20 skhan@wku.ac.kr page 13
14. OKT: Ontology Development Activities
Feasibility study
• Focus the domain, identify people involved
Kickoff
• Ontology Requirement Spec Doc: potential users, available sources, baseline description
users sources
from competency questions, brainstorming
Refinement
• Knowledge elicitation with domain experts, formalization
Inferencing
• F-logic, implementation issues
F logic,
Evaluation
• Check requirements, tests, quality (Ontoclean)
Application&Evolution
• Maintenance program, expected lifetime estimation
2009-03-20 skhan@wku.ac.kr page 14
15. Feasibility Study
Requirement
q Analysis
y
What is the goals of Ontology?
• usage, user specifications,…
What is relevant to fulfill the goals?
• entities, relationships, restrictions,…
What need to be modeled?
• key concepts and components …
components,
What granularity is useful?
Many
y factors other than technology determine the success
gy
of ontology development.
Focus domain for ontology
Identify people involved
GO / No GO decision
2009-03-20 skhan@wku.ac.kr page 15
16. Ontology Kickoff
Ontology Requirements Specification Document (ORSD)
Domain & Goal
Design guidelines
Available knowledge sources
g
Potential users and user scenarios
Applications supported by the ontology
Analyze knowledge sources
Develop baseline ontology description
Draft version, typically most important concepts and relations are identified
2009-03-20 skhan@wku.ac.kr page 16
18. Refinement
Knowledge elicitation with domain experts
Refine concepts and relations
Concepts should be close to entities (physical or logical) and relationships in the domain.
Typically axioms are identified
Consider
C id reuse of other ontology.
f th t l
Formalize
E.g. F L i
E F-Logic, DAML+OIL
Axioms depend on language capabilities
Develop and refine target ontology
Different tools may help in the implementation.
2009-03-20 skhan@wku.ac.kr page 18
20. Evaluation
Check requirements (ORSD)
Are all CQs answered?
Is the ontology within the scope?
Check completeness, consistence and avoid redundancy
Test in target application
Analyze usage patterns
Deploy
D l applications
li ti
Produce clear informal and formal documentation.
2009-03-20 skhan@wku.ac.kr page 20
21. Maintenance & Evolution
In real world things are changing – and so do the requirements and the
specifications for ontologies!
Manage organizational maintenance process
g g p
• Who is responsible?
• How is it done?
Support evolution of ontology-based application(s)
• Identify new requirements
• Ch
Change specifications and accordingly application(s)
ifi ti d di l li ti ( )
2009-03-20 skhan@wku.ac.kr page 21
22. Methontology Framework
Ontology
gy Development Process (
p (which activities)
)
• Management, Development, Support
Life Cycle (order of activities)
• Evolving Prototype.
Methodology (how to carry out)
• Specification
• Knowledge Acquisition
• Conceptualization
• Integration
• Implementation
• Evaluation
l i
• Documentation
2009-03-20 skhan@wku.ac.kr page 22
24. METHONTOLOGY: Specification
Produce an Ontology Specification Document (OSD)
Content
Purpose
Scenarios of use
Possible end users
Level of formality of the ontology
Scope
Granularity
Technique
Competency Questions
C t Q ti
Output
Ontology
Specification
Document
2009-03-20 skhan@wku.ac.kr page 24
25. METHONTOLOGY: Conceptualization
Organize
g and structure the knowledge acquired during the
g q g
knowledge acquisition
Terms glossary from Ontology Spec Doc
Primiti es for Modelling Ta onomies
Primitives Taxonomies
• Subclass-of
• Disjoint decomposition
• Exhaustive-Decomposition
• Partition
Ad-hoc l i
Ad h relations
Definition of
• Concept Dictionary, Instance Attributes,Class Attributes
• Formal axioms, Rules, Instances
2009-03-20 skhan@wku.ac.kr page 25
26. Ontology Building
gy g
Some slides are from University of Manchester and University of Southampton.
27. Building Ontologies
No field of Ontological Engineering equivalent to Knowledge or
Software Engineering;
No sta da d methodologies for bu d g ontologies;
standard et odo og es o building o to og es;
Such a methodology would include:
a set of stages that occur when building ontologies;
t f t th t h b ildi t l i
guidelines and principles to assist in the different stages;
an ontology life-cycle which indicates the relationships among stages.
Gruber's guidelines for constructing ontologies are well known.
2009-03-20 skhan@wku.ac.kr page 27
28. The Development Lifecycle
Two kinds of complementary methodologies emerged:
Stage-based, e.g. TOVE [Uschold96]
Iterative evolving prototypes, e.g. MethOntology [Gomez Perez94].
Most have TWO stages:
Informal stage
• ontology is sketched out using either natural language descriptions or
some diagram technique
Formal stage
• ontology is encoded in a formal knowledge representation language,
language
that is machine computable
An ontology should ideally be communicated to people and
gy y p p
unambiguously interpreted by software
the informal representation helps the former
the formal representation helps the latter.
latter
2009-03-20 skhan@wku.ac.kr page 28
29. A Provisional Methodology
A skeletal methodology and life-cycle for building
ontologies;
i
Inspired by the software engineering V-process model;
The left side charts The right side charts the
the processes in guidelines, principles and evaluation
building an ontology used to ‘quality assure’ the
ontology
gy
The overall process moves through a life-cycle.
p g y
2009-03-20 skhan@wku.ac.kr page 29
31. An Ontology Building Life‐cycle
Identify purpose and scope
Consistency
Kn w g
Knowledge acquisition
qu n Checking
Building
Language and
Conceptualisation
C t li ti
representation
Integrating
existing Available
Encoding ontologies development
tools
t l
Evaluation Ontology L
O t l Learning
i
2009-03-20 skhan@wku.ac.kr page 31
32. Questions
How do we obtain our conceptualisation?
The role of texts, experts and other sources
How do we derive conceptualisation from texts etc
How do we cope with tacit conceptualisations?
How do we use models with the expert?
HHow do we validate the conceptualisation?
d lid t th t li ti ?
2009-03-20 skhan@wku.ac.kr page 32
33. Knowledge Acquisition
The process of capturing knowledge including
various forms of conceptualisation from
whatever source including experts, documents,
manuals, case studies etc.
Knowledge Elicitation
g
techniques that are used to acquire knowledge direct
from human experts
Machine Learning
use of AI pattern recognition methods to infer patterns
from sets of examples
f t f l
2009-03-20 skhan@wku.ac.kr page 33
34. First Steps ‐
Initial Understanding of the Domain
Initial Understanding of the Domain
Problem Description
p
List knowledge resources (verify that knowledge really
exists)
Experts, Technical Authorities
Text Books, Training Material
M
Manuals and P
l d Procedures
d
Databases and Case Histories
Produce domain “yellow pages”
Establish performance metrics
Initial task environment analysis
2009-03-20 skhan@wku.ac.kr page 34
35. Document and Text Analysis
Look at the structure
how material is organised into topics and sub-topics
Co te t a a ys s
Content analysis
Extract major linguistic categories
• nouns - objects and concepts
• verbs – relations
• modifiers - properties and values
• connectives - rules and links
Use Intermediate representations
Pseudo production rules
Small concept networks and hierarchies
2009-03-20 skhan@wku.ac.kr page 35
36. Problems of Document and Text Analysis
Documents and texts are written for specific p p
p purposes that
may not reveal real knowledge or explicit
conceptualisations
Duty l
logs and rosters
d
Teaching texts
All textual
t t l analysis is a form of content analysis - th
l i i f f t t l i the
interpreter may or may not be imputing the correct
conceptualisation
p
Difficult to reconstruct the context – need to capture
acquisition and design rationales
2009-03-20 skhan@wku.ac.kr page 36
37. Session Plan
The importance of an acquisition p
p q plan
A detailed agenda of what is to be covered during a KA
session.
Should include:
an introduction describing the objectives
description of the techniques to be used
questions to be asked (if required)
timings
Should be sent to the expert at least one day in advance of
the session
2009-03-20 skhan@wku.ac.kr page 37
38. Knowledge Acquisition (KA)Techniques
Methods that help acquire and validate knowledge from an expert
during a KA session.
Three important types:
ee po ta t
natural techniques
contrived techniques
modelling and mediating representation techniques
2009-03-20 skhan@wku.ac.kr page 38
39. KA Typology
unstructured interview
interviews semi-structured
interview
structured interview
natural techniques
observation techniques
group meetings
questionnaires
card sorting
three card trick
rep grid technique
KA techniques
limited time
contrived techniques constrained tasks
limited information
20-questions
commentating
teach back
laddering
process mapping
modelling techniques
concept mapping
state diagram mapping
2009-03-20 skhan@wku.ac.kr page 39
40. Natural Techniques
KA techniques that involve the expert performing tasks they
would normally do as part of their job.
V i ti
Variations:
Interviews
Observational techniques
q
(Group meetings)
(Questionnaires)
2009-03-20 skhan@wku.ac.kr page 40
41. Interviews
KA technique in which the knowledge engineer asks
q g g
questions of the expert or end user.
Essential method for acquiring explicit conceptualisations
and k
d knowledge, b t poor f t it k
l d but for tacit knowledge.
l d
Variations:
Unstructured interview
Semi-structured interview
Structured interview
2009-03-20 skhan@wku.ac.kr page 41
42. Unstructured Interview
A
An i
interview in which the knowledge engineer has no pre-
i i i i
defined questions.
Basically a chat to find out broad aspects of the expert s
expert’s
knowledge.
An aid to designing a KA session plan.
g g p
2009-03-20 skhan@wku.ac.kr page 42
43. Semi‐structured Interview
An interview in which pre-prepared q
p p p questions are used to
focus and scope what is covered
Also involves unprepared supplementary questions for
clarification and probing.
l ifi ti d bi
Questions should be:
designed carefully
sent to the expert beforehand
asked verbatim (read-out as written)
include timings
The recommended interview technique at the start of most
KA projects.
projects
2009-03-20 skhan@wku.ac.kr page 43
44. Structured Interview
An interview in which the knowledge engineer follows a
g g
pre-defined set of structured questions but can ask no
supplementary questions.
Often involves filling-in a matrix or generic headings.
2009-03-20 skhan@wku.ac.kr page 44
45. Contrived Techniques
KA techniques that involve the expert performing tasks
they would not normally do as part of their job.
Most of these techniques come from psychology.
U f l f capturing t it knowledge, excellent for
Useful for t i tacit k l d ll t f
conceptualisations.
Important types:
card sorting
three card trick
repertory grid technique
constrained tasks
20 questions
20-questions
commentating
teach back
2009-03-20 skhan@wku.ac.kr page 45
46. Card Sorting
KA technique in which a collection of concepts ( other
q p (or
knowledge objects) are written on separate cards and
sorted into piles by an expert in order to elicit classes based
on attributes
attributes.
Also enables significant elicitation of properties and
dimensions
Used to capture concept knowledge and tacit knowledge
Use in conjunction with triadic method
Can also sort objects or pictures instead of cards
2009-03-20 skhan@wku.ac.kr page 46
47. Triadic Elicitation Method
KA technique used to capture the way in which an expert
q p y p
views the concepts in a domain.
Involves presenting three random concepts and asking in
what way t of th
h t two f them are similar but diff
i il b t different f
t from the
th
other one.
Answer will give an attribute.
A good way of acquiring tacit knowledge.
Book Paper Computer ???
2009-03-20 skhan@wku.ac.kr page 47
48. Repertory Grid technique
KA technique used for a number of p p
q purposes:
to elicit attributes for a set of concepts
to rate concepts against attributes using a numerical scale
uses statistical analysis to arrange and group similar concepts and
t ti ti l l i t d i il t d
attributes
A useful way of capturing concept knowledge and tacit
knowledge
Requires special software (PC-PACK)
2009-03-20 skhan@wku.ac.kr page 48
50. Constrained Tasks
KA technique in which the expert p
q p performs a task they
y
would normally do, but with constraints.
Variations:
limited time
limited data
Useful for focusing the expert on essential knowledge and
priorities
2009-03-20 skhan@wku.ac.kr page 50
51. 20‐Questions
KA technique in which the expert asks y
q p yes/no q
questions to
the knowledge engineer in order to deduce an answer.
The knowledge engineer need not know much about the
domain, or have an answer in mind, just answer “yes” or
“no” randomly.
no
The q
questions asked pprovide a ggood way of q
y quickly
y
acquiring attributes in a prioritised order.
2009-03-20 skhan@wku.ac.kr page 51
52. Commentating and protocol generation
KA technique in w c the expert p ov des a
ec que which e e pe provides
running commentary of their own or another’s
task performance.
A valuable method for acquiring process
knowledge and tacit knowledge.
Variations:
self-reporting
p g
imaginary self-reporting
self-retrospective
shadowing
retrospective shadowing
2009-03-20 skhan@wku.ac.kr page 52
53. Teach back
KA technique in which the knowledge engineer explains
knowledge from part of the domain back to the expert.
The expert then makes comments.
Helps reveal misunderstandings and clarifies terminology.
2009-03-20 skhan@wku.ac.kr page 53
54. Laddering
KA technique that involves the co s uc o ,
ec que vo ves e construction,
modification and validation of trees.
A valuable method for acquiring concept
q g p
knowledge and, to a lesser extent, process
knowledge.
Can make use of various trees:
concept tree
p
composition tree
attribute tree
process tree
decision tree
cause tree
2009-03-20 skhan@wku.ac.kr page 54
55. Modelling Techniques
KA techniques that use knowledge models as the
focus for discussion, validation and modification of
knowledge.
Can use any form of model, but important types
include:
process mapping
concept mapping
state diagram mapping
2009-03-20 skhan@wku.ac.kr page 55
56. Process Mapping
KA technique that involves the construction, modification
and validation of process maps.
A valuable method for acquiring process knowledge and
tacit k
t it knowledge.
l d
2009-03-20 skhan@wku.ac.kr page 56
57. Process Map ‐ Example
aims of research information sources
T1
Conduct literature review
senior investigator
literature review is empirical research required?
resources available
yes no
T2
Conduct empirical research
researcher
empirical results
T3
Write-up research research report
h t
senior investigator
2009-03-20 skhan@wku.ac.kr page 57
58. Concept Mapping
KA technique that involves the construction, modification
and validation of concept maps.
p p
A good method for acquiring concept knowledge.
2009-03-20 skhan@wku.ac.kr page 58
59. Concept Map ‐ Example
written by
itt b
Author
Oliver Twist is a
Charles Dickens
wrote
is a
shorter wrote admired
is a than Dostoevsky
Bleak House wrote on
Book is a born in
Russia
has part
Page Paper
made from
2009-03-20 skhan@wku.ac.kr page 59
60. State Diagram Mapping
KA technique that involves the construction, modification
and validation of a state diagram.
A different approach to process mapping.
Useful for capturing process knowledge, concept
knowledge and tacit knowledge.
2009-03-20 skhan@wku.ac.kr page 60
61. State Diagram ‐ Example
Your number is dialed
On hook - no ringing On hook - ringing
Lift receiver
Person at other end rings off
Lift receiver
Off hook - conversation
Hang up Phone i answered at
Ph is d t
Off hook - dialing tone
other end
Hang up
Dial number Off hook - ringing tone
Off hook - dialing
Complete dialing
C l t di li
2009-03-20 skhan@wku.ac.kr page 61
63. Designing a KA plan
We need different techniques because
eed d e e ec ques bec use
there are different types of knowledge
acquiring a certain type knowledge is made more efficient
q g yp g
using the right technique
• e.g. can't get tacit knowledge using interviews
Three types of KA techniques
Natural (e.g. interviews, observation)
Contrived (e.g. commentary, rep grid, 20-questions)
Modelling (e.g. process mapping)
2009-03-20 skhan@wku.ac.kr page 63
64. Designing a KA Session Plan
1.
. Be clear what knowledge you want from the
ece w ow edge w o e
session.
2. Write an introduction summarising what
knowledge y want from the session.
g you
3.
3 Select the best KA technique/s to use.
use
How do we do this? …..
2009-03-20 skhan@wku.ac.kr page 64
65. Designing a KA Session Plan
4. Place the techniques selected in a clear and
. ce e ec ques se ec ed ce d
logical order
e.g. interview questions first
g q
e.g. commentary and protocols before process mapping
5. Always end the session plan with the following
question:
"Bearing in mind the goals of this session, what vital
knowledge have we not yet covered“
6. Assign timings to each section.
2009-03-20 skhan@wku.ac.kr page 65
66. Designing a KA Session Plan
7. If possible, check the session p
p , plan with y
your p j
project
manager or colleague and make amendments if
necessary.
8. Send (email, fax) the session plan to the expert at least
one day before the session.
9. Make any changes the expert suggests.
y g p gg
10. During the session, stick to the plan and keep to the
timings
2009-03-20 skhan@wku.ac.kr page 66
67. Which KA technique to use
Decide
ec de what type/s o co cep u s o and
w ype/s of conceptualisation d
knowledge you need from the expert
Is it structural objects oriented knowledge? (i.e. of concepts,
j g ( p
attributes, states & relationships)
Is it process knowledge? (i.e. how to do things)
Is it explicit knowledge? (i.e. easily explained)
Is it tacit knowledge? (i.e. not easily explained)
Use the diagram shown next to select the best
technique/s to use..
use
2009-03-20 skhan@wku.ac.kr page 67
70. Types of Ontology Tools
Ontology
gy development tools
p
Editors and browsers
Graphical editors
Translators
Ontology library management
Ontology documentation
Ontology population
Evaluation
Evolution
Merge and alignement tools
Ontology-based annotation tools
QQuerying tools and inference engines
i t l di f i
Ontology learning tools
2009-03-20 skhan@wku.ac.kr page 70
72. References
Methodologies for building ontologies from the scratch
Cyc
C methodology URL: http://www.cyc.com
//
Uschold and King URL: Not available
Grüninger and Fox URL: Not available
KACTUS methodology URL: Not available
METHONTOLOGY URL: Not available
SENSUS methodology URL: Not available
On To Knowledge
On-To-Knowledge Methodology URL: http://www ontoknowledge org/
http://www.ontoknowledge.org/
Methodologies for reengineering ontologies
Method for reengineering ontologies integrated in Methontology URL: Not available
g g g g gy
Methodologies for cooperative construction of ontologies
CO4 methodology URL: Not available
(KA)2 methodology URL: Not available
2009-03-20 skhan@wku.ac.kr page 72
73. References
Ontology learning methodologies
Aussenac-Gille's and colleagues methodology URL: http://www.biomath.jussieu.fr/TIA/
Maedche and colleagues' methodology URL: Not available
Ontology merge methodologies
O t l th d l i
FCA-merge URL: Not available
PROMPT URL: Not available
ONIONS URL: Not a ailable
available
Ontology evaluation methods
OntoClean: Guarino's group methodology URL: Not available
Gómez Pérez's evaluation methodology URL: Not available
2009-03-20 skhan@wku.ac.kr page 73