This document summarizes a study on the effects of computational thinking-based software (SW) coding education on creative problem solving. The study derived a creative problem-solving model based on the nine main concepts of computational thinking and applied this model to SW coding education for an experimental group. Pre- and post-tests were administered to measure changes in students' affective and cognitive abilities. The results showed that computational thinking-based SW coding education positively influenced students' creative problem-solving skills.
2. A Study on the SW Coding Education and Creative Problem Solving
http://www.iaeme.com/IJARET/index.asp 327 editor@iaeme.com
is increasing to explain creative problem solving by establishing nine main concepts called
Computational Thinking. Computational Thinking provides an empirical methodology for the
process of solving problems in each step by applying the principles of computer science to
problem -solving.
Therefore, in this study, we will apply the model of Creative Problem-Solving, which has
been focused on creative talent education so far, to SW education. Through this, a more
empirical and systematic software education creative problem -solving model is proposed.
2. RELEATED BACKGROUND
2.1. Creative Problem Solving and Concept of Computational Thinking
Creative problem solving is a mental process of creating a Creative Problem-Solving method
in order to solve a problem. Creative problem solving means not getting any help and being
able to solve a problem independently [1].
The Creative Problem-Solving model is the most well-known model approaching creative
problem solving in a systematic way. Osborn initially proposed a 7 step Creative Problem-
Solving model in 1953 which further improved through several changes. Recently, Isaksen &
Treffiner(2005) improved to version 6.1TM
with 4 components(Understanding the Challenge,
Generating Ideas, Planning your Approach, Preparing for Action) and 8 steps(Constructing
Opportunities, Exploring Data, Framing Problems, Generating Ideas, Developing Solutions,
building Acceptance, Appraising tasks, Designing Process) and is being used in various
researches [2-3]. Computational Thinking was first used in 1996 by Seymour Papert and
became widely known by Wing [4-5]. Computational Thinking uses the basic concepts and
principals of computer science. Computational Thinking is defined as problem solving, system
design and the understanding of human deed Computational Thinking also includes abstract
thinking, recursive thinking, analytical thinking, procedural thinking and logical thinking. ISTE
defined the main concepts of Computational Thinking using the results of research earned from
Barr, Harrison, & Coney and is shown in the Tables 1 [6-7].
Tables 1 Computational Thinking Main Concepts
Concept Definition
Data Collection
Problem understanding, analysis and collect data based on analysis to
solve the problem
Data Analysis
Carefully sorting and analyzing the data collected and data provided in the
problem
Data
Representation
Express data in problem using graphs, charts, words and images
Problem
Decomposition
Dividing and analyzing the problem to solve the problem
Abstraction Defining the main concepts to reduce the complexity of the problem
Algorithm and
Procedures
Expressing the steps required to solve the problem until now
Automation
Creating an algorithm of the solution procedure for a computing machine
to carry it out
Simulation Creating an experimental model to solve the problem
Parallelization Coming up with a common objective to solve a problem
3. Jungin Kwon
http://www.iaeme.com/IJARET/index.asp 328 editor@iaeme.com
2.2. Creative Problem-Solving Model based on Computational Thinking
The main concepts of Computational Thinking were divided into 3 stages, and then we applied
it in Creative Problem-Solving model. In the introduction stage for problem understanding,
Data Collection, Data Analysis and Data Representation concepts are mainly used. Lesson
development based on Problem Decomposition, Abstraction, Algorithm and Procedure effect
the speed of reaching fast learning goals. Using automation, simulation and Parallelization for
the solutions earned through lesson development to organize data help solve complex and
difficult problems. Figure 1 shows a model of such hypotheses organized together. Therefore,
we can establish a research model which shows the 9 main concepts of computational thinking
effect the improvement of creating problem solving [8].
Figure 1. Creative Problem-Solving Model based on Computational Thinking
3. METHODS AND RESULTS
3.1 Research Subject and Methods
In order to verify the experiment of the research model, a freshman from S University in Seoul
was selected as an experiment subject. In this experiment, a total of 434 people were randomly
divided into the experimental group and the control group.
Table 2
Group Target number of people
Experimental Group 218
Controlled Group 216
Total 434
In order to analyses the effectiveness of Computational Thinking-based SW Coding
education on creative problem solving, the experimental group and the controlled group were
divided. Prior to the experiment, to test the homogeneity of the experimental group and the
controlled group, a two-independent sample t-test was conducted based on the results of the
pre-test. As a result, the two groups were found to be homogeneous. In the experimental group,
training in which SW Coding based on Computational Thinking was applied to the Creative
Problem-Solving teaching-learning course was conducted 10 times out of 15 weeks, and
controlled group was applied the teaching-learning course plan for general SW coding.
Education was conducted to examine the differences between the two groups, and the degree
of improvement in the affective part and the improvement in the cognitive part were
investigated. Test papers were used for pre-test and post-test examination of the affective
domain of creative problem solving. This test paper was developed by the MI Research Team
4. A Study on the SW Coding Education and Creative Problem Solving
http://www.iaeme.com/IJARET/index.asp 329 editor@iaeme.com
at Seoul National University's Psychology Laboratory in 2004 based on the “Simple Creative
Problem-Solving Ability Test and Development Research (I)” (2001) studied by the Korea
Educational Development Institute [9]. The simple Creative Problem-Solving ability test
consists of self-confidence and independence, diffuse thinking, critical and logical thinking,
and motivational thinking. It is composed of 5 questions for each area and is composed of a 5-
point scale, and each area is quantified through total and average. The larger the number of
results, the higher the Creative Problem-Solving ability.
A test sheet was developed to test the cognitive domain of creative problem solving. This
test paper was developed by adjusting the problem-solving literacy area of the 2003 PISA open
questions and some of the 2006 open questions to match the level of the first year of university,
and some of the items of the logical thinking ability test paper (GALT) to match the level of
the first year of university [10]. After the test sheet was developed, the questions were revised
based on the review of education experts and field teachers, and pre-test and post-test were
conducted.
3.2. Experiment Verification Methods
3.2.1 Pre- Test
1) The affective test of creative problem solving
For the affective test of Creative Problem-Solving, the number of students who participated in
the experiment collected a total of 463 questionnaires, 238 in the experimental group and 225
in the control group. However, a total of 434 surveys were statistically analyzed, with 218
people in the experimental group and 216 people in the control group, excluding test papers
such as unfaithful responses or typographical errors. A two-independent sample t-test was
performed to compare the mean of the experimental group and the control group and to verify
the similarity between the two groups are shown in the Tables 2.
Table 2 Statistical Analysis of Experimental Group and Control Group (Definitive Area) _Pre-
Verification
Group
classification
Mean
Standard
deviation
Standard error
of the mean
Self-confidence
and
independence
Experimental
group
13.8440 3.48181 .23582
Controlled group 13.8935 3.32192 .22603
Divergent
thinking
Experimental
group
15.2248 3.45210 .23381
Controlled group 15.3611 3.37283 .22949
Critical and
logical thinking
Experimental
group
16.4266 2.95381 .20006
Controlled group 16.3333 3.29623 .22428
Motivational
factor
Experimental
group
17.0550 3.13104 .21206
Controlled group 17.0972 3.06439 .20851
Shown in the Tables 3. In the results of the independent sample t-test of the experimental
group and the control group, the significance probability is 0.318, 0.518, 0.286, and 0.935 (self-
confidence and independence, divergent thinking, critical/logical thinking, and motivational
factors), respectively, which is greater than the significance probability of 0.05. It can be said
that the variance of the two groups is the same. In the case of self-confidence and independence,
the t-value is -0.151, which is greater than -1.96, and the t-value has a significance probability
of 0.88, which is greater than 0.05, so it can be concluded that there is no difference in the mean
5. Jungin Kwon
http://www.iaeme.com/IJARET/index.asp 330 editor@iaeme.com
between the two groups. In the case of divergent thinking, it can be concluded that there is no
difference in the mean between the two groups because the t value is -0.416, which is greater
than -1.96, and the t value has a significance probability of 0.678, which is greater than 0.05.
In the case of critical and logical thinking, it can be concluded that there is no difference in the
mean between the two groups because the t-value is 0.311, which is less than 1.96, and the t-
value has a significance probability of 0.432, which is greater than 0.05. In the case of the
motivational factor, the t-value is -0.142, which is greater than -1.96, and the t-value has a
significance probability of 0.887, which is greater than 0.05, so it can be concluded that there
is no difference in the mean between the two groups.
Therefore, it is possible to conclude that the experimental group and the control group are
statistically homogeneous in the positive experimental verification.
Table 3 Independent sample t-test result (Definitive area) _Pre-validation
Levne`s test of
equal variance
T-test for equality of means
F
Significanc
e
probability
t
Degree of
freedom
Significance
probability
(both sides)
Mean
difference
Standard
error of
the
difference
95% confidence
interval of the
difference
Minimum Maximum
Self-
confide
nce and
indepen
dence
Equal
variance
is
assumed
.998 .318 -.151 432 .880 -.049 .3267 -.691 .592
No equal
variance
is
assumed
-.151 431.3 .880 -.049 .3266 -.691 .592
Diverge
nt
thinkin
g
Equal
variance
is
assumed
.418 .518 -.416 432 .678 -.136 .3276 -.780 .507
No equal
variance
is
assumed
-.416 431.9 .677 -.136 .3276 -.780 .507
Critical
and
logical
thinkin
g
Equal
variance
is
assumed
1.143 .286 .311 432 .756 .0932 .3003 -.497 .683
No equal
variance
is
assumed
.310 426.0 .756 .0932 .3005 -.497 .684
Motivat
ional
factor
Equal
variance
is
assumed
.007 .935 -.142 432 .887 -.042 .2974 -.626 .542
No euqal
variance
is
assumed
-.142 431.9 .887 -.042 .2974 -.626 .542
6. A Study on the SW Coding Education and Creative Problem Solving
http://www.iaeme.com/IJARET/index.asp 331 editor@iaeme.com
2) Cognitive test of creative problem solving
Table 4 Statistical analysis of experimental and control groups (cognitive domain)_Pre-verification
Group classification Mean
Standard
deviation
Standard
error of the
mean
Cognitive test
score
Experimental group 51.88 19.383 1.313
Controlled group 53.03 18.849 1.283
To compare and evaluate the prior cognitive abilities of the experimental group and the
control group, a two-independent sample t-test was conducted to verify the similarity of the two
groups, similar to the analysis method of the affective domain. The results of the pre-cognition
verification of the two groups are shown in Table 4.
Shown in the Tables 5 as a result of the independent sample t-test of the experimental group
and the control group, the significance probability of the F value was 0.765, respectively, which
is greater than the significance probability of 0.05, so the variance of the groups can be said to
be the same. If the two groups have the same variance, the hypothesis is tested with the t-value
and the significance probability when equal variance is assumed, so the t-value is -0.627, which
is greater than -1.96, and the t-value is 0.531, which is greater than 0.05. It can be concluded
that there is no difference between them. Therefore, it is possible to conclude that the
experimental group and the controlled group are statistically homogeneous groups in cognitive
experiment verification.
Table 5. Results of independent sample t-test (cognitive domain) _Pre-test
Levne`s test of
equal variance
T-test for equality of means
F
Signifi
cance
proba
bility
t
Degree
of
freedo
m
Significan
ce
probabilit
y
(both
sides)
Mean
differen
ce
Standar
d error
of the
differen
ce
95% confidence
interval of the
difference
Minimu
m
Maximu
m
Cogniti
ve test
score
Equal
variance
is
assumed
.090 .765 -.627 432 .531 -1.152 1.836 -4.759 2.456
No equal
variance
is
assumed
-.628 431.8 .531 -1.152 1.835 -4.759 2.456
3.2.2. Experimental Treatment
Among the Creative Problem-Solving model, one of the systematic approaches to creative
problem solving, Treffinger, Isaksen & Dorval prepared a course plan according to the model
stage of Creative Problem-Solving v6.1 released in 2006. The Creative Problem-Solving v6.1
model consists of four components and eight specific steps: understanding the challenge,
generating ideas, preparing for action, and planning the approach. In this study, among the four
stages of the Creative Problem-Solving model, understanding challenges was applied as
understanding problems, generating ideas as designing solutions, and preparing for
implementation as the best choice. And the teaching-learning stage was organized according to
the stages of introduction, development, and arrangement in the teaching-learning process. In
order to Understanding Problem, it was possible to understand the learning problem through
7. Jungin Kwon
http://www.iaeme.com/IJARET/index.asp 332 editor@iaeme.com
Data Collection, Data Analysis, and Data Representation. At this time, data collection, analysis,
and expression should be parallelized at the same time, and learners should be able to increase
the effect of understanding problems. In addition, for generating ideas for solving problems, a
teaching-learning process plan is constructed to go through the process of abstraction and
algorithms & procedures. Abstraction utilizes the learner's empirical knowledge to create a
solution for problem solving, and design it so that the created solution can be logically and
critically structured.
Finally, the teaching-learning process plan was designed to find the best solution through
computer-based simulation to select the best solution for problem solving.
3.2.3. Post- Test
1) The affective test of creative problem solving
For the affective test of creative problem solving, the experimental group and the controlled
group performed a post-test through the simple Creative Problem-Solving ability test paper,
which is the same test paper as the pre-test. shown in the Tables 6.
Table 6 Statistical analysis of experimental group and control group (definitive area) _Post
verification
Group
classification
Mean
Standard
deviation
Standard error
of the mean
Self-confidence
and independence
Experimental group 15.1697 2.88067 .19510
Controlled group 14.1019 3.18847 .21695
Divergent
thinking
Experimental group 15.7844 2.87598 .19479
Controlled group 15.5463 3.12495 .21263
Critical and
logical thinking
Experimental group 18.0734 2.34847 .15906
Controlled group 16.4769 3.11029 .21163
Motivational
factor
Experimental group 18.0046 2.33289 .15800
Controlled group 17.1759 2.96594 .20181
Table 7 Independent sample t-test result (Definitive area) _Post-validation
Levne`s test of
equal variance
T-test for equality of means
F
Significanc
e
probability
t
Degree
of
freedo
m
Significanc
e
probability
(both
sides)
Mean
differenc
e
Standard
error of
the
differenc
e
95% confidence
interval of the
difference
Minimu
m
Maximu
m
Self-
confidence
and
independe
nce
Equal
variance
is
assumed
2.293 .131 3.662 432 .000 1.067 .2916 .49467 1.64108
No equal
variance
is
assumed
3.660 426.8 .000 1.067 .2917 .49438 1.64136
Divergent
thinking
Equal
variance
is
assumed
.658 .418 .826 432 .409 .2381 .2882 -.32844 .80465
No equal
variance
is
assumed
.826 428.3 .409 .2381 .2883 -.32867 .80488
Critical
and logical
thinking
Equal
variance
is
assumed
9.208 .003 6.038 432 .000 1.596 .2644 1.07687 2.11622
8. A Study on the SW Coding Education and Creative Problem Solving
http://www.iaeme.com/IJARET/index.asp 333 editor@iaeme.com
No equal
variance
is
assumed
6.031 400.0 .000 1.596 .2647 1.07609 2.11699
Motivation
al factor
Equal
variance
is
assumed
11.900 .001 3.237 432 .001 .828 .2560 .3254 1.33187
No euqal
variance
is
assumed
3.233 407.6 .001 .828 .2563 .3248 1.33250
Shown in the Tables 7 as a result of the independent sample t-test of the experimental group
and the control group, the significance probability of the F value for self-confidence and
independence was 0.131 each, which is greater than the significance probability of 0.05, so the
variance of the groups can be said to be the same.
Therefore, since the variance of the two groups is the same for self-confidence and
independence, the t-value and the significance probability are hypothesized when equal
variance is assumed. At this time, since the t-value is 3.662, which is greater than 1.96, and the
significance probability of the t-value is 0.000, which is less than 0.05, it can be concluded that
there is a difference between the two groups. Since the significance probability of the F value
for diffuse thinking is 0.418 each, which is greater than the significance probability of 0.05, the
variance of the groups can be said to be the same. Therefore, in diffusive thinking, since the
variance of the two groups is the same, the hypothesis test is performed on the t value and the
significance probability as a case where equal variance is assumed. Since the t-value at this time
is 0.826, which is less than 1.96 and the significance probability of the t-value is 0.409, which
is greater than 0.05, we can conclude that there is no difference between the two groups. Since
the significance probability of the F value for critical and logical thinking is 0.003 each, which
is less than the significance probability of 0.05, it can be said that the group's variance is not
the same. Therefore, in critical and logical thinking, since the variances of the two groups are
not the same, the t-value and the significance probability are hypothesized when equal variance
is not assumed. Since the t-value at this time is 6.031, which is greater than 1.96, and the
significance probability of the t-value is 0.000, which is less than 0.05, it can be concluded that
there is a difference between the two groups. Since the significance probability of the F value
for the motivational factor is 0.001, which is less than the significance probability of 0.05, it
can be said that the variance of the groups is not the same. Therefore, since the variance of the
two groups is not the same as the motive factor, the t value and the significance probability are
hypothesized when equal variance is not assumed. Since the t-value at this time is 3.233, which
is greater than 1.96, and the significance probability of the t-value is 0.001, which is less than
0.05, it can be concluded that there is a difference between the two groups. Therefore, the
experimental group and the controlled group can conclude that self-confidence and
independence, critical and logical thinking, and motivational factors are statistically
heterogeneous groups in the positive post-experimental verification. Divergent thinking
appeared as a homogeneous group.
2) Cognitive test of creative problem solving
The post-cognitive test for creative problem solving in the experimental group and the
controlled group is similar to when the preliminary test sheet was formed, and the problem-
solving literacy area of the 2003 PISA open questions and some of the 2006 open questions
were adjusted to suit the level of the first year of university, Some of the items in (GALT) were
used, but they were not duplicated with those used in the pre-test. After the test sheet was
developed, the questions were revised based on the review of education experts and field
9. Jungin Kwon
http://www.iaeme.com/IJARET/index.asp 334 editor@iaeme.com
teachers, and post-tests were conducted as in the pre-test. The results of the post-test by the
cognitive test paper are shown in Table 8.
As a result of the post-test, the average score of the experimental group and the controlled
group was 59.01 and 55.21, respectively, which improved the average score in both groups
compared to the pre-test. An independent sample t-test was conducted to verify whether the
difference between the means of the two groups was a statistically significant difference
Table 8. Statistical analysis of experimental and control groups (cognitive domain) _Post-verification
Group classification Mean
Standard
deviation
Standard
error of the
mean
Cognitive test score
Experimental group 59.01 16.588 1.124
Controlled group 55.21 19.489 1.326
Shown in the Tables 9 as a result of the independent sample t-test of the experimental group
and the controlled group, the F-value significance probability is 0.012 each, which is less than
the significance probability of 0.05, so it can be said that the group's variance is not the same.
If the variances of the two groups are not equal, the hypothesis test is performed with the t value
and the significance probability as the case where equal variance is not assumed.
Since the t-value at this time is 2.189, which is greater than 1.96, and the significance
probability of the t-value is also 0.029, which is less than 0.05, it can be concluded that there is
a difference between the two groups. Therefore, the experimental group and the controlled
group can be concluded as statistically heterogeneous groups in cognitive experimental
verification.
Table 9. Results of independent sample t-test (cognitive domain) _Post-test
Levne`s test of
equal variance
T-test for equality of means
F
Significance
probability
t
Degree
of
freedom
Significance
probability
(both sides)
Mean
difference
Standard
error of
the
difference
95% confidence
interval of the
difference
Minimum Maximum
Cognitive
test score
Equal
variance
is
assumed
6.382 .012 2.191 432 .029 3.805 1.737 .392 7.219
No
equal
variance
is
assumed
2.189 420.0 .029 3.805 1.738 .389 7.222
3.3. Results of Experimental Verification
3.3.1. Affective Verification Result of Creative Problem Solving
Shown in the Tables 10 as a result of pre-tests and post-tests for the affective tests of the
experimental group and the control group, the items of self-confidence and independence,
critical and logical thinking, and motivational factors were measured for the effectiveness of
the experiment. But there is no measurement of effectiveness in divergent thinking.
10. A Study on the SW Coding Education and Creative Problem Solving
http://www.iaeme.com/IJARET/index.asp 335 editor@iaeme.com
Table 10. Comparison of pre- and post-tests between the experimental group and the control group
(definitive domain)
Group classification N
Mean
Pre-test Post-test
Self-confidence and
independence
Experimental group 218 13.8440 15.1697
Controlled group 216 13.8935 14.1019
Divergent thinking
Experimental group 218 15.2248 15.7844
Controlled group 216 15.3611 15.5463
Critical and logical
thinking
Experimental group 218 16.4266 18.0734
Controlled group 216 16.3333 16.4769
Motivational factor
Experimental group 218 17.0550 18.0046
Controlled group 216 17.0972 17.1759
In other words, it was concluded that the SW coding class applying the Creative Problem-
Solving model based on Computational Thinking can influence creative problem solving such
as self-confidence and independence, critical and logical thinking, and motivational factors.
3.3.2. Cognitive Verification Result of Creative Problem Solving
In the experimental group, we took a SW coding class that applied the Creative Problem-
Solving model based on Computational Thinking. In the controlled group, a SW coding class
that applied the Creative Problem-Solving model was conducted. And, shown in the Tables 11
as a result of comparing the pretest - posttest for the cognitive tests of the experimental group
and the controlled group, the effectiveness of the cognitive domain of the experimental group
was measured, although incomplete.
Table 11. Comparison of pre- and post-tests between the experimental group and the control group
(cognitive domain)
Group classification N
Mean
Pre-test Post-test
Cognitive test score
Experimental group 218 51.88 59.01
Controlled group 216 53.03 55.21
In other words, it was concluded that SW coding education applying the Creative Problem-
Solving model based on Computational Thinking can affect the cognitive domain of creative
problem solving.
4. RESULT
A study result was drawn that suggests that SW Coding education based on Computational
Thinking can affect creative problem solving. The stage of creative problem solving is largely
the process of understanding the problem, generating ideas for solving the understood problem,
and selecting among the best solution based on designed problem-solving process. The main
concepts of Computational Thinking are applied to each stage of creative problem solving and
used as follows.
First, to understand the problem, it provides learners and instructors with the process of
collecting data, analyzing the collected data, and expressing the analyzed data. Then, presents
practical actions on what learners and instructors should do to understand the problem. At this
time, each step is not separated, and understanding of the problem brings more effective results
when work is done at the same time.
11. Jungin Kwon
http://www.iaeme.com/IJARET/index.asp 336 editor@iaeme.com
Second, if you understand the problem, you must design a solution process to solve the
problem based on the understanding. At this time, in order to design the solution process, the
process of problem solving should be designed using empirical knowledge or common sense
that the learner or instructor already knows. Based on this, a solution process should be designed
through repetitive convergence of diffuse and divergent thinking, and the designed solution
process should be procedural as a single process.
Third, among the methods of solving problems through a procedural process, the best
method is adopted. At this time, the fact that experimentation is impossible in reality or
experimentation is helpful in selecting the best solution for problem solving by using
simulation.
In this study, in order to investigate the influence of software coding education based on
Computational Thinking on creative problem solving, an empirical method was proposed for
learners and instructors to creatively solve problems at each stage. It is expected to be able to
provide more empirical experiences and opportunities for divergent and diffuse thinking to
instructors and learners, as well as increase the efficiency of problem solving.
REFERENCES
[1] K. H. Kim, Exploring the direction of informational education policy based on creating problem
solving to enhance core capabilities of future Koreans, KIEC, 2012.
[2] A. F. Osborn, Applied imagination: Principles and procedures of creative problem-solving. New
York: Scribner's, 1953.
[3] Isaksen, Scott G. & Donald J. Treffinger., “Creative problem solving: The History,
Development, and Implications for Gifted Education and Talent Development”. gifted child
quarterly, 49(4), 2005,pp.342-353.
[4] Seymour Papert., “An Exploration in the Space of Mathematics Educations”. International
Journal of Computers for Mathematical Learning, 1(1), 1996,pp.95-123.
[5] J. M. Wing, Computational Thinking. Communications of the ACM, 49(3), 2006,pp.33-35.
[6] Computer Science Teachers Association & International Society for Technology in Education,
Computational Thinking Teacher Resources, 2011,
http://csta.acm.org/Curriculum/sub/CompThinking.html.
[7] D. Bar, J. Harrison and L. Conery, Computational thinking: A Digital Age Skill for Everyone.
Learning & Learning with Technology, 38 (6), 2011,pp.20-23.
[8] Jungin Kwon, A Study on the Effectiveness of Computational Thinking based Teaching and
Learning on Students’ Creative Problem-Solving Skills. Ph.D. Dissertation, Sungkyunkwan
University, 2014.
[9] Cho Seok-hee , A study on the development of simple creative problem solving ability test (Ⅰ),
Seoul: Korea Educational Development Institute, 2001.
[10] Korea Institute for Curriculum Evaluation, OECD / PISA 2003 Evaluation Framework and
Sample Questions-Math, Science, Problem Solving Skills, Seoul: Korea Institute for
Curriculum Evaluation, 2004.