This document discusses assessing L2 writing ability. It defines the target language use domain of academic writing for community college and the language use task of writing narrative/persuasive essays. It outlines facets of L2 writing ability to assess, including control of content development, rhetorical control, grammatical control, control of register and vocabulary, and task fulfillment. The document then provides a holistic scoring rubric that grades essays based on criteria such as position taking, development of ideas, organization, use of examples, command of language for academic purposes, grammar, and sentence variety.

1. Facets of Testing
L2 Writing Ability
Pre-Convention Institute
TESOL Conference Philadelphia 2012
Cynthia S. Wiseman, Ed.D.
Borough of Manhattan Community College
City University of New York

2. Agenda
♦ Introductions
♦ General features of language assessment &
L2 writing assessment
♦ Activity
♦ Language Use Argument & L2 Writing
♦ Defining the construct: L2 Writing
♦ Framework of Task Characteristics
♦ Examination of items
♦ Review: crossword

3. Assessment is…
Homework
Implicit
•T & Ss may be unaware -- organic
•Continuous – graded, ungraded
EXAMS!!! free, pre-writing
•Instantaneous – in-class written free wrClassroom
•Cyclical – reflective practice observation
Explicit
•T & Ss are aware of assessment
•Clearly distinct from Teaching - Exam
Quiz Portfolios
Participation Self-Assessment

4. Purpose of L2 writing assessment: to collect
information about teaching & learning of L2
writing to make decisions
Teacher & Student:
–Formative decisions
• To correct S errors or not – based on student draft
• To change question of inquiry – Revise prompt? description of
assignment?
• To model a structure – based on student writing, do we need to model
a paragraph in class?
• To go to next lesson or review?
• To go more in-depth in content area – based on thin development, do
we need to do more research, discussion, reading?
• Use a different strategy to respond to write an essay or explain a
genre
–Summative decisions
• To place, pass, fail or promote a student

5. Decisions about….
♦ Individuals
– Selection for admission/employment – screening exams, like
CATW
– Placement into course of study – Department exams, multiple
measures
– Certification for profession – LAST (NY Teacher Certification
Exam)
– Prediction of future performance – TWE, DIALANG
♦ Program
– Formative, to make changes to improve program
– Summative, to continue existing program
♦ Research
– To decide on new research questions or methodology
– To change/modify view/understanding of language phenomenon

6. Uses of Language Assessments
♦ Intended use:
– To collect information for making decisions
– Beneficial consequences for stakeholders
• E.g., ESL writing teacher is teaching lesson on
cohesive devices in essay writing
– To make decision about instruction
– To change/improve instruction so Ss will effectively use
cohesive devices to improve writing
– Short fill-in-the-blank paragraph to get feedback on Ss’
learning & effectiveness of teaching

7. Introductory Activity
Think of an L2 writing assessment
development situation that you are familiar
with. Describe the context, participants, and
the test development process. What
decisions were to be made based on the
results of this assessment?
♦ What were some of the intended beneficial
consequences of assessment use in this situation?
♦ Did any problems came up as a result of this
assessment development?

8. Steps in creation of an assessment
♦ Assessment Use Argument (AUA)
♦ Set of claims:
– Conceptual links between TTs performance on
assessment and interpretation about the ability
– Decisions to be made
– Consequences
– What would the AUA of the L2 writing
assignment that you just described look like?

9. Assessment Use Argument --Bachman & Palmer (2010)

10. Assessment Development Use Bachman & Palmer (2010)

11. Initial Planning in test development
♦ What beneficial consequences do we want to happen? Who
are the stakeholders (i.e., intended TTs, etc.)? Who will be
directly affected by the use of the assessment? How?
♦ What are the specific decisions that need to be made to reach
the intended consequences?
♦ What do we need to know about the ELL’s language ability
as demonstrated in writing to make the intended decision?
♦ What sources could we use to make that decision? Is an
existing assessment available? Is it appropriate?
♦ Do assessment tasks correspond to TLU tasks?
♦ Does the developer provide evidence justifying intended uses?
♦ Do we really need to develop our own assessment?

12. Case scenario
♦ Educational Context: CC, US, diverse urban population,
85% ELL/bilingual background, open admissions
♦ Decision: To place S in ENG101, 3-credit composition
class?
♦ Beneficial consequences?
♦ What do we need to know about the ELL’s language ability
as demonstrated in writing to make the intended decision?
♦ What sources could we use to make that decision? Is an
existing assessment available? Is it appropriate?
– ACT Compass (Writing Sample Test)
♦ Do assessment tasks correspond to TLU tasks?
♦ Does the developer provide evidence justifying intended
uses?
♦ Do we really need to develop our own assessment?

13. Steps in Creating an Assessment
♦ Identify the target population/test context
♦ Identify the type of assessment
♦ Specify the specific purpose of the test
♦ Define the construct: Describe the Target Language Use
Domain & Target Language Tasks
♦ Write specifications for the test
♦ Write items/tasks that operationalize the construct &
incorporate task characteristics that correspond to TLU
tasks
♦ Create the test of items/tasks with clear instructions
♦ Create an answer key/rubric

14. Test Development Model
Bachman & Palmer, 1996

15. What is L2 writing
ability?

16. Communicative Language Ability
--Bachman (1990)
♦ Language characteristics
• Organizational Characteristics
• Grammatical (vocabulary, syntax, phonology, graphology)
• Textual (cohesion, rhetorical, conversational organization)
• Pragmatic characteristics
• Functional (ideational, manipulative, heuristic, imaginative)
• Sociolinguistic (genres, dialect/variety, register, naturalness, cultural
references and figurative language)
• Topical characteristics
• Metacognitive Strategies/Competence

17. Target Language Use (TLU)
Domain
♦ “…a set of specific language use tasks that
the TT is likely to encounter outside of the
test itself, and to which we want our
inferences about language ability to
generalize.” (p. 44)
– Distinguishing characteristics of language use
tasks to describe language use domain
– Inferences that generalize to specific domains in
which TT is likely to need to use the language
– Inferences about TT’s ability to use language in
a target language use domain

18. Language Use Task
♦ Language use task: an activity that
involves individuals in using language
for the purpose of achieving a particular
goal or objective in a particular situation
– Specific situations
– Goal-oriented
– Active participation of language users

19. TLU Domain Language Use Settings Language Use Tasks
English for Business Managing & operating •Writing memos
Communication an office •Preparing reports
•Taking phone msgs
•Writing letters
•Writing emails
•Texting
Negotiating with clients •Writing proposals
& customers •Responding to written
offers
•Writing emails
•Texting
Promoting products or • Writing advertising
services copy
• Writing solicitation
pitch

20. Characteristics of Tasks
♦ Link between tasks in the domain of test tasks
and the domain of non-test tasks –
♦ Selection or design of tests that correspond in
specific ways to language use tasks
♦ Extent and ways TTs’ language ability is
engaged
♦ Degree of correspondence between
characteristics of given test task and a particular
language use task: authenticity, validity of
inferences, domain to which inferences
generalize
♦ Control of characteristics of the test task through
test design and development

21. Language Task Characteristics
Characteristics of the setting •Physical Characteristics
•Participants
•Time of task
Characteristics of the test rubrics •Instructions
•Structure
•Time allotment
•Scoring method
Characteristics of the input •Format
•Language of input
Characteristics of the expected •Format
response •Language of expected response
Relationship between input and •Reactivity
response •Scope of relationship
•Directness of relationship

22. Framework for Language Task characteristics
Characteristics of •Physical
the setting characteristics
•Participants
•Time of task
Characteristics of •Instructions •Language (L1, L2)
the test rubrics •Channel (aural,visual)
•Specification of procedures and
tasks
Structure •# of parts/tasks
•Salience of parts/tasks
•Sequence of parts/tasks
•Relative importance of
parts/tasks
•# of tasks/items per part
Time allotment
Scoring method •Criteria for correctness
•Procedures of scoring the
response
•Explicitness of criteria and
procedures

23. Framework for Language Task characteristics
Characteristics Format •Channel (aural, visual)
of the Input •Form (language, non-language, both)
•Language (native, target, both)
•Length
•Type (item, prompt)
•Degree of speededness
•Vehicle (live, reproduced, both)
Language Language characteristics
of input •Organizational Characteristics
•Grammatical (vocabulary, syntax,
phonology, graphology)
•Textual (cohesion, rhetorical,
conversational organization)
•Pragmatic characteristics
•Functional (ideational, manipulative,
heuristic, imaginative)
•Sociolinguistic (dialect/variety, register,
naturalness, cultural references and
figurative language)
•Topical characteristics

24. Framework for Language Task characteristics
Characteristics Format •Channel (aural, visual)
of the •Form (language, non-language, both)
Expected •Language (native, target, both)
Response •Length
•Type (item, prompt)
•Degree of speededness
•Vehicle (live, reproduced, both)
Language Language characteristics
of input •Organizational Characteristics
•Grammatical (vocabulary, syntax,
phonology, graphology)
•Textual (cohesion, rhetorical,
conversational organization)
•Pragmatic characteristics
•Functional (ideational, manipulative,
heuristic, imaginative)
•Sociolinguistic (dialect/variety, register,
naturalness, cultural references and
figurative language)
•Topical characteristics

25. Framework for Language Task characteristics
Relationship Reactivity •Reciprocal
between •Non-reciprocal
Input & •adaptive
Response
Scope of •Broad
relationship •Narrow
Directness of •Direct
relationship •Indirect

26. Language Use Tests comprised of
Tasks
♦ Language test: a procedure for eliciting
instances of language use from which
inferences can be made about an
individual’s language ability
– Language test should consist of language use
tasks
– The elemental activities and situations of
language use
– Performance of a set of interrelated language
use tasks
♦ Framework of task characteristics

27. Test Items
♦ Is this writing task similar to tasks that the 2nd language
learner would have to do in real life?
♦ What aspect of second language writing ability is the item
testing?
♦ Does this task require the TT to demonstrate that aspect of
L2 writing ability?

28. EAP ESL Low Intermediate L2 writing class, 20-25 students/class, 40-minute class, 4x/week,
college preparation program w/ 7 levels, level & program exit exam: basic skills writing
proficiency test: argumentative essay.
Task Characteristics TLU Task Test Task
Characteristics of the Setting
o Physical characteristics
e Participants
e Time of task
Characteristics of the test rubrics
o Instructions
o Structure
o Time allotment
o Scoring method
Characteristics of the input
o Format
o Language of input
Characteristics of the expected response
o Format
o Language of expected response
Relationship between input and expected response
w Reactivity
w Scope of relationship
n Directness of relationship

29. Common Core State Standards
For ELA & Literacy
♦ Council of Chief State School Officers
(CCSSO) & National governors
Association (NGA)
– Aligned with college and work expectations
– Standard was included based on the best available evidence that
its mastery was essential for college and career readiness in 21st c,
globally competitive society

30. Language use domain: literate in
a global 21st century world
♦ Close attentive reading to understand and enjoy complex
works of literature
♦ Critical reading for important points
♦ Able to handle large amounts of information
♦ Actively seek wide, deep, thoughtful engagement with
high-quality literary and information texts that builds
knowledge, enlarges experience & broadens worldviews
♦ Reflexively demos cogent reasoning and use of evidence
essential to private deliberation and responsible citizenship
in democracy

31. Writing: K-5: College & Career Readiness
anchor Standards text types and Purposes*
♦ Write arguments to support claims in an analysis of
substantive topics or texts, using valid reasoning and
relevant and sufficient evidence.
♦ Write informative/explanatory texts to examine and
convey complex ideas and information clearly and
accurately through the effective selection, organization,
and analysis of content.
♦ Write narratives to develop real or imagined experiences
or events using effective technique, well-chosen details,
and well-structured event sequences.

32. Anchor Standards for Writing 6-12
Text types and Purposes
♦Write arguments to support claims in an analysis of
substantive topics or texts, using valid reasoning and relevant
and sufficient evidence.
♦Write informative/explanatory texts to examine and convey
complex ideas and information clearly and accurately through
the effective selection, organization, and analysis of content.
♦Write narratives to develop real or imagined experiences or
events using effective technique, well-chosen details, and
well-structured event sequences.

33. College & Career Readiness
Anchor Standards for Language
Conventions of Standard English
♦Demonstrate command of the conventions
of standard English grammar and usage when
writing or speaking.
♦Demonstrate command of the conventions
of standard English capitalization,
punctuation, and spelling when writing.

34. Production & Distribtion of
Writing
♦ Produce clear and coherent writing in which
the development, organization, and style are
appropriate to task, purpose, and audience.
♦ Develop and strengthen writing as needed
by planning, revising, editing, rewriting, or
trying a new approach.
♦ Use technology, including the Internet, to
produce and publish writing and to interact
and collaborate with others.

35. Agenda
♦ Rubrics: facets of L2 writing ability
♦ Comparison of rubrics measure L2 writing
ability
♦ Design of writing task
♦ Wrap-up & evaluation

36. Scoring
♦ Answer key – objective scoring
♦ Rubric - subjective scoring
– Construct
– Rubric
– Partial credit

37. L2 Writing ability
Target Language Use Domain: Academic writing in a
community college:
Language Use Task: write narrative/persuasive essays
Control of content development
Rhetorical control
Grammatical control
Control of register & vocabulary
 Task fulfillment (McNamara, 1996)

38. This exceptionally executed essay takes a clear position and
exceptionally succeeds in expressing a point of view or
telling a story. The thorough development of ideas includes at
least two outstanding points directly related to the topic, and the
examples used, particularly those from personal experience, are rich,
e.g., occasional citation of statistics or reference to personal readings.
The essay is clearly and logically organized with no
digressions; the writer demonstrates skillful command of
cohesive devices. Writer demonstrates ability to write in the
appropriate academic register and demonstrates extensive
range of vocabulary for academic purposes, with few
problems in word choice or usage. A few grammatical errors are
noticeable but rarely do the grammar errors interfere with meaning.
Sentence variety and complexity reflect a sufficient command of
standard written English to ensure reasonable clarity of expression.

39. Holistic Scoring Scale: Criteria for Grading ESL Papers
6
This exceptionally executed essay takes a clear position and exceptionally succeeds in expressing a point of view or telling a story. The thorough
development of ideas includes at least two outstanding points directly related to the topic, and the examples used, particularly those from personal
experience, are rich, e.g., occasional citation of statistics or reference to personal readings. The essay is clearly and logically organized with no
digressions; the writer demonstrates skillful command of cohesive devices. Writer demonstrates ability to write in the appropriate academic register and
demonstrates extensive range of vocabulary for academic purposes, with few problems in word choice or usage. A few grammatical errors are noticeable
but rarely do the grammar errors interfere with meaning. Sentence variety and complexity reflect a sufficient command of standard written English to
ensure reasonable clarity of expression.
5
The focus of this competently executed essay is clear but there may be a few digressions. The writer provides substantial support in the development of
the essay although all examples may not be entirely relevant or appropriate for the topic. The essay is effectively organized, demonstrating systematically
competent use of cohesive devices. The writer demonstrates ability to use a variety of patterns of sentence construction but with some errors. Range of
vocabulary for academic purposes is generally competent, and the writer demonstrates accurate and generally appropriate control of word choice, word
forms and idiomatic expressions for academic writing. Some errors in language use, but errors do not generally interfere with meaning.
4
In this adequately executed essay the writer’s position is clear despite some possible digressions and contradictions. The writer provides adequately
detailed support of two or more points that directly relate to the topic. The essay is generally organized, demonstrating generally accurate and appropriate
use of cohesive devices. The writer demonstrates some sentence variety with simple, compound, and some complex sentences though not always
correctly. The essay may contain frequent errors that may occasionally interfere with meaning. Vocabulary is adequate in range, but there are some
inappropriate or inaccurate word choices and word forms.
3
The essay minimally succeeds in taking a position or relating a narrative with a discernable organizational pattern (introduction, body, conclusion) but
may lack clear focus in development of the central idea. The writer makes an attempt at development although examples are sometimes irrelevant. The
writer makes minimal use of cohesive devices and he/she demonstrates a minimal range of sentence variety and vocabulary, with some inaccurate and/or
inappropriate word choices or inappropriate register. The essay demonstrates minimal control of language, with frequent errors, some of which interfere
with meaning. .
2
The paper represents limited success in writing a persuasive or narrative essay. The writer provides limited development of the topic with one or more
points that directly or indirectly relate to the supporting argument or story. The writing shows limited evidence of organization of ideas (paragraphs are
often one sentence) or accurate or appropriate use of cohesive devices. The range of vocabulary and word choice and the use of academic register is
limited. The control of language is uneven, with frequent errors, many of which obscure meaning. The writing lacks sentence variety.
1
The paper is a failed attempt to write an essay. The writer does not fully develop the topic, lacking related support. There is often no clear organizational
pattern, lacking a clear beginning, middle and end. The writer does not use cohesive devices. The writer demonstrates a narrow range of vocabulary.
There is little evidence of appropriate word choice or usage or academic register. The writer demonstrates little control, with frequent errors of all types.
The errors generally obscure meaning. The writing lacks basic sentence structure and variety. In some cases, the paper may even be written in the
writer’s first language.

40. Template for Holistic Rubrics
Score Description
5 Demonstrates complete understanding of the problem. All requirements
of task are included in response.
4 Demonstrates considerable understanding of the problem. All
requirements of task are included.
3 Demonstrates partial understanding of the problem. Most requirements
of task are included.
2 Demonstrates little understanding of the problem. Many requirements
of task are missing.
1 Demonstrates no understanding of the problem.
0 No response/task not attempted

41. Criteria Beginning 1 Developing 2 Accomplished 3 Exemplary 4 Score
1 Description Description Description Description
reflecting reflecting reflecting reflecting highest
beginning level movement toward achievement of level of performance
of performance mastery level of mastery level of
performance performance
2 Description Description Description Description
reflecting reflecting reflecting reflecting highest
beginning level movement toward achievement of level of performance
of performance mastery level of mastery level of
performance performance
3 Description Description Description Description
reflecting reflecting reflecting reflecting highest
beginning level movement toward achievement of level of performance
of performance mastery level of mastery level of
performance performance
4 Description Description Description Description
reflecting reflecting reflecting reflecting beginning
beginning level movement toward achievement of level of performance
of performance mastery level of mastery level of
performance performance

42. Creating a task to test L2 writing
♦ Describe the population you teach
♦ Define the purpose for an L2 writing
assessment task
♦ Define L2 writing ability: Think of a TLU
domain task in which that language ability
would be demonstrated
♦ Using the framework of task characteristics,
design a task that would require L2 writing
ability to accomplish
♦ Share your task with a partner

43. Self-Assessment
♦ I could create an Assessment Use Argument (conceptual link between
assessment & intended decisions & consequences).
♦ I can define the construct of L2 writing ability.
♦ I can articulate a conceptual framework for designing and evaluating
L2 writing assessment tasks.
♦ I can identify the steps in constructing a language test.
♦ I can evaluate strengths and weakness of some specific L2 writing
items.
♦ I would be able to create a rubric to serve the purposes of an L2
writing assessment in my program.
♦ I practiced item/task evaluation.
♦ I created an L2 writing task that would be suitable for my program.

44. THANK YOU
for your participation!