Assessment and Accountability
in Programs for Adult English Language
Learners:
What Do We Know? What Do We Have in Place? What Do We Need?
Under the Workforce Investment Act (WIA) of 1998, programs that provide instruction to adults must be accountable for the outcomes of that instruction and must use standardized assessment procedures to document outcomes (Workforce Investment Act [WIA] of 1998, Pub. L. No. 105-220, 112 Stat.936, 1998). Now, five years later, what progress has the field of adult ESL instruction made in accomplishing these goals and meeting these requirements?
On Friday, May 16, 2003, at a symposium on Issues and Challenges in Assessment and Accountability for Adult English Language Learners, hosted by the National Center for ESL Literacy Education (NCLE), over 100 teachers, program administrators, researchers, test developers, and policy makers gathered to discuss the field’s vision for ESL program accountability and learner assessment. In addressing the following questions, we were able to review what we have in place and assess what we still need.
What Is Our Vision?Cheryl Keenan, director of the Division of Adult Learning and Literacy in the Office of Vocational and Adult Education, U.S. Department of Education, described the Department’s vision for assessment and accountability.
What Do We Know?
The learner population in adult ESL programs is large and growing.
The population of adults in the United States who are learning English as a second language is large and growing. According to the most recent statistics for program year 2000-2001, 42% of all participants (1,119,589 out of a total of 2,673,391) enrolled in state-administered adult education programs were enrolled in ESL classes (U.S. Department of Education, Office of Vocational and Adult Education, Division of Adult Education and Literacy, 2002a). A large percentage of this population is enrolled in beginning literacy or beginning ESL classes (U.S. Department of Education, Office of Vocational and Adult Education, Division of Adult Education and Literacy, 2002b).
As Mark Kutner (Vice President for Education, American Institutes for Research) described, we will know more about this population after the results of the National Assessment of Adult Literacy (NAAL) are published in 2004.
The NAAL household survey of functional literacy has the following components:
The FAN will assess the ability of adults to decode letters, words, and numbers; comprehend a reading passage; and read with fluency (based on number of words per minute read).
The ALSA will assess the basic reading skills of low-literate English speakers and Spanish speakers who are not yet proficient enough in English to take the NAAL, based on their ability to read information printed on actual products and materials used in everyday life.
The health literacy of adults learning English will also be assessed, based on their ability to understand basic health information. English or Spanish versions of these assessments will be used, depending on the language skills of the individuals involved.
Read more about the NAAL. (PowerPoint)
Programs have varying amounts of experience working with adult English language learners.
Some states have considerable experience with adults learning English as a second language and have put in place effective programs. For example, as Bob Bickerton (Director of Adult Education, Commonwealth of Massachusetts) described, Massachusetts has a large immigrant population and has been running ESL programs for years. Massachusetts has developed adult ESL curriculum and learner performance standards, professional standards, assessment policies and procedures, a data system for documenting continuous improvement, and a performance accountability working group. Different curriculum and learner performance standards are developed for different subgroups of learners (e.g., English language speakers with limited literacy, English language learners, and homeless populations), and resources are devoted to teacher and test administrator training.
In many states (e.g., Arkansas, Nebraska, and Tennessee), the population of English language learners is new and has not been served before. Programs in these states need research-based information and help from states with more experience to guide their development of curriculum frameworks, instructional practice, and assessment systems.
For more information about the population of adults learning English and the programs that serve them, see Adult English language instruction in the 21st century.
Federal and state policies guide the development and use of assessments.
As Carol Van Duzer (NCLE) described, federal policies related to assessment of adult learners are in place, and state policies and procedures are being put in place. The federal government has developed guidelines for states to follow to ensure that instruction is of sufficient quality and duration to help learners meet their goals and that tools and procedures for measuring learner progress are standardized. The National Reporting System (NRS) (U.S. Department of Education, Office of Vocational and Adult Education, Division of Adult Education and Literacy, 2001) was established to ensure that learner outcomes are reported systematically. Within these guidelines, states are selecting assessments that programs can use, establishing timeframes for assessments to be given (either at specific times during the year or after a given number of hours of instruction), and training program staff to administer assessments.
Assessments need to be aligned with learner goals; with content standards, which define what learners need to know and be able to do; and with curriculum and instruction. They need to be valid and reliable, with standardized and consistent procedures for administration and scoring.
Program staff need to know the federal and state policies regarding learner assessment; they also need to participate in decisions that affect the learners in their programs (such as which assessments should be used, with whom, for what purposes, over what time periods, and with what expected outcomes). Program administrators need to make sure that staff are trained to administer the assessments selected. Teachers need to know what is being assessed and why. Test administrators need to follow the guidelines provided with the tests they are using.
Assessments must be appropriate, valid, and reliable.
Many adult ESL programs use a variety of assessment tools to place learners in classes, inform instruction, evaluate learner progress, and report outcomes of instruction. These assessments include standardized tests, materials-based and teacher-made tests, portfolios, projects, and demonstrations.
In selecting appropriate tests, we need to know whether we want to assess learners’ proficiency or achievement. Achievement testing measures what students have learned (what they know and can do) as a result of instruction. Did they learn what was taught? Proficiency testing focuses on what students can do in real-life situations, whether or not it was learned in the classroom (Buck, Byrnes, & Thompson, 1989). The focus in adult ESL education is primarily proficiency.
We know a great deal now about how to assess language learners in valid, reliable, and appropriate ways. As Dorry Kenyon (director, Language Testing Division, Center for Applied Linguistics) described, the National Research Council (with funding from the Office of Vocational and Adult Education) has published a book on performance assessment in adult education, which addresses the following issues relevant to those that we face in adult ESL education (National Research Council, 2002).
Appropriate assessments
In adult ESL education we need to be able to determine the relationship between learner outcomes and the various factors that influence those outcomes, which include curriculum, classroom instruction, and factors outside the educational setting (e.g., learner personality and learning styles, prior education and life experiences, and opportunities to use English outside the program). The NRS requires that programs report learner outcomes in terms of “educational functioning levels,” using valid and reliable assessments. This indicates a need for performance assessments which require test takers to demonstrate their skills and knowledge in a manner that closely resembles a real life situation or setting. Examples of performance assessments include oral or written reports, projects, and demonstrations. Peformance assessments are not easy to develop, administer, score, and validate. For each test developed, we need to know the following:
Valid and reliable assessments
Validity is the degree to which the information gained from an assessment matches the inferences or decisions that programs make about learners, or actions that they take as a result of that information. Whether or not an assessment is valid depends on the uses of the outcomes achieved with it. In adult ESL education, we need to determine what uses we will make of assessment outcomes. Will they be used only to produce reports for the NRS, or will they also be used to diagnose learner needs, track learner achievement, and improve curriculum and instruction?
Reliability is the consistency of the measurement when the testing procedure is repeated on a different population of individuals or groups. Reliability depends first on test developers and distributors, who determine that an assessment is reliable. It also depends on those who create the conditions for testing and administer and score the test. We need to ensure that the assessments selected by states and local programs are used for the purposes, with the populations, and in the ways that they were designed to be used (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999).
Read more about standards for assessments. (PowerPoint)
What Assessments Are Available?Programs for adult English language learners an assessment framework and standardized assessments of reading, writing, and oral English proficiency
EFF Assessment Framework
One tool that we have is not an assessment per se, but rather an assessment framework. As Regie Stites (program manager, SRI International) described, the Equipped for the Future (EFF) assessment framework is aligned with content standards for adult education, which define what is important for adults to know and be able to do. These standards were developed with broad input from learners and practitioners in the field. The assessment framework includes standardized tests for accountability purposes and a variety of other assessments that can be used for informing and improving instruction. A prototype assessment package related to the “Read With Understanding” standard is in development. It will include model standardized tests, technical quality data on the assessments, validity and reliability data, test administrator and scorer training materials, scoring rubrics, scenarios describing how the assessments might be used, and samples of student performance at each of the six EFF performance levels. Practitioners will be able to use the assessments in the package or use them as models to build their own assessments.
CASAS Assessment System
As Linda Taylor (director of assessment development, CASAS) described, the CASAS assessment system is designed to integrate curriculum, assessment, and instruction into one comprehensive system. It includes standardized multiple-choice tests of reading, listening, and math and standardized performance-based measures of writing and speaking. These assessments test learner skills from beginning literacy to advanced ESL levels. The tests are used in ESL, EL civics, citizenship, vocational ESL, employability, and workplace programs. The CASAS skill levels are aligned with the NRS proficiency descriptors, and training for test administrators is required.
Read more about the CASAS assessment system. (PowerPoint)
REEP Writing Assessment
Suzanne Grant (director of the Arlington Education Employment Program, REEP, in Arlington, VA) described a performance-based writing test developed by REEP staff that measures writing gains for adult English learners on six levels and is aligned with NRS (Beginning to Advanced ESL) and SPL levels (SPL 2 to 8). Performance levels are assessed on five dimensions: content, vocabulary, organization, structure, mechanics, and voice. Writing tasks (or prompts) for pre- and post-testing, warm up activities, and a scoring rubric are included. Research carried out by the What Works Literacy Partnership concluded that the assessment is valid and reliable and that learner gains can be measured reliably after a 12-week period of 100 hours of instruction. The assessment is now being tested in intensive and non-intensive programs, closed entry and open entry programs, and in individual and group administrations. REEP teachers were involved in developing the writing prompts, the test administration procedures, scoring rubric, and training materials. This involvement, and use of the assessment, has had a positive impact on instruction, because both teachers and learners are informed about what the learners are able to do in writing.
Read More about the REEP writing assessment. (PowerPoint)
BEST Plus Oral Language Assessment
BEST Plus is a performance-based assessment, an oral interview to assess the functional oral language skills of adults learning English, in the context of interpersonal communication. As Dorry Kenyon (Center for Applied Linguistics) described, BEST Plus assesses learners from Beginning ESL Literacy to High Advanced NRS levels (SPLs 0-10) in the areas of listening comprehension, language complexity, and communication. It was developed by the Center for Applied Linguistics, following rigorous test development procedures and with full involvement of stakeholders in adult ESL education and assessment. It can be used for learner placement, to document and provide a diagnostic report about learner progress, for NRS reporting, and for program evaluation. Both computer-adaptive and print-based versions of the assessment are available, and training for test administrators is required. The score report includes a BEST Plus scale score, which is correlated with the appropriate NRS and SPL levels, as well as diagnostic information about the learner’s performance.
Read more about BEST Plus. (PowerPoint)
What Do We Need?
We have some very good assessments to choose from. However, these are not enough. We need more
Assessments: We still need assessments that will measure English literacy. There are not rigorously researched assessments that demonstrate English language learners’ reading and writing skills.
Resources: In addition to the resources needed to plan and implement effective programs, financial resources are needed so that programs and states can purchase assessments that are appropriate for program goals and learner goals and needs and so that more assessments can be developed.
Professional development: In order for the assessments to be used reliably, professional development is needed so that teachers, testing specialists, and test administrators will understand the assessments they use and the reasons they use them and will administer the assessments appropriately.
Time: Programs need time for assessments to be given and for assessment procedures and instruments to be aligned with instruction.
Why is Assessment Important?
As Cheryl Keenan pointed out, “If we look at the many different subpopulations in adult education and at the subpopulations within the ESL portion of adult education, we see that not all adult learners are in programs for the same reasons, or for the same outcomes, or at the same places along the way. It is our responsibility to get them to the next step, whatever that next step is for them.” Valid and reliable assessments that adult ESL professionals understand and can use effectively in their programs are a critical part of this responsibility.
References
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Author.
Buck, K., Byrnes, H., & Thompson, I. (Eds.) (1989). The ACTFL oral proficiency interview tester training manual. Yonkers, NY: American Council on the Teaching of Foreign Languages.
National Research Council. (2002). Performance assessments for adult education: Exploring the measurement issues: Report of a workshop. Washington, DC: National Academy Press.
U.S. Department of Education, Office of Vocational and Adult Education, Division of Adult Education and Literacy. (2001). Measures and methods for the National Reporting System for adult education: Implementation guidelines. Washington, DC: Author.
U.S. Department of Education, Office of Vocational and Adult Education, Division of Adult Education and Literacy. (2002a). State-administered adult education program 2001 enrollment. Washington, DC: Author.
U.S. Department of Education, Office of Vocational and Adult Education, Division of Adult Education and Literacy. (2002b). State-administered adult education program enrollment by educational functioning level, ethnicity and sex, program year 2000-2001. Washington, DC: Author.