Literacy and Numeracy for Adults Assessment Tool

When a learner completes a reading or numeracy assessment using the Adult Literacy and Numeracy Assessment Tool, an individual learner report is generated. The learner can choose to see this report directly after they complete an assessment online. An Educator can generate this report for any learner that’s assigned to them.

The Individual learner report consists of:

• a scatter plot showing the relative difficulty (the progression step level) of each question and whether the learner got each question right or wrong
• a list detailing the context and intent of each question.

The following video discusses how to read and interpret the scatter plots of learners' answers generated by the Assessment Tool for Individual Learner Reports.

You will need to download the full sample reports (PDFs) below, as the video and text both refer to them.

Key points:

• Black dots are questions answered correctly; white dots are incorrect answers.
• The questions are ‘mapped’ to the Learning Progression steps (indicated by the number of fronds on the koru) so the vertical position of the dots indicates the question’s relative level of difficulty.
• An adaptive assessment hones in on the learner’s achievement level by generating each question based on the result of the previous one. If a learner answers a question correctly, the next question is likely to be at a level at or above the correctly answered question. Conversely, the Tool will likely decrease the level slightly for a question following an incorrect answer.
• It’s important to explain the adaptive nature of the assessments to your learners before they do an assessment. They need to know that it will likely feel like the questions are getting harder, and that they are not expected to know all of the answers. The purpose of the assessment is to find out what they do and don’t know. View the videos about briefing learners to take an assessment for more information on preparing your learners to sit assessments.
• The scatter plot doesn’t display the question numbers sequentially from left to right. They are placed on the scatter plot according to the progression the question assessed.
• The general numeracy assessment assesses all three number strategy progressions (additive, multiplicative, and proportional reasoning strategies), the measurement progression, and the three number knowledge progressions (number sequence, place value, and number facts). The scatter plot indicates whether the question assesses strategies or knowledge, but doesn’t have enough detail to tell which strategy or knowledge progression a question relates to. (The question detail section of the report provides these details.)

Scale scores and step progression

Learner placement on the scales
Note in the Learner Assessment Report that along-side the progression step scale (koru), there is an equal interval numerical scale from 0 to 1000. The number of questions the learner gets correct, (their raw score) is weighted against the questions’ difficulty and converted by the Tool to a scale score.

So, as well as the assessment questions being located on the scale, the learner is located on the scale based on the questions answered correctly.

The scale score for this numeracy learner report shown is 478 with a margin of error of 28 points, either side of 478. This means that, based on this ‘sample’ of the learner’s knowledge, we are reasonably confident that the learners ‘true’ level of achievement is between 450 and 506. This achievement level range is displayed as the grey band on the learner report.

Learners are expected to get about 70% of the questions located at their achievement level correct. They are expected to get much less than 70% of the questions located above their achievement level correct, and much more than 70% of the questions located below their level correct.

A detailed explanation of the in placement of the learner in relation to the questions answered is in the pdf Understand the Dots, below.

Interpreting learner achievement score
As an educator planning how to address a learner’s literacy and numeracy needs, the most important thing about the scale score is how it aligns with the Adult Learning Progressions. It is the Progression step descriptions that tell us what the learner is likely to be able to do at each step so we can compare their likely capabilities with course and vocational demands.

For details about how the scale scores align with the Progression steps read the PDF Assessment Tool - Boundary Points, below. Knowing the learner’s achievement level in relation to the Progressions means we have a general idea of their capabilities. By comparing learners’ achievement levels to vocational and course literacy and numeracy demands, you can start to plan embedded teaching strategies to help learners meet the demands.

Planning 'what to do'

• 
  Are there calculations with decimal numbers in your course? A step 3 learner is likely to struggle to do these with understanding. So perhaps you plan some activities to build learners estimation strategies for multiplying decimals like 1.8 x 2.1. Or perhaps an activity that compares the area of a 2 m x 1 m space with a 2 m x 0.5 m space to develop a sense of what happens when we multiply by decimals less than 1. The desired outcome is that learners can decide whether a calculated answer using decimal numbers looks reasonable.
• Are there specialised words with more than one syllable in your course? A step 3 learner is going to struggle to read these with understanding. Perhaps you plan a Pair definition activity or get learners to predict and define new words in a text to get them using strategies for building vocabulary.

Use the the Progression Strand Charts and the Learner step and capability profiles to get a general idea about learners’ capabilities at each step

Assessment Question Details

Comparing a learner’s step level to step profiles is a good starting place for determining their probable literacy and numeracy capabilities. But as with all generic profiles, the step profiles provide only a generalised summary of likely capabilities at a particular step.

The Learner Reports provide opportunity to derive a more personalised profile for a learner. The images below show the Assessment Questions detail section from a Reading assessment along with the scatter plot.

Download the PDF files of the two pages: Highlighted Reading Report p1 and Highlighted Reading Report p2 below.

Analysis of report

Firstly, the scatter plot tells us that the learner’s overall achievement level is Step 3 of the Read with Understanding strand of the Adult Learning Progressions. The learner consistently answered Vocabulary and Text Features questions located at Step 3 correctly. Likewise, reading comprehension questions at the lower end of Step 3 were answered correctly. The learner was only able to correctly answer one question which required reading critically.

Next, note that the Assessment Questions section identifies the context, assessment intent and text type for each question, as well as the progression and result (which are also evident in the scatter plot).

An educator has gone through this report and used highlighters to colour code, according to progression, the intent of each question the learner got wrong. For example, the key intent of Question 3 was locate and link information in a flowchart, and it was assessing reading comprehension so the intent is highlighted pink for Comprehension.

Once the intents of all the incorrectly answered questions have been highlighted, we look for similarities or themes in the errors. In this example at least five of the missed questions were about locating synonymous information in texts or diagrams. The learner also had difficulty identifying key points in a text.

Possible next steps

Based on the above information, teaching and learning strategies that could be embedded in vocational teaching to support this learner could include

• Ensuring language and words used in class and in text are consistent
• Explicitly linking synonymous words and ideas in class
• Showing learners how to use a thesaurus
• Incorporating teaching and learning sequences from the Learning Progression resources such as Identifying Main Ideas or Selecting Relevant Information

This short video clip demonstrates how to find appropriate activities under the Learning Progressions tab of this website.

Finding T&L sequence

Numeracy assessment question details
The video clip below demonstrates a process for relatively quickly analysing the question detail section of the numeracy assessment examined the previous section.

Numeracy question detail

Key impressions
Going through the question detail for this learner confirmed that the learner does not understand decimal numbers well. This is in line with the expected profile of a learner achieving at step 3 of the numeracy progressions.

The learner knows some whole number multiplication/division basic facts and can do some large whole number calculatons.

The learner appears to have a pretty good sense of measure and correctly converted between digital and analogue time.

The learner is weak in proportional reasoning.

These are general impressions based on the results. If you have specific numeracy demands in your course, you may use the question detail section to look for learners’ results for quite specific intents. Avoid placing too much emphasis on the results of any single question.

Possible next steps
A shallow understanding of decimal numbers and low proportional reasoning skills are common issues for any group of learners who achieve below step 5 on an assessment. So teaching and learning activities that build learners' understanding of decimals and proportional thinking are likely to benefit this learner and many of his fellow learners.

Teaching and learning strategies that could be embedded in vocational teaching to support this learner could include:

• Exploring strategies for solving percentage and multiplication problems using proportional reasoning. For example if we know 10%, how can we find 5% or if we know 3 x 15 how can we use this to find 6 x 15?
• Including visual representations of decimal numbers in calculation activities. 
• Using tables and boxes to solve rate or scaling problems. 
• Reminding learners the check if an answer looks reasonable when performing calculations, and exploring estimation strategies for checking.

Scatter plot section

The scatter plot section of the report allows you to

• see at a glance the level of questions learners are able to answer successfully
• see the achievement level range for the learner which can be compared to the Learning Progression and step profile descriptions to give an indication of learners' abilities
• compare the learner's ability in one progression relative to the other assessed progression
• compare the learners current abilities with the identified demands of your course or vocation

Question detail section

The Question Detail section of the report allows you to:

• identify commonalities in the intents of questions the learner answered incorrectly or correctly
• identify areas of strength for the learner
• see more detail about which strategy or knowledge progression a question relates to
• check the learner's response for specific questions whose intent matches a course demand

If you have large groups of learners it's unlikely that you will read individual learner reports for every learner. However, the information is available should you wish to view it for any specific learner. The data in the individual learner reports enables educators to develop individual learning plans for learners where appropriate or look for common knowledge and strategy gaps to address with groups of learners.