A certified nursing assistant readying for a future with more electronic health records. A disabled father improving his digital skills toward a dream of starting an online business (and helping his daughter with computer-based homework). A Somali immigrant using his certificates as a credential for a front-line service career in healthcare. What do these people have in common? They are all using the NorthStar Digital Literacy Assessment to learn and prove that they have gained the computer skills required to access employment and economic opportunity.
The Northstar Digital Literacy Assessment is a suite of free online modules supported by the Minnesota Literacy Council, which test proficiency with essential digital skills. The modules are designed so that end-users who lack strong literacy or prior education can still complete them and demonstrate computer skills. The whole assessment is undergoing an update funded by the Employment Technology Fund and the Minnesota Department of Employment and Economic Development; many of the changes made are described in our Tech Tips blog post from last month.
Because Northstar is a tool that has the potential to improve employment opportunity for lower-skilled, low-income adults in the United States, the EdTech Center field tested the updated Northstar Assessment with funding from Walmart in partnership with the Employment Technology Fund. The field testing happened across the country and was supported by feedback from multiple sources:
- Online questionnaires were offered to facilitators who observed learners using the modules at several stages in the development process; learner end-users had the opportunity to complete a questionnaire when each module neared completion.
- Site visits occurred in key locations in Minneapolis/St. Paul, MN and Providence, RI, including adult learning and workforce development programs that have deeply integrated Northstar into their programs and outcomes. At site visits, we conducted interviews or held focus groups, and watched learners using Northstar assessments.
A defining characteristic of the research was a structured and iterative cycle of development, feedback, and revision that supported the development of a quality online assessment and illustrated the ways that Northstar has already had positive impact on the lives of many end users.
Key Findings
Test takers who have limited English language and literacy proficiency fare well in the new Northstar modules because test items employ simple language and creatively leverage visual media and audio to support comprehension.
Data from our site visits and questionnaires suggested that the language level and multimedia affordances supported end users with limited English language literacy. The questionnaire data showed that respondents felt that aligning visual, audio, and textual cues provide a multifaceted approach to support the assessment and user experience. One student said, “The pictures are very clear and help me understand” (Guadalupe Alternative Programs focus group participant). Another student from the same program said that it is “….easier to know what to do on the new module.”
A volunteer facilitator supporting Northstar in a computer lab at Goodwill Easter Seals (GWES) in St. Paul, MN said, “There are no vague statements, which is good. Also, the graphics are actually up-to-date photos of what different hardware are like. Before they were more like caricatures. The old illustrations were not as effective” (Zach, GWES volunteer).
Both feedback data from the questionnaire and the interviews indicated that the use of audio is an important supportive element of the modules. “I even saw a learner repeat a question and then close their eyes to really listen to a question. They then went on to get the correct answer, so it is really neat to see audio in practice” (online questionnaire response).
Changes to navigation support ease of use for everyone.
Results from feedback questionnaires showed that respondents appreciated the layout of the website’s pages and the clear navigation cues. This was affirmed in an interview with a volunteer, “The pages are clearly laid-out; the way the questions are displayed seems crisper. Navigation is straightforward and consistent across the modules” (Zach, GWES volunteer).
One important change now allows learners to go back and review previously answered questions before submitting them as final answers. “The feature of being able to go back and change previous answers is really effective. I always point it out even though it’s in the orientation” (Julie, GWES volunteer). Questionnaire data showed many instances of positive feedback regarding this shift, including “Overall, I preferred the newer modules, particularly the fact that a participant could go back and review before submitting the test.”
“The pages are clearly laid-out; the way the questions are displayed seems crisper. Navigation is straightforward and consistent across the modules”
This version of the assessment does a good job limiting abstract questions, making the actions required to answer the questions more aligned with everyday computer use.
Northstar has done this in two key ways. First, the contexts created for different questions mirror popular contexts represented in daily internet use, including use of mobile devices, as shown in this example.
In another example, the email module uses a generic email environment that looks and functions quite similarly to Gmail. Questionnaire responses confirmed that this choice was the right one, “The Google browser and Gmail are by far the most frequently used browser and email service on the internet. As a result, it’s important the assessment emulates Google services.”
In another example showing a real context, Northstar placed ‘Netflix phishing scam’ in an email wrapper to make it look like an email a user might actually have received.
Finally, the questions in the new assessment modules can be answered in multiple ways. Project director Tom Cytron-Hysom said, “One of the biggest complaints was that because of the use of flash, in the initial version, there was only one way to answer each question. So in the redesign we invested significant resources into integrating multiple means by which to answer each question.” This was a well-received change. “Good to have keyboard shortcuts and click pathways both as acceptable ways to accomplish question tasks, for example, click the B for bold of use Ctrl + B” (online questionnaire response).
This strategy has also potentially mitigated a problem created by the “one way to answer each question” problem. In the past, it was easier for tutors to teach to the test by showing the correct way to answer a question. This had the potential to undermine the validity of the assessment. Equally concerning, it potentially created a scenario where learners might only receive instruction on only one way to accomplish a task, the way that happened to be the correct answer. Now, because it is less possible to teach to the test, learners are more likely to learn a range of means by which to accomplish computer tasks.
Success Stories
Perhaps the best way to understand the impact of the new Northstar Digital Literacy Assessment is through the stories told by end users. Along the way, we have gathered several.
Miguel, Providence, RI public housing computer lab
Miguel is disabled and cannot work because of his disability. He wants to start an internet business in the future. He only knew computer basics when he started the class. Now he is working on module 4 and has done very well in the tests for modules 1 – 3.
I am confident and comfortable using a computer and am learning so much. Some things I never thought about, capital letters, why use one shift one side and not the other side. I have a computer at home so I go home and practice every single day. I see the computer in a different way now. Before I only saw it as a way to use Facebook and YouTube, and now I know how to use email, and make it work for me. I plan to start an online business in the near future. Also now I do my travel bookings and pay my bills all online and with confidence!
“Before I only saw it as a way to use Facebook and YouTube, and now I know how to use email, and make it work for me. I plan to start an online business in the near future.”
My daughter is 13 years old, she saw me using computer and now I have been teaching her at home too. She did not know about different programs on the computer. I now help with her schoolwork on the computer too. I am the computer expert at home now! I love it!
Yesika
Yesika works as a Certified Nursing Assistant (CNA) at a healthcare facility and attends digital literacy classes between shifts. She said, “We are just starting to change to electronic records at my job. I realized I needed to improve my computer skills or I would be left behind.” She plans to go back to school, and this new knowledge and comfort with computers is going to be a big help.
I mostly use my phone to use the internet and I blame that for not having developed computer skills before. Before I couldn’t do basic things like attach and open documents in email. Now I can also help my son with his homework. We go to the public library and I can help him use the computers to do research for his homework.
Yesika hopes to get her laptop and free internet access for a year after she passes all four Northstar modules, an incentive offered by the Providence Housing Authority. She says that will be a big help to her and her family.
Guled – Goodwill Easter Seals computer lab in St. Paul, MN
In the summer of 2018, Guled began working with digital skills tutors at Goodwill Easter Seals in St. Paul, a new Minnesota Literacy Council instructional site. After several sessions working with tutors, Guled has passed his Windows Operating System assessment. “It feels great to get the certificate! “I didn’t know anything about computers, except email. I’m happy to have this!”
He has told many of his friends from Somalia about Northstar because he wants them to feel the same success he has felt after passing the assessments. He plans to leverage the certificates he’s earned as he applies for frontline service jobs, possibly in a hospital and said that he thinks he’ll have more success now because he can use his new skills to write a resume and organize his online job search. “This is a success. This is the thing I needed. [Holding up certificate]. This is my second certificate. When I get ten of them, I can show them and get a new job, like at a front desk that needs to use a computer. They always ask about experience with desktop. So this thing is my lifeline to my success.”
“This is my second certificate. When I get ten of them, I can show them and get a new job, like at a front desk that needs to use a computer. They always ask about experience with desktop. So this thing is my lifeline to my success.”
The Employment Technology Fund is the first social impact fund to invest in technology solutions that improve employment opportunity for lower-skilled, lower-income adults in the United States. By shaping the future of work, ETF and its portfolio companies will close the employment opportunity gap facing 103 million adults who lack the skills, training and opportunities to access family-sustaining jobs and meaningful employment, including women, minorities, and immigrants. Since launching in August 2017, the Fund has made seven investments including: Cell-Ed, NorthStar,
Thank you to Walmart:
World Education’s Technology Testing for Adult Learning & Employment initiative was supported by grants from Walmart and the Walmart Foundation. The opinions and ideas expressed in this report are those of the authors alone and do not necessarily reflect the opinions of Walmart and Walmart Foundation.
