By Jen Vanek
We are building new features for SkillBlox as part of our partnership with the American Institutes for Research (AIR) in IES-funded design and development research. Our qualitative approach is guided by Design Based Research (DBR) (Brown, 1992; McKenney & Reeves, 2018; Sandoval, 2014), through which we’re able to learn from end-users what they want to see in the new features, especially in the user experience and user interface (UX/UI). Just as important, DBR gives us a process to elucidate findings about teaching and learning more broadly by establishing, revisiting, and updating conjectures. A conjecture is the research team’s best guess as to how elements of design impact teaching and learning (Sandoval, 2014). Ultimately, the goal of the research is to deliver useful and relevant SkillBlox features, continuously refine our conjectures, and, along the way, answer our research questions about how SkillBlox can support expanded TSTM-aligned instruction.
- What SkillBlox features best support the use of Open Education Resources (OER) in TSTM-aligned instruction?
- What level of description and tagging of resources do teachers identify as helpful?
- What mechanisms for prioritization and representation of OER streamline the OER search and selection process?
- What user interface and user experience best promote use of SkillBlox?
This work is happening in three distinct phases, which are in various stages of completion at the time of writing: 1) exploratory work, 2) iterative feature development and testing, and 3) a usability and feasibility study.
Phase 1: Exploratory Work.
Our exploratory research, concluded in late 2022, included a review of prior research on the use of OER in various education settings, previous SkillBlox development and accompanying data collection, and TSTM Implementation research led by AIR. Through these sources, we gleaned some guiding information:
Prior research on the use of OER in adult basic education is thin (Newman et al., 2015), but the literature does articulate both barriers to OER use and strategies that support it.
| Barriers to OER Use
Admiraal, 2022; Belikov & Bodily, 2016; Hilton et al., 2010; Taylor & Taylor, 2018; Wolfenden et al., 2019 |
Strategies to Support OER Use
Marin et al., 2022; UNESCO, 2019; OER: Adult Learning and Sustainability (2022) |
| Perceived lack of quality | Infrastructure for organizing |
| Lack of accessibility | Professional development to inform about use |
| Low relevance of content | Access to repositories of relevant content to ease discovery |
| Challenges with reusability (e.g., time, technical capacity to update, licensing limitations) | Technical assistance for locating, evaluating, and customizing OER |
Past development of SkillBlox, funded by the World Education Services Mariam Assefa Fund (no connection to World Education), showed us that users: need to have hubs to support SkillBlox sharing, want to know the level of the activity and the context for how it is used, and prioritize being able to select the language (if the resource that has multiple language options).
In the TSTM implementation research led by AIR, teachers noted that they need access to granular instructional resources and activities aligned to the different TSTM skills that they could then adapt for their own use. However, they also reported having limited time and capacity for this kind of work. To partially address this problem, they welcomed opportunities to collaborate with others.
Collectively, these observations informed the development of our initial DBR conjecture:
To increase TSTM-aligned instruction, teachers need easy access to quality, free and open resources. This is possible if teachers make use of new SkillBlox features that support searching for, evaluating, selecting and organizing OER, and sharing their SkillBlox with other teachers. Such use is more likely if teachers have access to and follow guidance about how they might adapt activities to meet their needs and use them in a way that supports TSTM development.
Phase 2: Prototype and Iterative Feature Development and Testing.
This phase is happening in two stages: prototype testing and iterative building and testing. Our prototype testing began in early 2023. Over several months, we gained insight about our initial designs through focus group meetings, questionnaires, and participant engagement in SkillBlox. Our focus group discussions, held as three meetings with six volunteer educators, were especially fruitful. We employed a qualitative coding process to examine notes and transcripts from those meetings, and triangulated our observations with what we learned in our exploratory work, especially the implementation study introduced above. We have also gleaned information on prototype design from our EdTech Maker Space data analysis. Collectively, these data and analyses supported our original conjecture and resulted in a prioritized features list that responds to both the research questions and our design goals.
At the time of writing, we are just beginning the second part of this phase, working with a software developer to build these prioritized features.
Our participants are a volunteer group of 10 educators known to have some familiarity with SkillBlox, TSTM, or both. These participants will use SkillBlox to complete assigned activities and then debrief on that experience in focus group discussions, one for each of the three feature sets in development. Data sources include a record of their use of SkillBlox and focus group transcripts and notes. After each focus group, we’ll meet with the developer to discuss updates required to better align the features with the UX/UI preferences of the end users. For our larger goal of answering our research questions and refining our conjecture, we will revisit data, including research memos written after our calls with the developer.
At the end of this cycle of iterative design and testing, we will have a complete set of newly developed features and training materials needed to prepare teachers for using SkillBlox in TSTM-aligned instruction.
Phase 3: Usability and Feasibility (U/F) Study
Starting in June 2024 we will pilot these new features and the training with 10 teachers and their learners. The U/F study will be the first opportunity to observe real use of the application’s new feature sets and test our onboarding process and resources. Again, because we are working with DBR, we will be able to not only make improvements to both the application and resources, but we will gain insight into teachers’ use of OER and SkillBlox to support teaching and learning.
Future Research
The IES-funded study will conclude with an experimental pilot study of the new SkillBlox features and PD activities. The research, conducted in 2 – 4 states beginning in fall of 2024, will investigate whether the PD and other supports are sufficient for teachers using TSTM-SkillBlox, and whether teachers find the new features and content of SkillBlox easy to use and adapt in their instruction. The pilot will also identify remaining challenges teachers experience in implementing TSTM-aligned instruction using the application and elucidate needs teachers have for additional support to be able to use TSTM-SkillBlox easily.
References
Admiraal, W. (2022). A typology of educators using open educational resources for teaching. International Journal on Studies in Education, 4(1), Article 1. https://doi.org/10.46328/ijonse.60
Belikov, O. M., & Bodily, R. (2016). Incentives and barriers to OER adoption: A qualitative analysis of faculty perceptions. Open Praxis, 8(3), Article 3. https://doi.org/10.5944/openpraxis.8.3.308
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141–178.
Hilton III, J., Wiley, D., Stein, J., & Johnson, A. (2010). The four ‘R’s of openness and ALMS analysis: Frameworks for open educational resources. Open Learning: The Journal of Open, Distance and e-Learning, 25(1), 37–44. https://doi.org/10.1080/02680510903482132
Marín, V. I., & Villar-Onrubia, D. (2022). Online infrastructures for open educational resources. In Handbook of Open, Distance and Digital Education (pp. 1–20). Springer Singapore. https://doi.org/10.1007/978-981-19-0351-9_18-1
McKenney, S., & Reeves, T. C. (2018). Conducting educational design research. Routledge.
Newman, A., Rosbash, T., & Sarkisian, L. (2015). Learning for life: The opportunity for technology to transform adult education. Boston, MA: Tyton Partners. http://tytonpartners.com/tyton-wp/wp-content/uploads/2015/03/Learning-for-Life_The-Oppty-for-Tech-to-Transform-Adult-Education_March-20151.pdf.
OER: Adult Learning and Sustainability. (2022, March 21). EPALE – European Commission. https://epale.ec.europa.eu/en/blog/oer-adult-learning-and-sustainability
Promising practices and lessons learned from early implementation of the TSTM framework (2023). American Institutes for Research. Create Adult Skills Network. https://createadultskills.org/index.php/resource/promising-practices-and-lessons-learned-early-implementation-tstm-framework
Sandoval, W. (2014). Conjecture mapping: An approach to systematic educational design research. Journal of the Learning Sciences, 23(1), 18–36. https://doi.org/10.1080/10508406.2013.778204
Taylor, C., & Taylor, M. W. (2018). I’m never doing this again!: Identifying and solving faculty challenges in adoption of open educational resources. Online Journal of Distance Learning Administration, 21(2).
United Nations Educational, Scientific and Cultural Organization (UNESCO). (2019). Recommendation on Open Educational Resources (OER) | UNESCO. https://www.unesco.org/en/legal-affairs/recommendation-open-educational-resources-oer
Wolfenden, F., & Adinolfi, L. (2019). An exploration of agency in the localization of open educational resources for teacher development. Learning, Media and Technology, 44(3), 327–344. https://doi.org/10.1080/17439884.2019.1628046
