Checkout My Domian

Formative Design

Research Plan

This Spencer Foundation Small Research Grant will utilize formative experiments (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003; Reinking & Bradley, 2008) to ask what could be from a mixed-methods perspective, instead of what is betterorbestfrom a quantitative perspective, or what is from a qualitative perspective. This project will use four phases of iterative design to meet our goals: 1:) collect baseline data, 2:) Develop Intervention. 3:) Implemenet Intervention, and 4:) Retrospective Analysis

Research Goals

We begin our study with the pedagogical goals of improving the digital literacy skills of the participants while also supporting the quality of the local web. We will be actively desigining our interventions to meet these goals and tweaking variables that we believe enhance these outcomes. Specifically we seek to know if:

Recruitment Plan

In year one we will work with participating libraries (see attached appendices) to recruit participants. In year one we will have 30 participants at two libraries for a total of 60 patrons. One library is an urban city and the other in a suburban town. Particpants will be recruited from existing tech programs and participants who attend adult literacy programs. This approach should hopefully lead to a diverse pool.

In year two the recruitment plan will rely heavily on year one participants to encourage people in their existing local networks to join. We will also be doing outreach through the libraries. We will not have a target population in year two. We will allow up to 100 participants at each library and rely on year one participants to serve as leads.

Phase 1:Baseline Data

The project will begin with a brief snapshot of the digital landscape of communities and partnerships. First, after seeking IRB proposal, local co,mmunity stakeholders and activist will be recruited to complete a questionaire about the state of the local web. I will also seek out interviews with community leaders for a semi-structured interview abotu the effect the web has on the local community

Social media will be examined to see if there are popluar local hashtags. No effort will be made to join or measure any private town group given the expectation of privacy. All data from social media will only be reported in the aggregate. Only artifacts, like post to social media, will be included for participants with explicit permission. Any social media artifacts, beyond aggregate numbers collected from non-participants, even though the data is openly avaialble, will be destroyed.

Baseline data for participants will be conducted in one to one interviews about how they use the web. Semi structured interview questions will ask about past social media experience, coputer experience, website builidng, social media, and htm ability.

Phase 1: Data Analysis

In order to understand the local context for where we will iterate on our design the quantitative and qualitative data collected prior to the study will be triangulated using thematic network analysis (attride-Sterling, 2001). This data reduction technique allows for the collapsing of codes into patterns and then themes.

As active designers and participants we will also draw heavily on our personal experience in teaching in open enviornments and working with learners with a variety of technical skills. We will also design training events based on existing professional development models (Hobbs & Coio, 2018; indieweb.org)

We will then use persona design (Klein, 2015) as a method of case study analysis. In persona design you create fictional representations of users. You write background stories and predict their needs. By empathisizing with user needs we will be able to design interventions for our participants.

Phase 1: Expected Outcomes

By the end of phase 1 we will have an understanding of the users and local communities.

Phase 2: Develop Interventions

In year' one and two phase 2 will begin with iterating on the interventions. Dr. McVerry and the graduate student will use the persona and community data to develop the first face to face to lessons. Lesson design will begin with pedagogical goals.

The lesson drafts will then be shared with a panel of experts including technology and pedagogy experts. This will help ensure content validity and provide feasiblity checks. The sessions will be first delivered in year one to a smaller population and then using data collected in the intervention they will be revised and repeated in year two.

Phase 2 Data Analysis

Data analysis on phase two will focus on feasability and ecological validity. Field notes and reflections from the designers will be analyzed as well as interview data with the expert panel.

Phase 2: Expected Outcomes

A draft of all learning activities that should help enhance our pedagogical goal

Phase 3: Intervention Implementation

In years one and two the implementation of interventions will follow roughly the same schedule

The first training will focus on definig what the local web shoudl look like. Sessions may explore focus groups on social media in town and the role of tracking on the web. We will also include technical sessions on launching a website. We will also develop a Code of Conduct together.

The second training will focus on, "Building the Most Credible Self." Sessiosn may include topics on building about me pages, personal branding, and identity formation. Technical sessions will be determined by participant observation in between the two training sessions.

Following each training an option set of online activities will be conducted. These will also be designed furing phase two of each year but we will iterate based on data collected in-situ.

Phase 3 Data Analysis

To measure the progress of our pedagogical goals both qualititative and quantitative data will be collected throughout the study.

Participant HTMl ability will be used as a proxy measure for overall digital literacies skills. However patrons of th program will not take a "build a website" or HTMl test. Instead blog post and/or website pages will be collected as set intervals. HTML parsers, which can seperate pieces of websites based on specific tags, will be used to quantify the use of tags and to track changes to wesbites over time. We will examine CSS stylesheets as well to look for growth in digital literacy skills.

While writing HTML is a proxy, and this will not encapsualte all that is involved in digital literacy, it is the foundation of writing on the web. Furthermore we also believe that analyzing participant writing for growth over time provides greater ecological validity than current measures of digital literacy skills.

To track network effect a variety of tools will be used. First an RSS feed of all user websites and comments will be collected. Second local hashtags will be monitored for frequency of growth. Finally webmentions.io will be used to capture whenever a website recieves a mention across the web. A webmention is a link between two websites where website A let's website B know, "I linked to your site," and then website A can display that mention.

While these webmentions, will occur in the open and be available data for anyone to crawl webmentions from websites and people outside of the study will only be reported in the aggregate.

In terms of qualitative data we will also take extensive field notes and write weekly reflections as participant observers in the study. We will also conduct content analysis of users post to identify explicit mentiosn of their own improving digital literacy skills or the overall health of the local web.

Given the formative nature of this experiment we will actively be iterating on the assessment and both the qualitative and quantitive data to bring us closer to our pedagogical goal.

Phase 3:Expected Outcomes

After phase three in year one twenty particpants, ten at each location, will have undergone the training and will blog throughout the year. If they attend three of the online sessions and the second training participants can check out their domain for a second year.

After phase three in year two twenty additional pariticipants will complete the training. This will allow us to measure the impact of the network and the digital literacy skills of participants at numerous timepoints.

Retrospective Analysis

According to Creswell (2002), triangulation “is the process of corroborating evidence from different individuals,... types of data,...or methods of data collection” (p. 280). FurthermoreTashakkori and Teddlie (1998) called for “theoretical triangulation” (p. 41), which will align with us developing a theory and units of analysis in the perezhivane of the web. We believe the data collected and analyzed in years one and two of the study will allow us to triangulate our findings during the phase 4 retrospective analysis.

We will conducting an analysis of all the data to push our theoretical understandings forward so we can make recommendations for making the public library the keystone of the local web. By employing “a variant of Glaser and Strauss’s (1967) constant comparative method” (p. 38) we will identify key moemnts that illustrate how the intevention either enhanced or inhibited the pedagogical goal.