Quality Matters Recertification

1. If 15% or more of the course has been revised
2. If the certification has expired/is close to expiring

Before a course can be submitted for recertification, the course alignment document must be reviewed and updated to the latest template, the course contents must be reviewed to ensure it complies with QM standards, and the QM Worksheet must be filled out correctly.

Alignment

Quality Matters

All standards

Alignment Verification

• Verify the alignment document is in the newest template (available on Google Drive Template Gallery). This document has been reviewed by QM and contains the necessary items for the QM review such as:
  • Course Level Objectives
  • Unit Level Objectives
  • Standards Link
  • Instructional Materials Used
  • Learning Activities
  • Unit Assessments
  • Tools Utilized
• Check for alignment and measurability of Unit- and Course-Level Objectives

Content Verification

A course must meet the minimum threshold according to the QM Rubric and Standards (85% for most standards), or it will be sent back for an amendment period, which gives us 14 weeks to make any required revisions. QM reviewers utilize a thorough and in-depth rubric when reviewing courses, so it is not feasible for us to check a course on each of the QM standards prior to submission. As such, the pre-submission content check focuses on the following key areas:

• Ensure each assignment has either an answer key or a rubric
• Check for alignment and measurability of Unit- and Course-Level Objectives
• Look at content for potential issues in terms of currency / relevancy and accessibility / inclusion
• Run Link Checker to look for broken links (CourseArc)

Ensure each assignment has either an answer key or a rubric

For each assignment that students are required to submit that is not a quiz or test (e.g., discussion boards, written assignments, projects, etc.), there should be a student-facing rubric or checklist that outlines the requirements of the assignment and how students will be graded. While a simple checklist outlining the things a student needs to do to receive full credit on an assignment has been sufficient for QM in the past, preferably, assignments will feature more descriptive rubrics that outline the various levels of success possible on the assignment and the criteria for achieving each level.

To check an assignment, open the assignment page. A rubric should be attached as a link at the bottom of the page. While rubrics can be uploaded as a separate document, building them directly in Brightspace provides a better workflow for instructors.

A list of rubrics present in the course can be viewed by accessing the Course Tools dropdown menu from the horizontal menu at the top of the course page and then selecting Rubrics. Note: Just because a rubric is present in this list, it does not necessarily mean it’s in use in the course. If the rubric is linked to a course assignment, a chain icon will appear after its name.

For more information about rubrics, consult the style guide entry on rubrics. Style Guide: Brightspace: Rubrics

Check for alignment and measurability of unit and course level objectives

Using the alignment chart, review the unit and course level objectives.

To determine their measurability, review the verb(s) in the objective. This verb should describe a measurable action taken by the student that a teacher could assess. A framework like Bloom’s Taxonomy can be helpful in reviewing the measurability of verbs. Be sure that the objectives do not rely on a hard-to-measure formulation like “Demonstrate an understanding…”

In this context, alignment refers to several things. First, consider the alignment between the objectives and any associated assessments to ensure that the objectives are actually being assessed. Any objective listed on the alignment document will need to be aligned with an assessment. Additionally, consider the appropriateness of the assessment: can the stated objective actually be measured with the type of assessment listed? For example, QM has previously given Michigan Virtual feedback that objectives using verbs like “explain” and “describe” are likely not aligned with quizzes that feature only multiple choice questions as the learner would not be given the opportunity to actually explain or describe something. This doesn’t have to be a definitive analysis of the assessment (i.e., does this particular assessment measure this particular objective) but rather a quick judgment to determine if the assessment could measure the objective.

The other aspect of alignment to consider is alignment between the different levels of objectives. In general, objectives should scaffold from the more discrete (lesson level objectives) to more comprehensive (course level objectives). In the event that a course features both lesson and unit level objectives (i.e., the unit level objectives are not just simply the collected lesson level objectives), lesson level objectives should scaffold into the unit level objectives. However, while it’s likely that lesson level and unit level objectives were both created while planning the class, we’ve had more success with QM in the past if we just collect the lesson level objectives and label them as the unit level objectives. The most important thing here is that all objectives listed are aligned with an assessment.

It is not necessarily required to explicitly map out the alignment between unit level and course level objectives beyond listing the course level objectives targeted in the unit, though it is recommended (e.g., “ULO 1: Analyze the causes of the Great Depression [CLO: 1, 2]).

For more information about alignment, consult the style guide entry on alignment. Style Guide: Alignment

Look at content for potential issues in terms of currency/relevancy and accessibility/inclusion

For some content, there may be issues with the currency of the information. This is more likely in subject matter areas that are prone to quickly changing and becoming out of date. For example, during the recertification process, QM returned our Economics course for revisions due to a large amount of data being out of date (e.g., GDP figures being cited from 2014).

If this is an area where you’re a subject matter expert, you should be able to make this determination. If you’re not an expert in the subject, instead of trying to make this determination yourself, watch for dates that seem excessively old and other indicators that content may be outdated, such as dated references to “current” events. Additionally, watch for irrelevant information. Again, this may require subject matter knowledge to determine, though some things may be obvious and should be flagged for further review (e.g., several lessons unrelated to the subject of the course).

While accessibility is covered in much greater depth in other areas of the style guide, the following are common areas of accessibility that should be assessed prior to QM submission:

• File accessibility: Are files (Word, PowerPoint, PDFs, etc.) that learners are required to utilize accessible? For Microsoft Office files, use Microsoft’s built-in accessibility checker. Pay particular attention to PDFs that were created externally to Michigan Virtual as these often have accessibility issues. For quick tips on spot-checking PDFs, please see Checking PDFs for Accessibility.
• Caption/transcripts: All content that features audio should have captions or a transcript. If a video requires captions, these can be requested using the iPD help request in Wrike. Style Guide: Closed Captions
• Alt-text: All visual content should have alt-text that describes what is being displayed for non-sighted learners. Style Guide: Alt Text

Run link check to look for broken links (Brightspace and CourseArc)

In CourseArc, use the link checker in the course report to identify any broken links. The link checker will return one of several codes for each link in the class:

• OK (200): this means the link should be working.
• Error (403): this means the link may be slow to load, but it should still work. Click the link to verify.
• We were not able to check this link: this typically refers to links to downloadable files such as a worksheet. The link should be working.
• Error (404): this means the link is broken and will need to be replaced.

For detailed instructions on running the link checker in CourseArc, consult the following resource: Course Reports.

Brightspace also has a built-in link broken link viewer that can be accessed by selecting Course Admin from the horizontal menu at the top of the course page. Detailed directions for running the broken link viewer can be found here: Broken Link Viewer – Brightspace. You may have limited success using this tool.

Content Verification Conclusion

For many of the potential issues described above, a SME may be required to help complete the revisions. Therefore, after completing this review, check in with your supervisor before beginning revisions.

For those who are curious, the complete QM rubric can be accessed here: Style Guide: Quality Matters Rubric. Again, your pre-submission content check should focus on the key areas identified above, though feel free to also flag additional relevant issues you identify.

QM Course Worksheet

The QM Course Worksheet is a document we fill out to provide information about the course to the QM review team.

This document will be present in the Google Drive folder for the course submission (and/or it may be linked in a Wrike task). The document has instructions on how to complete the worksheet. 

For any issues or questions that arise as you complete the worksheet, add comments directly to the Google Doc and @mention our designated QM course representative (as of December 2023, this is Amy Marchese).

• Style Guide: Quality Matters Rubric
• Style Guide: Alignment