Quality Matters Recertification

1. If 15% or more of the course has been revised
2. If the certification has expired/is close to expiring

Before a course can be submitted for recertification, the course alignment document must be reviewed and updated to the latest template, the course contents must be reviewed to ensure it complies with QM standards, and the QM Worksheet must be filled out correctly.

Alignment

Quality Matters

All standards

Alignment Verification

• Verify the alignment document is in the newest template (available on Google Drive Template Gallery). This document has been reviewed by QM and contains the necessary items for the QM review such as:
  • Course Level Objectives
  • Unit Level Objectives
  • Standards Link
  • Instructional Materials Used
  • Learning Activities
  • Unit Assessments
  • Tools Utilized
• Check for alignment and measurability of Unit- and Course-Level Objectives

Content Verification

A course must meet the minimum threshold according to the QM Rubric and Standards (85% for most standards), or it will be sent back for an amendment period, which gives us 14 weeks to make any required revisions. QM reviewers utilize a thorough and in-depth rubric when reviewing courses, so it is not feasible for us to check a course on each of the QM standards prior to submission. As such, the pre-submission content check focuses on the following key areas:

• Ensure each assignment has either an answer key or a rubric
• Check for alignment and measurability of Unit- and Course-Level Objectives
• Look at content for potential issues in terms of currency / relevancy and accessibility / inclusion
• Run Link Checker to look for broken links (CourseArc)

Ensure each assignment has either an answer key or a rubric

For each assignment that students are required to submit that is not a quiz or test (e.g., discussion boards, written assignments, projects, etc.), there should be a student-facing rubric or checklist that outlines the requirements of the assignment and how students will be graded. While a simple checklist outlining the things a student needs to do to receive full credit on an assignment has been sufficient for QM in the past, preferably, assignments will feature more descriptive rubrics that outline the various levels of success possible on the assignment and the criteria for achieving each level.

To check an assignment, open the assignment page. A rubric should be attached as a link at the bottom of the page. While rubrics can be uploaded as a separate document, building them directly in Brightspace provides a better workflow for instructors.

A list of rubrics present in the course can be viewed by accessing the Course Tools dropdown menu from the horizontal menu at the top of the course page and then selecting Rubrics. Note: Just because a rubric is present in this list, it does not necessarily mean it’s in use in the course. If the rubric is linked to a course assignment, a chain icon will appear after its name.

For more information about rubrics, consult the style guide entry on rubrics. Style Guide: Brightspace: Rubrics

Check for alignment and measurability of unit and course level objectives

Using the alignment chart, review the unit and course level objectives.

To determine their measurability, review the verb(s) in the objective. This verb should describe a measurable action taken by the student that a teacher could assess. A framework like Bloom’s Taxonomy can be helpful in reviewing the measurability of verbs. Be sure that the objectives do not rely on a hard-to-measure formulation like “Demonstrate an understanding…”

In this context, alignment refers to several things. First, consider the alignment between the objectives and any associated assessments to ensure that the objectives are actually being assessed. Any objective listed on the alignment document will need to be aligned with an assessment. Additionally, consider the appropriateness of the assessment: can the stated objective actually be measured with the type of assessment listed? For example, QM has previously given Michigan Virtual feedback that objectives using verbs like “explain” and “describe” are likely not aligned with quizzes that feature only multiple choice questions as the learner would not be given the opportunity to actually explain or describe something. This doesn’t have to be a definitive analysis of the assessment (i.e., does this particular assessment measure this particular objective) but rather a quick judgment to determine if the assessment could measure the objective.

The other aspect of alignment to consider is alignment between the different levels of objectives. In general, objectives should scaffold from the more discrete (lesson level objectives) to more comprehensive (course level objectives). In the event that a course features both lesson and unit level objectives (i.e., the unit level objectives are not just simply the collected lesson level objectives), lesson level objectives should scaffold into the unit level objectives. However, while it’s likely that lesson level and unit level objectives were both created while planning the class, we’ve had more success with QM in the past if we just collect the lesson level objectives and label them as the unit level objectives. The most important thing here is that all objectives listed are aligned with an assessment.

It is not necessarily required to explicitly map out the alignment between unit level and course level objectives beyond listing the course level objectives targeted in the unit, though it is recommended (e.g., “ULO 1: Analyze the causes of the Great Depression [CLO: 1, 2]).

For more information about alignment, consult the style guide entry on alignment. Style Guide: Alignment

Look at content for potential issues in terms of currency/relevancy and accessibility/inclusion

For some content, there may be issues with the currency of the information. This is more likely in subject matter areas that are prone to quickly changing and becoming out of date. For example, during the recertification process, QM returned our Economics course for revisions due to a large amount of data being out of date (e.g., GDP figures being cited from 2014).

If this is an area where you’re a subject matter expert, you should be able to make this determination. If you’re not an expert in the subject, instead of trying to make this determination yourself, watch for dates that seem excessively old and other indicators that content may be outdated, such as dated references to “current” events. Additionally, watch for irrelevant information. Again, this may require subject matter knowledge to determine, though some things may be obvious and should be flagged for further review (e.g., several lessons unrelated to the subject of the course).

While accessibility is covered in much greater depth in other areas of the style guide, the following are common areas of accessibility that should be assessed prior to QM submission:

• File accessibility: Are files (Word, PowerPoint, PDFs, etc.) that learners are required to utilize accessible? For Microsoft Office files, use Microsoft’s built-in accessibility checker. Pay particular attention to PDFs that were created externally to Michigan Virtual as these often have accessibility issues. For quick tips on spot-checking PDFs, please see Checking PDFs for Accessibility.
• Caption/transcripts: All content that features audio should have captions or a transcript. If a video requires captions, these can be requested using the iPD help request in Wrike. Style Guide: Closed Captions
• Alt-text: All visual content should have alt-text that describes what is being displayed for non-sighted learners. Style Guide: Alt Text

Run link check to look for broken links (Brightspace and CourseArc)

In CourseArc, use the link checker in the course report to identify any broken links. The link checker will return one of several codes for each link in the class:

• OK (200): this means the link should be working.
• Error (403): this means the link may be slow to load, but it should still work. Click the link to verify.
• We were not able to check this link: this typically refers to links to downloadable files such as a worksheet. The link should be working.
• Error (404): this means the link is broken and will need to be replaced.

For detailed instructions on running the link checker in CourseArc, consult the following resource: Course Reports

.

Brightspace also has a built-in link broken link viewer that can be accessed by selecting Course Admin from the horizontal menu at the top of the course page. Detailed directions for running the broken link viewer can be found here: Broken Link Viewer – Brightspace. You may have limited success using this tool.

Content Verification Conclusion

For many of the potential issues described above, a SME may be required to help complete the revisions. Therefore, after completing this review, check in with your supervisor before beginning revisions.

For those who are curious, the complete QM rubric can be accessed here: Style Guide: Quality Matters Rubric. Again, your pre-submission content check should focus on the key areas identified above, though feel free to also flag additional relevant issues you identify.

QM Course Worksheet

The QM Course Worksheet is a document we fill out to provide information about the course to the QM review team.

This document will be present in the Google Drive folder for the course submission (and/or it may be linked in a Wrike task). While the document has instructions within it, the following questions may require further clarification:

19: “For reviews of labs…”

• This series of questions refers to the presence of labs or lab components within a course. While some of our classes may contain assignments labelled a lab, this refers more to a full-fledged lab component (i.e., a lecture course with a lab component, like you would encounter in a collegiate setting).

20: “What is the source(s) of the course-level objectives or competencies?”

• The state standards were adapted to be the course-level objectives.

23a: “Do state and/or other accepted content standards exist for the course?”

• Standards should be identified on the alignment document. Identify and share the link to these standards within the course worksheet.

24: “If necessary, provide further comments about the content standards. (For example, the specific proficiency level for a foreign language course or whether content standards exist, but this course is one in a series and will only align to a subset of the standards. ) Otherwise, enter ‘N/A.’”

• For most of our courses, the course will only address a certain subset of the standards. For example, a literature class may only address some of the ELA standards with the rest being covered in a later class in the sequence. If that’s the case, be sure to note that here. Additionally, electives are unlikely to address all required standards.

28a: “Does the course include real-time learner-instructor interaction?”

• If the course contains synchronous interactions between learners and the instructor (such as a discussion-based assessment where learners are talking to their instructor in real-time), note that here.

28d: “List other ways learners are required to interact with the instructor within the course:”

• Typically, learners will interact with their instructor through discussion boards and assignment submissions. Note any additional required interactions, as well.

40: “Was this course pre-reviewed (internally) with QM Standards?”

• Yes

For any issues or questions that arise as you complete the worksheet, add comments directly to the Google Doc and @mention our designated QM course representative (as of December 2023, this is Amy Marchese).

• Style Guide: Quality Matters Rubric
• Style Guide: Alignment