Section 3: Assessment in Practice: Academic Program Outcomes Assessment

Introduction & Overview

Program outcomes assessment includes the important steps of defining and then collecting evidence about program learning outcomes of a program. This process leads to improvement or validation of student-focused best practices. Program faculty reflect on the evidence collected and draw conclusions about the meaning of the evidence, develop a plan to improve student performance, and then continue to collect evidence to determine the effect of the changes.

Who participates in Program Outcomes Assessment?

College programs, whether they are academic or administrative, are required to perform assessment of their program outcomes, goals or initiatives.

• Every academic program that leads to a certificate or degree must report student learning outcomes data and plans for improvement and/or growth to support student success and completion.
• Special programs and academic areas that do not award a degree or certificate are also responsible for submitting a program outcomes assessment as part of both the Year-3 Integrated Report and the CAR
  • Special Programs are programs designed to support student success and learning that further the mission and vision of the college. 

A faculty-based workgroup is convened to complete the reflection document. The makeup of this faculty workgroup is determined by the Dean and Chair and can include staff or administrator(s) if deemed necessary.

 

Student Learning Outcomes (SLO’s)

The first step in the assessment process is to identify, clarify, and in some cases to create new program outcomes that are specific and measurable. Current program outcomes can be found in the MC Catalog. If clarification or updates are needed, this should be addressed through the curriculum process prior to the CAR, during year 5 of the assessment cycle.

For each measurement (outcome), program faculty will set satisfactory performance criteria. In some cases, faculty will need to establish rubrics that describe acceptable student performance. Program faculty will determine a benchmark, (an expected level of outcome achievement) that will be used to verify the level of achievement for a specific outcome.

 

Organizing & Completing Assessment Activities

Assessment at MC is intended to be a faculty-driven process. Academic and special programs can choose to organize and complete their assessment functions based on several factors: the size of the discipline and the number of courses offered, the ration of full-time to part-time faculty, and the leadership styles of the program’s Coordinator(s), Chair(s) and Dean.

Below is the most common approach that programs use when completing the yearly reports in the program assessment cycle.

• In many areas it is the program coordinator or the Chair’s designee who will act as the lead for program outcomes assessment activities and will often form a faculty workgroup to gather and analyze data and complete reports for their program. Depending on the Dean and the makeup of the program’s discipline, these workgroups and the discipline leads will either be determined by the dean or the Dean’s designee (Chair, coordinator, etc.)
• The program leads for outcomes assessment will be responsible for the following activities 
  
  • Submit curriculum mapping, program assessment plans, and capstone course assignments/rubrics designed to determine student achievement levels and submit the completed form(s) every 6 years as part of the CAR (or as part of a curriculum action).
  • Consult with program faculty to formulate a data collection plan to score and upload assessment results during years 1, 2, 4 & 5 of their respective assessment cycle.
  • Submit the completed data collection plan(s) for all programs to the Collegewide Assessment Team for review, feedback, and approval.
  • Organize and oversee the data collection process.
  • Set up regular meetings every semester to discuss progress and assessment results. Discussion from these meetings will inform the Year-3 Integrated Report and the CAR Report.
  • In conjunction with all workgroups, prepare and submit all required reports, to include:
    
    • Annual Data Collection / Assessment updates
    • Year-3 Integrated report
    • Year-6 Program Outcomes Assessment (incorporated into the CAR report)
• Regardless of the program’s approach to assessment, the following applies:
  • The Chairs are responsible for providing support and guidance for faculty members (as needed) throughout the assessment process and may, in conjunction with faculty members, assist with the assessment activities (when applicable).
  • The Deans are responsible for ensuring that any assessment requirements for their areas are completed and submitted by the required due dates.

 

Curriculum Mapping: Choosing Courses that Best Support your Outcomes

By mapping required course outcomes to program outcomes, noting which topics are introduced, reinforced, and applied throughout the curriculum, faculty will begin to see what knowledge, skills and attitudes are currently taught. This information will be the basis for a discussion among faculty as they work to define what they want the program to achieve.

Choosing appropriate course(s) in the program curriculum (where students are expected to demonstrate their achievement of the most advanced levels of program outcomes), provides a way to measure student performance for program expectations. A method of assessment that can provide measurement of acceptable performance for a student majoring in the program will provide evidence that can be collated, reported, and discussed by faculty as part of the Program Outcomes Assessment report (See: Capstone Assignments, next section).

 

Goals, Student Insight, Data

Valuable data points are available once assessment scoring takes place. By linking student scores to M numbers, numerical data are generated that not only include information on student performance, but also demographic information. This information can help the workgroup consider the implications of possible outside factors on scores and should lead to strategies and plans to improve specific outcomes for specific groups of students.

Special programs and academic areas that do not award a degree or certificate can enter scores and access integrated data for courses, special projects, and/or enrollment trends. These Programs are expected to reflect on the relationship between program goals and student success data, and to make recommendations for improvement during the Program Assessment portion of the CAR.

 

Student Feedback

The College Area Review (CAR) requires a channel for student feedback to be considered by faculty as part of that review. Some type of student survey or feedback mechanism should be part of your assessment planning. Polling students on attitudes and challenges related to collegewide support and achievement like academic planning (SAP), program advising, transfer plans, financial aid and program improvements could lead to necessary, and sometimes simple changes in policy or strategy that help a student navigate, improve instruction, enrollment and/or graduation rates.

A survey or feedback channel is also a source for the topics covered in the Year 3 Integrated Report:

Creating Your Program Outcomes Assessment Plan

Your Assessment Instrument

Assessment instruments are the tools that will be used to collect data that reflect student performance. These may take many forms and depend on the discipline, course material, and the competencies or outcomes being measured. Examples include, but are not limited to, the following: written exams, oral exams, practical exams/exercises, papers, course assignments, lab exercises, capstone projects, high impact practices, etc.

Questions to consider when choosing an assessment instrument:

• Does the assignment or exercise truly reflect student performance for the specific outcome you want to collect data for? (Validity)
• How can it be measured? (Scoring Scale? Correct/incorrect answer? Rubric with defined parameters? etc.)

The key to a successful implementation of a Program Outcomes Assessment Plan is clear and timely communication with all participants. This could include faculty (FT & PT) who will be teaching the program’s capstone course(s) and the courses that support it, or staff and administrators in a special program. The Program Assessment Plan determines “what” will be assessed and benchmarks that indicate the degree of success achieved by students.

 

Benchmarking

Benchmarking refers to the process by which a discipline or program sets a standard of measurement (or benchmark) for evaluating or comparing student performance during and after the assessment and reassessment process. For instance, what percentage of students is expected to attain expected outcomes, or exceed expectations? What percentage of students may fall below standards of success?

Please see the quick tips below regarding benchmarking:

 

Evidence Gathering

Evidence gathering for academic programs at MC consists of scoring and student performance levels on program outcomes. The results derived from these data are reported in both the Year-3 Integrated Reflections Report and the College Area Review (CAR) Report.

Creating Your Data Collection Plan

The process of gathering (or collecting) program outcomes “data” refers to the actual collection of student assignments or projects to be “scored” in accordance with your discipline’s program outcomes assessment plan. This collection of student performance across each competency takes place during years 1-2 and years 4-5 of your discipline’s assessment cycle and follows the data collection plan created by your discipline. Once the data are collected, the act of “scoring” or evaluating student performance takes place, to be uploaded to the MC assessment system for analysis (see “Scoring & Uploading your Collected Data” – next section.)

For example, certain disciplines might only collect data on an annual basis and might only score a random sample of the student assessments because of the sheer numbers of students in their program. But smaller disciplines/programs with fewer students in their key program courses might collect data every semester, or every time a capstone course is run, to ensure that scoring is sufficient to be reflected upon and to determine possible plans for improvement.

Since faculty are the experts for their specific courses and requirements, it is up to the discipline coordinator and faculty members teaching in the discipline to determine the plan for collecting data withing years 1-2 and 4-5 of their designated assessment schedule. The Data Collection Plan should be created with careful consideration of the following:

When creating your data collection plan, you should:

• Examine when and how the assessment instrument is assigned (semesters, campuses, modality, etc.) in order to capture a diverse and rich sample of student performance for each program outcome across different settings if possible.
• Ensure that data collection encompasses an appropriate number of course sections and that the number of students per section is suitable for an assessment sample (see sampling below.)
• Take into consideration the faculty members who may be teaching multiple sections of a given course and weigh any potential effects that might be created from additional time requirements/duties regarding scoring responsibilities.
• Guarantee that the appropriate data will be collected, scored, and available for analysis/discussion in order to meet the requirements and deadlines for completing the Year-3 Program Outcomes Assessment Reflection Report and the 6-Year Reflection Report (incorporated into the CAR).

How much data do we need to collect?

*In addition to the information below, please also refer to the data chart in Appendix C for guidance on how much data may be appropriate for your program.

The amount of data to collect depends on the size and number of sections and students completing the program. Some common data collection questions/answers have been provided below for guidance; however, when in doubt, please contact the Office of Assessment for assistance in determining the appropriate amount of data for your discipline.

Please see “Assessment in Practice” below for an example of a program outcomes data collection plan.

Your Comprehensive General Education Assessment Plan

The Program Assessment Plan form provided can be filled in with the program name, discipline name, contact information and Dean. Each outcome is listed down the left-hand side, exactly as it is published in the catalog. The form provides examples of how to fill out these four columns of information:

• Methods of Assessment: What will the assessment instrument be? What will you use to assess the students?  
  • Example: An essay paper, a project, an oral presentation, a portfolio.
• Assessment Course: In what class or event will information be gathered to assess this outcome?
  
  • Example: TVRA 260 Video Production Portfolio
• Assessment Scoring: What score or level of performance is considered to be acceptable for a student majoring in the program?
  • Example: An acceptable level is for students to earn 85 out of 100 points on the project.
• Benchmark or Expected Level of Outcome Achievement: What percentage of your students do you expect to meet the criteria of success for this outcome?
  • Example: We want at least 75% of the students to receive 85 points on the project.

The illustration below represents an example of how outcomes, methods of assessment, scoring and benchmarks are reflected in a program assessment plan.

Assessment Plan

 

Data Scoring and Analysis

Scoring & Uploading Your Collected Data

Before You Score...

Before any scoring takes place, it is strongly recommended that all participants in the scoring process meet to discuss how to interpret and evaluate the data and to know what is expected of them. There are often differences of opinion regarding what values to assign for a student’s work and it is imperative that all faculty members are in agreement to ensure consistency in scoring to produce the useful data. It is suggested that at least one piece of student work is scored together and discussed as a group before further scoring takes place to promote inter-rater reliability.

Scoring student work for Assignment

Depending on the size of the discipline and the number of certificate and/or degree programs, a group of faculty representatives from the discipline (versus every individual instructor) may be organized to complete the final scoring and uploading of program assessment data.

Scoring (or rating) the performance of students in your program is an important evaluation process to determine how well students are successfully attaining program outcomes. The scoring process for program outcomes involves the use of predetermined measures to assess the performance level demonstrated by students on designated assignments/exercises/etc. The scoring criteria and benchmarks for student performance that are utilized during this process are determined by faculty who teach in the program and can be found in the Program Assessment Plan (see “Benchmarking” above for more information.)

While scoring and grading can be interrelated, there are important differences between these two functions. All participants involved in the scoring process should be made aware of these differences to ensure that the scoring process produces the most accurate and useful data results possible. For an explanation of how “grading” and “scoring” differ, please see “Why Grades are not Enough” on page 1 (recommended reading for all faculty involved in assessment activities).

Uploading the Data

• Once scoring is complete for a given data set, designated faculty members will enter the results into the designated area specified by the Office of Assessment to be stored for immediate and/or future analysis.
• Collectively, these data provide an important holistic view of student performance on program learning outcomes across the discipline.
• Statistical analysis will be performed on the data by the Office of Assessment and/or lead faculty members, resulting in aggregated results that are NOT identified with any instructors. These results should then be provided to all faculty members teaching in the program for analysis and reflection. (Please see next section “Data Analysis: Making Sense of the Results” for more information.)

 

Data Analysis: Making Sense of the Results

Everyone teaching the course that was assessed should be informed of how students performed and be involved in any discussions for future planned actions relating to assessment.

This communication should include:

• the purpose in assessing learning outcomes
• the learning outcomes that are being assessed
• the common assessment instrument to be administered
• when the assessment is to be administered during the semester
• what students should be told about the assessment and its purpose
• the common rubric or answer key to be used in scoring the assessment
• how to enter scores into the scoring spreadsheet that will be provided
• the fact that assessment results will never be reported in a way that could reflect on the performance of an individual faculty member or student
• and of course, the results and specific actions that the discipline/program plans to implement for improvement.

Some departments have prepared and distributed a memorandum to all faculty who will be participating in the assessment that provides information on the items listed above. An example of such a memorandum is included in Appendix D: Memo: Assessment Data Collection - Beginning of semester “Heads up”

Summary Reflections & Planned Actions for Improvement

What is a Summary Reflection?

College programs, whether they are academic or administrative, are required to perform assessment of their program outcomes, goals, or initiatives as they relate to the MC mission and strategic plan. To document the important faculty discussions and planned actions that come from analysis of the data collected, the program workgroup submits a Summary Reflection form (see step 5 of “Examining the Data”, page 42.) These notes and results are used to complete this important part of the Year 3 summary report and the CAR report. Capturing this collaborative thinking is what makes assessment valuable and effective. Summary reflections are revisited in year 3 (by October 1) and year 6 (by October 1) when there is new data to compare and when it is time to reflect on the effectiveness of changes to instructional practices that were recommended as part of the previous summary.

This part of “Program Assessment” gives faculty and others involved a voice in the evaluation and analysis of the data and the ability to suggest or support new recommendations based on the analysis of data. Collegewide initiatives and goals should be part of this reflection. The reflection report includes the number of students assessed, the semester(s) that data was collected, a summary of the data including percentages and demographic results, and a comparison of results to previous data collection periods.

This summary is repeated for each program outcome. The last portion of the summary report asks you to reflect on planned actions from your most recent report (3 years ago) and determine if those recommendations should continue or be updated. There is an opportunity to add planned actions based on the most recent data. This report is then reviewed by the Collegewide Assessment Team (CAT) and feedback is provided to faculty members.

 

Creating Planned Actions for Improvement

Planned actions are actions created by faculty members to address any weaknesses or gaps that have been identified in student performance. These actions should be specific, measurable, and clearly defined. They should also be timely (promptly executed) within a reasonable time frame (i.e., starting next semester, etc.).

Please see the examples below for “Assessment in Practice.”

Zoology faculty have analyzed the results from their capstone assignment(s) data and have discovered that their students are excelling in many of the outcomes but seem to be struggling on the fourth outcome – “…communicate scientific information through effective formal and informal writing”.

Practical Example:

The discipline faculty have developed planned actions that they believe will support student improvement in these areas. Please see the examples below:

 

Sharing Results & Tracking Improvement

The Important Concept of “Closing the Loop"

While some faculty may associate the assessment process with data collection and writing reports, these activities represent only half of the process of assessing student learning. One of the most important aspects of assessment is the expectation that results garnered from the process will then be interpreted and utilized to initiate positive, measurable change for improvement. This critical, final step of analyzing and implementing action to improve student learning, is known as “closing the loop” of the assessment/reassessment cycle. Closing the Loop takes place at MC as part of the Year-3 Integrated Report and as part of the College Area Review (CAR). These positive, measurable changes are represented in assessment reports as “planned actions” or “recommendations.” These actions are formulated through discussion by faculty and administrators and are intended to be “action plans” for immediate implementation (respectively). Feedback on proposed actions is then provided by assessment committees and administrative leaders.

The final step in “Closing the Loop” is to review the results (data collected) after these actions have been implemented to determine what impact they have had on student success, thus initiating the assessment cycle (or loop) once again for improvement.

Reporting for Program Assessment

The reporting requirements for the assessment of programs consist of annual updates during years 1, 2, 4 & 5 for each certificate or degree and/or special program for its assessment activities, and a report every three years that analyzes student performance on program outcomes or goals. Assessment reports are evaluated by the Collegewide Assessment Team and reviewed by the College Area Review Committee and as part of a program’s CAR report in Year 6.

Assessment reports for Program Assessment consist of:

1. Annual Update: Interim Data Collection & Assessment Report - (Year 1, 2, 4 & 5)
2. Year-3 Program Assessment Integrated Reflections Report
3. Year-6 Program Assessment Reflection Report (as part of the discipline’s CAR report)

*For detailed descriptions of information required for each report, please refer to the individual report sections below. To view a template of each report, please visit the Assessment Repository on Blackboard (Once in the Repository, click on the folder on the main menu to your left titled “Assessment Resources, Forms, Toolkits, & Templates.”)