Methodology
Course selection:
The SBC faculty working groups (FWG) and the Office of Academic Assessment (OAA) integrated multiple strategies in selecting courses to balance faculty recommendations, convenience, and the structure of the curriculum as well as statistical considerations. This initial assessment of the SBC is intended to serve as a pilot with the goal of “checking the pulse” of the SBC while collecting a maximum amount of information with minimum impact on resources and valuable faculty time. This approach will inform future iterations of SBC assessment and will serve to inform faculty in making future decisions about selection criteria, features of representativeness, interpolation within groupings, and most importantly, the ability to extrapolate inferences to broader populations.
The first pass of course selection leveraged the intuition and content knowledge of the members of the FWG’s, who prioritized large courses taught by tenured faculty. On occasion, the committee selected courses taught by full time lecturers or assistant professors in the event that no other options were available. The FWG’s also considered departmental impact and other course characteristics such as cross-listed courses, and courses with multiple SBC certifications. OAA then conducted a representativeness check to examine the samples within each SBC category and estimated an average 5% margin of error based on 95% confidence level and maximum comparison variance. OAA also applied a commonly used criterion for general education assessment of 10% of enrollment or greater within each SBC based on Spring 2016 enrollment. Additionally, OAA balanced the selection of courses to minimize overburdening departments and faculty as well as to ensure non-overlapping course selection by SBC.
Notes on course selection:
- We tried to balance the course selection across all departments and schools, and include representative samples from east, west and Southampton campuses.
- We tried to target tenured faculty. On a few occasions, to get a statistically sound sample size, we selected courses taught by assistant professors, full-time lecturers, and --in departments where assessment is already expected for local accreditation -- a few adjunct instructors.
- For courses that carry multiple SBC certifications, we expect instructors to only evaluate one *specified* SBC objective. Unfortunately, however, to achieve a statistically sound sample, we cannot allow instructors to chose among the multiple certifications. Instructors can choose to evaluate more than one, but should at least evaluate the specified SBC for multiple-certified courses.
Rubrics:
The faculty working groups developed the rubrics based on the existing learning outcomes as developed by faculty committees between 2009 and 2014. We did not deviate from the published learning outcomes as published in the Undergraduate Bulletin. The faculty working groups developed evaluative criteria on a scale of 1 to 5 for each learning outcome in an effort to refine the definitions of student achievement. Faculty can adapt the rubrics to their individual courses by selecting an "operational verb" from Bloom's taxonomy (as indicated in a list that accompanies each learning outcome on each rubric).
Along side the scale of student mastery (1 to 5), each rubric also includes a scale of "or earned score" to trigger faculty's intuitive sense of traditional grading. Think of the evaluation of students based on learning outcomes as "micro grading" based on the evaluative criteria of each learning outcome.
Data collection:
We ask faculty to provide student data in a specific format. The Office of Academic Assessment will develop, with the guidance of the faculty working groups, an online interface to simplify the data collection process. More information to come. OAA will do its best to make it easy for faculty to do their part.
In cases where the department has recently conducted assessment of the specified
course and SBC objective, and if the data are significantly robust, the department
can work with OAA to translate recent evaluative data into the SBC rubrics.
Analysis:
The Office of Academic Assessment, in collaboration with the faculty working groups, will analyze the resulting data and develop action plans for each SBC objective based on the results. For this project to be meaningful, the results should lead to improvements of the curriculum or its delivery.
Exceptions:
During this round of assessment, we will not evaluate SBC objectives HFA+, SBS+, STEM+ or EXP+ due to their complexity.