The Scoring Algorithm

From WebPA Wiki
Revision as of 17:31, 24 March 2014 by Adjda (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

If you want a quick run through of how the scoring algorithm works, it may be better to check the overview of Web-PA.

The following is a worked example of how Web-PA would grade an assessment.

A Worked Example Just a couple of points before you start reading this worked example.

Firstly, as you will soon notice, this is a very simple example assessment focussing on a single group marking each other against a single criterion.

Also, for readability, most of the numbers used are rounded to 2 decimal places. The real algorithm uses much greater precision and accuracy!

The Data Performance in the group task

	Alice	Bob	Claire	David	Elaine	Total Given

Alice 4 4 3 2 1 14 Bob 3 5 3 2 0 13 Claire 4 4 4 4 4 20 David 3 5 4 3 1 16 Elaine - - - - - 0 Total Received 14 18 14 11 6 63 In this example, the rows across are the marks given by each student, whereas the columns down show the marks each student received. The marks received are totalled at the bottom.

A quick review of the scores shows that Bob performed the best, while Elaine was very much the worst performer. Also, Elaine didn't submit any marks for the peer assessment at all, as denoted by the dashes in her row.

Get the Total Marks Allocated by each Student For each student, we add up all the marks they awarded to everyone in the group, including themselves (i.e. total up each row).

Alice awarded: 4 + 4 + 3 + 2 + 1 = 14 marks

Bob awarded: 3 + 5 + 3 + 2 + 0 = 13 marks

Claire awarded: 4 + 4 + 4 + 4 + 4 = 20 marks

David awarded: 3 + 5 + 4 + 3 + 1 = 16 marks

Elaine awarded: 0 + 0 + 0 + 0 + 0 = 0 marks

If there were more criteria in this assessment, these scores would be totalled for all of them. In this case, however, there was only one criterion so we only have to add up one set of scores.

Normalise the scores allocated by each student Now we know how many marks each student allocated, we can normalise the scores they gave and calculate a fractional score for each one. In other words, every student in the group has an identical 'cake', but they can give different sized slices to each other (and themselves!).

Alice awarded a total of 14 marks (so her cake was split into 14 slices).

Alice gave herself 4 marks = 4 / 14 = 0.29

Alice gave Bob 4 marks = 4 / 14 = 0.29

Alice gave Claire 3 marks = 3 / 14 = 0.21

Alice gave David 2 marks = 2 / 14 = 0.14

Alice gave Elaine 1 marks = 1 / 14 = 0.07

These are the fractions awarded by Alice to each student. As a test, you can add up all the fractions and they should equal 1.

This process is repeated for every student.

Calculate the Web-PA Score Awarded to each Student Before we can calculate each student's Web-PA score, there's one more bit of information we need, how many students were in the group, and how many of them submitted marks.

In this case we have 5 members, which means 5 cakes should be split between 5 members. Unfortunately, Elaine didn't submit any marks, so only 4 cakes are available!

To compensate, we quickly calculate a multiplication factor to bring the total number of cakes back up to 5.

5 (students in group) / 4 (students submitting) = 1.25 (our fudge factor)

At long last, we can calculate the Web-PA scores. Taking Alice as an example, we add up all the fractional scores she was awarded. We calculated Alice's own fractions above, but the other students' fractions were produced in exactly the same way...

Alice gave herself 0.29 (as we saw above).

Bob gave Alice 0.23 (= 3 / 13).

Claire gave Alice 0.20 (= 4 / 20).

David gave Alice 0.19 (= 3 / 16).

Elaine didn't submit, so she effectively gave 0.00.

If you now add up all those individual factors, you get the Web-PA score for Alice:

Alice's Web-PA score = 0.29 + 0.23 + 0.20 + 0.19 + 0.00 = 0.91

If everyone had submitted, that would be fine, but remember we had one non-submission, so we have to bring in our fudge factor from earlier...

Alice's actual Web-PA score = 0.91 x 1.25 (the fudge factor) = 1.14

If do the same calculations for each of the other students, we find:

Bob's actual Web-PA score = 1.18 x 1.25 = 1.47

Claire's actual Web-PA score = 0.89 x 1.25 = 1.11

David's actual Web-PA score = 0.68 x 1.25 = 0.85

Elaine's actual Web-PA score = 0.33 x 1.25 = 0.41

As a quick check, if we add up all the Web-PA scores for every student, it should be equal to the number of students in the group. Adding up the scores gives 4.98, which allowing for rounding errors (we're using 2 decimal places in this example) is 5.

Calculate the Student's Final Grade Now we have the Web-PA score, we can easily calculate a student's final grade using the overall group mark.

Let's assume we've awarded this group's work 80%. Ordinarily, you may have given everyone in the group 80%, but now we can use the Web-PA scores to give proportionally higher or lower marks according to the students' actual performances. For now, we'll assume that we want to apply the peer assessment to the entire group mark...

Alice's Grade = 1.14 x 80 = 91.2%

Bob's Grade = 1.47 x 80 = 117.6% = 100% (we don't give grades above 100%!)

Claire's Grade = 1.11 x 80 = 88.8%

David's Grade = 0.85 x 80 = 68.8%

Elaine's Grade = 0.41 x 80 = 32.8%

So there you have the final grades for each student. As you can see, three students did better than the group mark would have suggested, but two students were worse off. According to the peer assessment, that's a fair result.

Weightings and Penalties That's not quite the end of the story though, as the calculation above assumes that the entire group mark should be adjusted using the peer assessment. As you can see, this can lead to large spread of marks. Fortunately, if you want to control the spread of marks, Web-PA lets you tweak the algorithm using a couple of extra parameters.

PA Weighting

The first parameter you can change is the PA Weighting. This defines how much of the overall group-mark should be fixed (given to all the students automatically), and how much should be peer-assessed (calculated using the algorithm above).

If instead of peer-assessing the entire group mark we used 50% instead, then all 5 students would automatically receive the other 50% of the group mark. In this case, with an overall group mark of 80%, that's an automatic 40% fixed, and the remaining 40% will be adjusted by the peer assessment. The students final grades would then become:

Alice's Grade = 40 + (1.14 x 40) = 85.6%

Bob's Grade = 40 + (1.47 x 40) = 98.8%

Claire's Grade = 40 + (1.11 x 40) = 84.4%

David's Grade = 40 + (0.85 x 40) = 74%

Elaine's Grade = 40 + (0.41 x 40) = 56.4%

As you can now see, the spread of grades has been reduced substantially. Bob's previous grade was 117.6% but this has been brought down to 98.8%. Similarly, Elaine's low score of 32.8% has been brought up to 56.8%.

Non-Completion Penalty

The second parameter is the non-completion penalty. For students who didn't take the peer assessment, you can choose to penalise them anything up to 100% of their final grade.

If we use a penalty of 10% as an example, most of our students would be unaffected, but Elaine failed to take the assessment, so she would receive a 10% penalty:

Elaine's Grade = 56.4 - 10% = 50.76%

For more information on these parameters please check the marking parameters page.