robdylan

The problem with player ratings…


Written by Rob Otto (robdylan)

Posted in :Original Content, Super Rugby on 10 Mar 2016 at 16:45
Tagged with : , , , ,

I’ve been having a little to and fro on Twitter over the last couple of days, regarding the decision of a popular rugby portal to crowd source weekly player ratings from fans and then use these ratings as the basis of articles that they carry. Given that it’s very hard to really get your point across in snippets of 140 characters, I’ve decided to try and put down my side of the argument in a more reasoned (and hopefully more coherent) format here.

It’s been well over a year since I’ve last written a “score out of ten” player ratings piece myself. Those who want to keep me honest can go back through the archives and check, but the most recent one I can find was published on 1 September 2014, after the Sharks played the Pumas in that year’s Currie Cup. There’s been a lot of soul-searching involved on my part, because I battled to reconcile the undoubted popularity of these articles with the feeling that I wasn’t really doing anybody justice (myself included) in publishing them. I like to feel I always tried to add additional colour to those pieces, explaining some of the thinking behind the ratings and not going out of my way to be mean when I felt a guy had played poorly, but even with that caveat in place, it just didn’t feel like a very constructive thing to be doing.

I eventually decided to stop doing it based on a few key factors. The first was that my own ratings were always subjective and I had to admit that I found it very difficult to not let my personal feelings about a player colour the way I perceived his performance. That’s not fair, because if you’re going to rate a guy on what he did this week, you cannot allow last week or the week before to come into it. I was called out time and again, usually for being “soft” on the players I liked most (or knew personally) and I came to realise that I actually valued those relationships more than I did the “sensationalistic hits” I might get by giving low ratings. Perhaps more concerning, tough, I would always find my ratings changing as I watched the match again (or looked at match statistics) and came to realise that it simply isn’t possible to arrive at a fair and accurate assessment of how every one of the 23 involved had performed over the course of 80 minutes. It would take a week of watching and re-watching to do that and by the time I was done, nobody would care any more anyway. A recent article by none other than Brendan Venter reveals that he suffer from the same problem.

In short, I didn’t want to publish ratings that were, for want of a better term, my own knee-jerk made-up rubbish. I like to think I know this game a bit, but I’m not a coach, I’m not an expert and I don’t have this brilliant level of insight that others don’t. Who am I, then, to think that I’m clever enough to accurately assess each player over 80 minutes and rate them, rather arbitrarily, on a scale for which no calibration has been agreed? Do I really know enough about what is expected of an outside centre over the course of 80 minutes and am I able, in real time, to accurately determine to what extend he has fulfilled his role over 80 minutes better than the tight lock has performed his? Do I know what personal goals the hooker has set himself (in terms of how many tackles, cleans, steals, runs, etc, he intents to make) and can I accurately judge to what extent he has met those goals? Do I know what his coach has asked him to concentrate on? Of course not. It gets even worse if I’m watching on TV, where a team of cameramen and editors are making their own decisions about which tiny fraction of the ongoing action I’m going to be shown. I’ll never know just how hard that lock is working off the ball, on his positioning to be able to make a crucial tackle, or just how much ground the fullback is covering (out of picture) in order to take that bomb.

These are the reasons I’m no longer prepared to come up with ratings for Sharks players, who play one game a week. If I’m not able to do this with any level of accuracy, then how is anyone meant to accurately rate EVERY player over the course of a Super Rugby weekend that comprises anywhere up to nine different matches? The reality is that they aren’t able to do this with anything like fact to guide them, so it ends up becoming all about subjective opinion, or worse, ratings based on a highlight reel. By this, I mean you give a guy 9/10 if you see him make a wonderful break, or score a great try, but give him 2/10 if he misses a sitter of a goal kick, or is unable to make the final cover tackle to prevent a try. Because you’re not able to accurately process the hundreds of data points you’d need to in order to reach a proper rating, you simply rate him based on the most obvious thing you can see. Or in some cases, it’s clearly just a guess, because in at least one case last week, a player managed an average rating of 3.5 despite not actually taking the field at any stage. Keep it up, Juan Schoeman, you sit a bench like a true professional, son! Garth April clearly didn’t sit as comfortably, because he scored 3.1 for doing the same thing.

Let’s leave the ratings for the coaches and only the coaches. Those are the guys who actually have the data and are qualified to make these calls. If a guy does something particularly good, let’s call it out as a great moment – and if a guy pulls off a shocker, let’s have a laugh about it and get on with our lives, without feeling the need to brand his entire 80 minutes worthless and cast aspersions on his ability (and often his character) in the process. Rugby does so much good work in terms of bringing people together and lifting them up. It doesn’t need this style of rating and certainly doesn’t need the style of writing that uses said ratings as an excuse to write nasty articles that denigrate players through childish name calling.



36 Comments

  • If it’s in seriously bad taste to objectify women by giving them some sort of arbitrary score based on some scale then it must follow that it is in bad taste at a fundamental human level regardless of whether “it’s just for fun”. The only people I’ve come across that seem adamant that it has any sort of value at all happen to work human resource departments. I’ll refrain from expanding on what I think of HR.

  • Comment 1, posted at 10.03.16 17:18:01 by vanmartin Reply
    Friend of Sharksworld Author
    vanmartinAssistant coach
     
  • Think April must really work hard on his smile and/or posture! Wonder what Lambie would have score if the same system was used for Bokke games?!?! One word comes to mind: Idiots!!!!!!

  • Comment 2, posted at 10.03.16 17:31:58 by JD Reply
    Friend of SharksworldCompetition Winner Author
    JDAssistant coach
     
  • @vanmartin (Comment 1) : I have a feeling a lot of people have a similar view of HR!!!!

  • Comment 3, posted at 10.03.16 17:37:44 by JD Reply
    Friend of SharksworldCompetition Winner Author
    JDAssistant coach
     
  • I have sympathy for your plight, Rob. After finding that I had rated Conrad Hoffman -4,563 for 243 consecutive weeks, I realised I might not be as objective as I had previously believed…

  • Comment 4, posted at 10.03.16 18:08:15 by Culling Song Reply
    Friend of Sharksworld Author
    Culling SongTeam captain
     
  • @vanmartin (Comment 1) : HR are the guys who, after the war, walk across the battlefield bayonetting the wounded…

  • Comment 5, posted at 10.03.16 18:09:46 by Culling Song Reply
    Friend of Sharksworld Author
    Culling SongTeam captain
     
  • @Culling Song (Comment 5) : hear they say the same thing about accountants.

  • Comment 6, posted at 10.03.16 18:27:31 by 50shadesofshark Reply

    Team captain
     
  • @50shadesofshark (Comment 6) : not that im either!

  • Comment 7, posted at 10.03.16 18:28:02 by 50shadesofshark Reply

    Team captain
     
  • @Culling Song (Comment 5) : @50shadesofshark (Comment 6) : hahahaha!!!!! I personally rate both 3/10

  • Comment 8, posted at 10.03.16 19:38:12 by JD Reply
    Friend of SharksworldCompetition Winner Author
    JDAssistant coach
     
  • I don’t like the rating system simply because it is so antiquated and simply adds no insight into what the players actually did (or fail at) on the field.

    I do however enjoy a rating system where players are identified by order of the value they added to (or subtracted from) the team effort on the day.

    I believe we had something like a good, the bad and the ugly scoring here on Sharksworld?

  • Comment 9, posted at 10.03.16 20:39:38 by FireTheLooser Reply

    Assistant coach
     
  • that lock with the surname Erasmus, that played so well for the lions last year in the currie cup, has he played any super rugby, i see krugers on the bench for them

    that kid really stood out in the cc, even more so than the really tall lock they played in super rugby last year

    I realise this is a sharks blog, but i figure some of you keep a close eye on all potentially good players and bok prospects

    maybe his injured

  • Comment 10, posted at 10.03.16 20:46:16 by revolverocelot Reply

    Team captain
     
  • @Culling Song (Comment 5) : they’ll rob them first as a fee for having to go to the trouble of bayonetting them

  • Comment 11, posted at 10.03.16 21:12:54 by vanmartin Reply
    Friend of Sharksworld Author
    vanmartinAssistant coach
     
  • @Culling Song (Comment 4) : Beaut… :grin:

  • Comment 12, posted at 11.03.16 00:55:03 by pastorshark Reply
    Friend of Sharksworld Author
    pastorsharkCoach
     
  • Nice article, Rob…

  • Comment 13, posted at 11.03.16 00:57:18 by pastorshark Reply
    Friend of Sharksworld Author
    pastorsharkCoach
     
  • I do know some very decent, helpful HR guys who have added a lot of value to corporations or projects I’ve been involved in… :grin: Just saying… :grin:

  • Comment 14, posted at 11.03.16 00:58:30 by pastorshark Reply
    Friend of Sharksworld Author
    pastorsharkCoach
     
  • @vanmartin (Comment 11) : @pastorshark (Comment 14) : The first half of my career was spent working for a large corporation, and I was often left with the impression that HR operated under the instruction to be a hindrance, rather than a help. Over the last ten years or so I’ve worked for smaller companies, and the HR departments have largely reversed my previously-held negative perceptions of them.

    Notwithstanding aforementioned, HR (much like industrial engineers) just makes for such easy targets that it’s hard not to hurl a little abuse their way every so often… :mrgreen:

  • Comment 15, posted at 11.03.16 07:09:35 by Culling Song Reply
    Friend of Sharksworld Author
    Culling SongTeam captain
     
  • @FireTheLooser (Comment 9) : That rating “The good, the bad and the ugly” was a good system and was hoping for something similar this season.

  • Comment 16, posted at 11.03.16 07:34:17 by Hulk Reply

    HulkSuper Rugby player
     
  • @50shadesofshark (Comment 6) : I’m an accountant and that’s not true.

    We chat to the wounded about forecast expenditure and budget models until they give in and take their own lives.

  • Comment 17, posted at 11.03.16 08:47:29 by ChrisS Reply
    Author
    ChrisSSuper Rugby player
     
  • I really felt that Schoeman did a great job warming up last week…definitely much better than April ;) I fully agree with your thoughts though Rob. If someone can say they have accurately assign a score to a player for a game without any bias they are lying through their teeth…or in WP guys case their gums ;)

  • Comment 18, posted at 11.03.16 08:56:40 by SheldonK Reply

    SheldonKAssistant coach
     
  • @ChrisS (Comment 17) : Chris you are clearly over excelling at your job. As an accountant you should merely tally the dead numbers, reconcile to original numbers…work out the percentages and then tell the man in charge he has overpaid the dead soldiers

  • Comment 19, posted at 11.03.16 09:03:30 by SheldonK Reply

    SheldonKAssistant coach
     
  • @Hulk (Comment 16) : I think we’ll move on to that sort of system again. Just want to let the guys settle in a bit more.

  • Comment 20, posted at 11.03.16 09:57:33 by robdylan Reply
    Competition Winner Administrator
    robdylanHead Coach
     
  • @ChrisS (Comment 17) : hahaha ja boet!!!! :twisted:

  • Comment 21, posted at 11.03.16 10:10:37 by JD Reply
    Friend of SharksworldCompetition Winner Author
    JDAssistant coach
     
  • @ChrisS (Comment 17) : Excellent :mrgreen:

  • Comment 22, posted at 11.03.16 10:14:05 by vanmartin Reply
    Friend of Sharksworld Author
    vanmartinAssistant coach
     
  • @vanmartin (Comment 1) : Hah Hah, someone scored lowly in their Key Performance Areas at their recent appraisal.

  • Comment 23, posted at 11.03.16 10:58:57 by The Great Couch Shark Reply
    Author
    The Great Couch SharkSuper Rugby player
     
  • @ChrisS (Comment 17) : Accountants have just gone up a notch in my estimation after that post… :lol:

  • Comment 24, posted at 11.03.16 11:02:56 by Culling Song Reply
    Friend of Sharksworld Author
    Culling SongTeam captain
     
  • I agree with you Rob and felt very sorry for poor Fred Z on his 2.5 rating. However, these are just opinions and should be taken with a pinch of salt. As are rankings for schools rugby etc. It just adds to the debate. I know Tank myself and he wasn’t meaning anything untoward, he was just having a laugh..

  • Comment 25, posted at 11.03.16 11:27:26 by Grasshopper Reply

    GrasshopperVodacom Cup player
     
  • @The Great Couch Shark (Comment 23) : Actually I was always above average at my previous job and I haven’t been around long enough at my current job to be rated yet. Even though I do okay doesn’t mean I like that someone who has no idea what my job actually entails is scoring me on some really vague KPA’s based on some even vaguer feedback from my manager who hates the whole scoring exercise even more than I do. I doubt it’s coincidence that nobody at me previous job and no one at my current job, aside from HR, thinks this has merit.

  • Comment 26, posted at 11.03.16 11:33:13 by vanmartin Reply
    Friend of Sharksworld Author
    vanmartinAssistant coach
     
  • @The Great Couch Shark (Comment 23) : @vanmartin (Comment 26) : You can probably tell I’m a little touchy when it comes to HR, it’s more involved than just performance appraisals though :mrgreen:

  • Comment 27, posted at 11.03.16 11:34:50 by vanmartin Reply
    Friend of Sharksworld Author
    vanmartinAssistant coach
     
  • @vanmartin (Comment 27) : The whole scoring system is bullshit. That’s why some of the top companies are now changing from performance appraisals to ‘future discussions’, ie ‘what have you done’ vs ‘where do you want to be”. That makes sense to me. Anyway, that’s my 2c worth on this rugby forum, hah hah.

  • Comment 28, posted at 11.03.16 11:39:45 by The Great Couch Shark Reply
    Author
    The Great Couch SharkSuper Rugby player
     
  • still waiting (perhaps naively) for a discussion about player ratings….

  • Comment 29, posted at 11.03.16 11:43:22 by robdylan Reply
    Competition Winner Administrator
    robdylanHead Coach
     
  • @robdylan (Comment 29) : yes this whole thing was hijacked by some people to discuss HR and accountants! :twisted:

    For me the great thing about individual sports like tennis is that the player that win is the best on the day. No points is awarded for style or anything else. Unfortunately in team sport were a player must be rated to select the best possible team, that will never be the case as a person is always (some less other more) influenced by your personal feelings and believes. If I do not know all the rules and all the requirements of a position (HR, position a rugby player plays, etc) I will never be able to accurately judge another persons performance. Therefor the idea of a person rating another will always be flawed.

  • Comment 30, posted at 11.03.16 12:19:16 by JD Reply
    Friend of SharksworldCompetition Winner Author
    JDAssistant coach
     
  • @robdylan (Comment 29) : Rob why don’t you desgn a voting/ballot system for each player that played and take the mean average ,that average will be the truest score

  • Comment 31, posted at 11.03.16 16:49:56 by benji Reply
    Friend of Sharksworld
    Team captain
     
  • @pastorshark (Comment 14) : yes Pastor cause the chaps you deal with tend to have wings and can do no wrong

  • Comment 32, posted at 11.03.16 16:51:51 by benji Reply
    Friend of Sharksworld
    Team captain
     
  • @Culling Song (Comment 4) : Classic

  • Comment 33, posted at 11.03.16 16:54:07 by benji Reply
    Friend of Sharksworld
    Team captain
     
  • @robdylan (Comment 29) : Bring back “the good, the bad, the ugly”.

    Players should be acknowledged for contributing to the team effort, but those having an off day can also be mentioned – not as an attack on them, but to discuss where we feel they went wrong on the day.

    If we were to get rid of any form of rating of players, then we might as well get rid of the score board, and let everyone compete for the fun of it.

  • Comment 34, posted at 11.03.16 17:05:45 by FireTheLooser Reply

    Assistant coach
     
  • I once heard a philosophy expert say that mathematical laws are the only laws us modern humans consider as absolute and inviolable.
    It must be why people like number based ratings. They appear accurate and objective, just like maths.

    But you are correct. If the method for coming up with the number is arbitrary, then the number is likewise arbitrary and if the method is not well defined or not defined at all it’s worse.

    If you want to do something better, i suggest you start by defining your method and then try to apply the method as objective as possible. Eg. All players starts with 10 and you deduct points based on mistakes made. Ratio that based on minutes played. OR All players starts with 0 and earn points based on actions with positive outcomes.

  • Comment 35, posted at 12.03.16 02:53:49 by fyndraai Reply
    Friend of Sharksworld
    fyndraaiTeam captain
     
  • I fully understand you Rob and you make a solid arguement. But now here comes my concern. Are you not being emotional about this right now? Because we come back to this site time and time again because we trust your Journalistic Abilitie. Are you saying that you dont trust your own abilities? Some people may see the ratings as a way to judge how much work needs to be done on certain player. The ratings can be a valuable source of information. I have never agreed with all your player ratings. What the ratings are is an opinion. One that we value. Lets not be emotional about this.

  • Comment 36, posted at 12.03.16 09:52:36 by AYA11 Reply

    AYA11Under 21 player
     

Add Comment

You must be logged in to post a comment.