Combatting Flawed Questionnaires

So what should one do if one is asked to fill out a questionnaire with flawed questions?

The answer depends upon a few factors.
If there are a significant number of flawed questions so as to make the whole process flawed, then you should make this known. Contact the questioners and see if they are receptive to withdrawing the questionnaire. If not, you have few options, all of which are less than satisfactory.
If there is a minimum return rate, so that boycotting the questionnaire will have an outcome, then simply failing to return your questionnaire is an option. In this case however, you will have to hope that enough people have noticed the flawed questions and will boycott it. You may need to spread the word!
If the questioners  have not set a minimum return rate, so any number of returns will count, then your only option is to make your views known via the questionnaire itself. You could write ‘Flawed Question‘ across the answer area or in the comments. The trouble with this is that you are relying on the questioners to recognise that it is a flawed question and ignore everybody’s answer. For instance, taking the example of:
‘Have you had experience of motorbikes riding on the heath?’ we know that it is impossible to find out from this whether we should encourage or discourage motorbikes (without another agenda) because opposite sides of the same argument will attract the same answer. Maybe ‘Withdraw this question’ should be written if you think that no answers to this question should be used.

Unfortunately, much work is put into writing questionnaires, whether flawed or not, so those involved will be unlikely withdraw questions or questionnaires. In this case, you are simply forced to accept the will of those in charge with no recourse.

Please feel free to comment below…

4 thoughts on “Combatting Flawed Questionnaires

  1. Anna Aukim

    The questionnaire is flawed: Answers to many questions (such as the one quoted above) will not bring any real information on which to base decisions. Having no lower limit on returns is a fundamental mistake. How else can the foremost aim of the process (to better represents view of the residents) be measured? Comparing it (favourably) in advance to a similar process 10 years ago that brought an 85% response rate is extremely hopeful (at best). A more recent example of opinion gathering showed a 85% NON RETURN RATE. This figure should be taken as a benchmark to be improved on. It’s a pity that lessons from the QL saga don’t seem to have been taken on board. There seems to be a wealth of available technical/other assistance in the village…perhaps the PC could have harnessed some of this to try and make village matters more relevant and inclusive to a wider audience? (especially to the younger generation…what about having a youth councillor to sit on the committees?) It’s a wasted opportunity. Surely any and all forms of communicating should be used when trying to gather as much as possible from as many as possible…it’s not an election and there isn’t a time deadline!

  2. Ian Kay

    It would be a great shame if anyone failed to return the questionnaire because they found a few questions flawed. There are many questions that simply gather information which is useful, such as how people use Church Field. If you find a question flawed simply write ‘flawed question’ in the comments box, or if there isn’t one, at the end, maybe explaining why.
    None of the examples of ‘flawed questions’ you Steve have given make any sense. ‘Do you want more Tree Preservation Orders?’, ‘Have you had experience of people turning in your drive?’ and ‘Have you had experience of motorbikes riding on the heath?’ aren’t in the questionnaire (I don’t know where you got them from – your imagination?) Unfortunately Anna A has picked up on this and now thinks the latter is actually there (presumably without checking).
    Obviously where we are asking for people’s opinions the number of answers will be relevant – the more answers the more weight we can give the results. But I see no point in stating in advance that if we get less than x% response to the questionnaire we will ditch the whole project. x would be a completely arbitrary figure. Comparisons with the QL survey are irrelevant – it is very likely that most people couldn’t care less about QL so didn’t bother to fill in the questionnaire. Maybe that applies to the village generally, but I hope not.
    We did try to get young people on the Steering Group and one if fact did join us, but other things prevented him from staying.

    1. Steve James

      Thank you for your comments.

      What is the use of asking people how they use the Church Field? Would it not be better to ask what they want the church field to be used for? That way, you gather opinions for the future and include the opinions of those who currently don’t find it useful.

      I was particularly trying to invent questions so as not to steer people directly to particular questions, but to allow them to decide for themselves whether questions were flawed or not – so yes, they came from my imagination. There is actually a question asking about increased TPOs, question 45. Maybe Anna is better at grasping the overall point and being able to analyse a questionnaire. I am disappointed that you think that all of the points are irrelevant.

      The motorbikes on the heath question has relevance to (for example) Question 33, which asks:
      In the last two years have you had personal experience of any of the following:

      • Mooring taken by another vessel
      • Unreported Damage to your moored vessel
      • Speeding on the River
      • Hazardous manoevre, e.g. someone taking or crossing too close
      • Dinghy Stolen or “borrowed”

      …etc

      I experience speeding each time I use the river as I drive over the speed limit. This means that I must answer ‘yes’ to this. Anyone that races (adult) sailing dinghies at WSC must (if they are being honest) also answer ‘yes’. I guess that those who object to speeding on the river it will also answer ‘yes’. This means that people who want to exceed the speed limit and people who want the speed limit enforced will give the same answer. How then is any conclusion to be drawn from this question?

      Likewise, I have had experience of another boat on my mooring and having my dinghy borrowed. These are not a problem to me but could be to some people. How are you going to tell the difference?
      I suspect that any ticks in these boxes will be taken as action is required against the speeders, the mooring takers and the dinghy borrowers. If not, what conclusions will be drawn? If I am correct however, how do I register that I don’t want enforcement?
      As explained in my text, it is not simply a case of not filling out that question as we need to know what conclusions will be drawn from everyone else’s answer so that we can agree or disagree.

      It is a pity that you see no point in stating a minimum return rate in advance because I have explained it on the page. To repeat however, people have said to me that they are so annoyed with it that they will simply throw it away. If there is a minimum return rate, throwing it away will be making a statement that can have a result. With no minimum return rate, throwing it away will simply weaken their view. People need to know beforehand if their protest will be recognised or ignored as in QL.

      I find it strange that you obviously care about this enough to reply, as you have with other points, yet you don’t accept that there is a whole section of the community who do not support the direction of the councillors. I do not know how to address this but would prefer to have the support of the council in trying to address this rather than ignoring it and ploughing on regardless.

      As mentioned by Anna, it is strange also that the steering group didn’t call on the wealth of ability and knowledge to make the best possible attempt at a successful questionnaire. If I were in your position, I would have enlisted those with a passion for technology to solve the technical challenges, those critical of questionnaires to review the draft and those who may not share my opinions to check for impartiality before it was published.

      I would ask that you think carefully about what conclusions you hope to draw from the answers given and completely discount those with no absolute conclusion.

      I am more than willing to meet with you and/or Janet if it would help. I realise that if you intend to plough on regardless, no help will be required.

  3. Ian Kay

    I should have made it clear that my previous posting expressed my personal views, not that of the Parish Council or the Parish Plan steering group, who I didn’t consult before writing it.

Leave a Reply