This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Forum: How do you decide if course feedback is actually trustworthy?

Hi everyone,

I was wondering how other people handle this for scientific training courses.

For our workshops, we try to keep it simple: if a participant gives permission, we publish the feedback with their real name and institute, where available. Google reviews stay as they are as well. We have never asked for a review to be removed.

We also think that not every comment needs to be perfect. Critical feedback can be the most helpful part, because it shows us what we should improve next time.

Does this sound like a sensible way to do it? Or have you seen a better approach for making training feedback more transparent and useful?

We’re always looking for ways to improve, and this has been discussed internally; now we’d like to hear the community’s views. Is there anything we can do better here?

Thanks for your honest feedback or your ideas.

trainings ngs feedback workshops
• 266 views • link
updated by GenoMax GenoMax • written by ecSeq Bioinformatics • ecSeq Bioinformatics

Not my particular area, but do you really need to make all the reviews visible individually? You could solve your problem by aggregating feedback and determining the signal from the noise of real consistent themes and offer summaries you've synthesised of what you're hearing?

• 0 views • link
written by Joe Joe

That is a really good point, thanks.

The reason we currently show individual feedback is mainly that we do not want to selectively interpret it ourselves. People can see the original wording and decide what they find relevant.

There is also a practical reason: a few times, people told us they noticed a review from someone they know personally or from an institute they trust, and that was what first made them look more closely at our courses.

But I agree that an aggregated summary of recurring themes could be much more useful, especially for people who do not want to read many individual reviews. For example: what participants consistently found helpful, where they struggled, and what we changed based on their feedback.

Maybe the best approach is both: keep the original feedback visible for transparency, but also add a short summary of the main patterns. That would probably make it easier to understand the overall picture.

Thanks, this gives us something useful to think about.

• 0 views • link
written by ecSeq Bioinformatics • ecSeq Bioinformatics

1 answer

While the reviews you post on your website and Google reviews look excellent, I don't believe this testimony-type of review is particularly useful. First, people tend not to trust them, in particular Google reviews, and if you use them for promoting your service, you can state a thousand times that you are honest and people will still doubt them. On the other hand, they don't give you the information you need to have to improve the learning outcomes of your courses. They are unsystematic and necessarily subjective and don't hold up to sociometric standards. You may want to take a look at how Software Carpentry, a highly successful not-for-profit project, does its surveys. The main point is that they are doing anonymous pre- and post-surveys to assess their courses.

The main points you need to better understand:

  • Background, demographics of the participants
  • Effect on self-reported skills and competencies pre- vs. post-course
  • The Net Promoter Score: How likely is it that you would recommend this course to others?

The last one is arguably the most relevant indicator to check if you are doing ok (and may somewhat correlate with Google star ratings).

So, to answer the feedback itself may be trustworthy, in particular, I have rarely experienced dishonest responses from course participants; frustrated, subjective, yes, but that's ok. But it's not the kind of feedback you or your customers need.

• 0 views • link
written by Michael Dondrup Michael Dondrup

This is a very fair criticism, and I think you are right that I was mixing up two different things.

The public testimonials are mainly there for prospective participants. They are not meant to be a rigorous measure of learning outcomes. For some people, it is useful to see that somebody they know, or an institute they trust, attended one of our courses. But that does not tell us in a systematic way whether participants actually improved their skills.

For improving the courses, anonymous and structured feedback is much more important. I really like the idea of separating this more clearly: a short pre-course survey to understand prior experience and expectations, a post-course survey to measure perceived changes in confidence and skills, and maybe even a follow-up after some months to see what participants actually used in their work.

I also agree that recommendation scores could be useful, although I would probably see them as one indicator among several, rather than the whole picture.

So yes, I think your point is very helpful: public feedback can be useful for transparency, but it should not be confused with a proper evaluation of learning outcomes. I will have a closer look at how Software Carpentry does this.

Thanks a lot for taking the time to write such a detailed answer.

• 0 views • link
written by ecSeq Bioinformatics • ecSeq Bioinformatics

For improving the courses, anonymous and structured feedback is much more important

The anonymous part is essential to receive frank feedback/criticism. In general, people are going to be reluctant to be either, especially after spending a few days in company of your instructors. You should also consider having a third-party host these evaluations, so the identity of respondents can be kept confidential. Some incentive (e.g. gift card for a small sum) may also entice users to complete the feedback.

You seem to run a lot of these so there must be enough positive referrals from people who attended prior courses. Do you also get a significant number of "repeat" customers?

• 0 views • link
written by GenoMax GenoMax

Thanks again, this is another very good point.

We do allow anonymous feedback already. Participants can decide whether their comment may be published, and whether it should appear with their name/institute or anonymously. But you are right: even with an anonymous option, people may still feel hesitant to give frank criticism when the evaluation is collected by the course provider.

Using a third-party system would make that separation much more credible. We will seriously consider this, because the goal should not just be to collect nice comments, but to get feedback that helps us improve.

Regarding repeat participants: we do see this quite often, although we have not analysed it systematically yet. Many people start with our introductory NGS course and later join one of the more specialised downstream courses. Participants who are especially comfortable with computational work also often come back for our Nextflow course.

We also regularly see former public-course participants bringing us into their own institutes through an on-site course booking. That is probably one of the strongest signs that they found the course useful.

But you are absolutely right that we should not rely on impressions here. We will start tracking this properly, so we have real numbers rather than just anecdotal observations. Thanks for pointing this out.

• 0 views • link
written by ecSeq Bioinformatics • ecSeq Bioinformatics

Log in to answer this question.

More posts like this