"UMass CTF 2021 Postmortem"
@jakob I pretty much agree with your self assessment. Been there, done that. Although I didn't end up releasing my equivalent of a haskell challenge, I didn't feel comfortable with putting up something I couldn't solve myself, lol.
You mention that playtesting is important. True, but it's like any other kind of testing. The more you have a person testing things, the more they become accustomed to the testing process and may end up unsuitable to perform any further testing for a number of reasons:
- They learn what to avoid and what to do, thereby not testing things thoroughly. For example they'd no longer bother trying out things that may take a lot of time because they don't have much time to spend.
- They become fatigued by testing, be it because they've already been designing CTF challenges or because they've been testing too much and started to hate it. Personally I'd recommend to take as much of a break as necessary the moment it becomes drudgery and ceases to be fun.
From what I've read in other media about usability testing, the solution to this is spending lots of money to keep recruiting new testers and cycling them (hello Apple). This is expensive and not an option for most institutions, let alone groups or individuals doing this for fun. Some recurring CTFs are organized by different teams though...
@wasamasa I am worried about the testing fatigue, which is why I'm proposing a full month where no challenge development occurs besides playtesting. We have a sizeable team of people working as staff, too, so I hope we'd be able to distribute the work in a way that doesn't cause too much stress.
I'll need to think about how to deal with the first point you've brought up. It might involve bringing in more volunteers, like you suggested, but it's a little hard to find ones with the right skills.
"I appreciate SDF but it's a general-purpose server and the name doesn't make it obvious that it's about art." - Eugen Rochko