Help support TMP


"On playtesting" Topic


27 Posts

All members in good standing are free to post here. Opinions expressed here are solely those of the posters, and have not been cleared with nor are they endorsed by The Miniatures Page.

For more information, see the TMP FAQ.


Back to the Playtest Message Board

Back to the Game Design Message Board


Action Log

03 Sep 2009 8:23 p.m. PST
by Editor in Chief Bill

  • Removed from Wargaming in General board
  • Crossposted to Game Design board
  • Crossposted to Playtest board

741 hits since 3 Sep 2009
©1994-2014 Bill Armintrout
Comments or corrections?

Connard Sage Inactive Member03 Sep 2009 12:00 p.m. PST

Following the 'worst riles' thread(s).

It must be assumed that rules are play-tested before being unleashed on an unsuspecting world. So how do some of these 'bad' ('bad' being in the eyes of the beholder of course) rules make it on to the market*?

"Well, it only took 10 minutes to work out the damage from a single rifle bullet. I think they're ready for release."

"Hell yeah, the end users can tweak the broken movement system"

"…and we can always post some errata on our webpage"


*I suspect I have an answer to that already

Daffy Doug Inactive Member03 Sep 2009 12:16 p.m. PST

Why should rules designers/writers by any different than the rest of humanity? EVERYTHING breaks down roughly like this: c. 5% of the participants/representatives of a given profession/activity are excellent and account for no less than 65% of the sales/sucesses/"kills"; another c. 15% are competent enough to get the job done without getting "killed" (going out of business, etc.): while the remaining (vast majority) are INCOMPETENT FOOLS who will spend some time in that special level of hell for assuming that they could engage unsuspecting customers in their nefarious incompetency. "Caveat emptor!" has never gone out of vogue….

SECURITY MINISTER CRITTER Inactive Member03 Sep 2009 12:18 p.m. PST

What I've learned over the years as an artist, and may also work for rules designers is this: It worked fine when I designed it, and put it down on paper. Why you can't make it work/figure it out,I don't know.
A good example is the drawings of Rube Goldberg.
link

aecurtis Fezian Inactive Member03 Sep 2009 12:25 p.m. PST

If you are writing rules or ancillary material, and don't involve people who are good at abusing, twisting, and breaking rules, you aren't adequately playtesting.

Allen

Connard Sage Inactive Member03 Sep 2009 12:26 p.m. PST

A good example is the drawings of Rube Goldberg.

Interesting. I can see where Rube got it from

link


Totally OT I know, but it's my thread so nerrr :)

Major Mike03 Sep 2009 12:28 p.m. PST

Many of the play testers are very familiar with the rules. They don't have to read the rule book so they do not see the problems in the organization of the rules or in the composition. Their familiarity allows them to quickly fly thru computations and they may even cut corners. We use to do that with Empire rules. We would work out the number if you rolled a sufficently low dice roll, otherwise it was on to the next item. Your play testors need to be critical, not just your gaming buddies.

Editing must be done with a critical eye to catch errors and omissions and to also iron out logic problems in the composition. Many to easily rely upon spell/grammer check on their pc. Also, writers can get caught up in their project. Items that are clear as day to them may not be to the average reader who has never seen the rules used in a game. Little or no exmaples are provided to illustrate key points or standard proceedures.

For me, a good example of rules writing is Blitzkrieg Commander. Organized logically and well written, a myriad of illustrations and examples of play. I found it very easy to translate from book to gaming table. A bad example would be Panzer Corps. It is a set of rules that has tremendous promise, but, poorly organized and half hearted examples. I felt the author(s) expected the reader to be already familiar with the rules. Never tried to inflict it upon my gaming group.

Terrement Supporting Member of TMP03 Sep 2009 12:50 p.m. PST

As Howard Whitehouse told me after I scoured and tested a set of his rules (paraphrasing)

"I'd never want to play a wargame designed by engineers, but I'll never put out a rules set without having it looked at by one."

I agree that familiarity is a big problem, especially for the things that were "worked out" on the gaming table and are now "understood" by the participants but not incorporated into the rules.

Kinda like some recipes that leave out steps because "…everyone KNOWS that you have to remove the bag of innards before stuffing the turkey!" or similar.

JJ

Darmok Inactive Member03 Sep 2009 1:01 p.m. PST

A few factors that I have encountered in my limited ruleset design/writing experience:

1) Committee Hell: Groups can bring a great deal of synergy to game design. However, getting everyone on the same page can be a chore. It is easy for some of the designers to miss important progress made by others. Sometimes, they just miss very small but critical rewrites. Also, its easy for group members to miss creating small but critical components that, by omission, make the game unplayable: "I didn't write the movement table-I thought YOU were going to write the movement table!" and so on.

2) Going Solo: If you are writing your own ruleset, covering everything with an objective eye can be a real challenge. Often, you will just forget to include (or rework) something obvious because no one is there to remind you. Even playtesters are helping you, the may miss some obviously flawed elements.

3) Getting the playtesters to playtest the actual elements of the rules needing review can be a challenge. Sometimes they misinterpret the rules and use them incorrectly, and sometimes they avoid flawed elements because they don't really add to the game. So when they report back on their experience, they give positive feedback, having skipped over the smelly stuff.

4) Ego. I have been guilty of this one. If you have an "innovative" pet mechanic or other rule element that you are really proud of, it can be hard to accept your playtesters' advice that it stinks or needs serious revision. Or perhaps you try to revise it, when in fact it is beyond fixing.

5) Editing Woes. It's easy to get your edits mixed up when you have half a dozen people working on the project, and in the last thirty days you have 42 different versions floating about. Inevitably, someone will wind up sneaking outdated material into the newer version because they thought they should be using v1.03334 instead of v1.03337.

Admittedly, my experience comes only from community work on free rulesets and mostly involved people who had many other professional and personal responsibilities that took priority. However, many of the obvious errors I have seen in commerical rulesets look very similar when viewed from this perpsective.

Connard Sage Inactive Member03 Sep 2009 1:10 p.m. PST

Thanks for your input, very interesting. However I was thinking more along these lines

Jovian1 posted on 'that' thread

Runner up and tied for Worst rules – Living Steel – where it takes an hour or two to make a character, three hours to figure out where you hit an enemy with your shot as you literally trace the shot THROUGH your opponent – and then after chart hopping for three hours – you discover that your shot instantly killed the enemy.

Allowing for exaggeration, when did all that seem like a good idea? Because you can't tell me that the playtesters could do it any differently?

show some respect for women Inactive Member03 Sep 2009 1:22 p.m. PST

I can only speak for myself.

First you have to find folks that are truly willing to playtest AND give feed back.

Then you have to LISTEN to that feed back.

I completely scrapped the early versions of Watch Your Six three times before I hit on the integrated move system -- which has been quite successful. Indeed the game would never have been published if we stayed with the earlier versions.

That's phase I.

Then you send it out to play testers that don't live in your area. Once again, half of these 'play testers' never provide feedback. They are just looking for a free copy of the rules. Their feed back is especially important because you don't have regular contact with them and indeed, sometimes you don't even know them.

That's phase II.

Finally you need to identify 2-3 folks that are willing to proof your rules that are not part of the original play test group. If these guys find problems, you need to address their issues.

That's phase III.

Finally, have a non-wargaming editor perform a review for layout purposes. This person will find all kinds of stuff. Make those changes and they you are ready to publish a professional set of rules. Hopefully that are crystal clear and easy to learn.

mjc

Grizwald Inactive Member03 Sep 2009 2:03 p.m. PST

Speaking from a background in software development, it is well known that the complexity of the testing required rises exponentially compared to the complexity of the program being tested. It is the same with wargames rules which are effectively the same thing as a computer program. So, the more complex a set of rules, the less likely it is that they have been rigorously tested and the more likely it is that they contain bugs (i.e. are somehow broken).

Which is why I have always (and still do) strive to design rules that are as simple as I can make them while still providing a satisfying game. Not an easy task …

These days, I reckon any set of rules that are more than (say) 10 pages long are too complex.

Grizwald Inactive Member03 Sep 2009 2:04 p.m. PST

"I'd never want to play a wargame designed by engineers, but I'll never put out a rules set without having it looked at by one."

Since I would regard myself as an engineer, I find that sentiment intriguing!

Personal logo Der Alte Fritz Sponsoring Member of TMP03 Sep 2009 2:15 p.m. PST

There is probably an element of "group think" involved among the rules author and his coterie of regular wargamers. They don't see the problems with a mechanism because they are too familiar with the rules and don't view them with a fresh eye.

The Black Tower Inactive Member03 Sep 2009 4:32 p.m. PST

But wargamers like different styles of game.
For any given set of rules some folks will say they are to detailed and others they are not detailed enough!

Some folks think it sould be a game with a veneer of the period tacked on, others that the rules must impose the correct tactics to be played.

Or it could be that they were a lot of fun when played drunk but don't make sense when read sober!

GarnhamGhast Inactive Member03 Sep 2009 4:32 p.m. PST

Rules that are writen from the "I know what I mean so you should too" perspective are very bad.
What I can't forgive is terrible editing and spelling, such that it makes the book unreadable.
I will nail my colours to the mast by saying that I have found some Peter Pig rules to be problematic in both these areas.

RudyNelson03 Sep 2009 11:10 p.m. PST

late at night and cannot sleep, so my comment may be a tangent.

Playtesting is an essential part of the rules design process. It is one area of Homegrown rules development or variant from already existib=g rules that is often overlooked. It comes in several segements.

1. While the initial rule mechanics, a mental excericse on mechanics and how they meet designrequirements (designer goals) and mesh with other mechanics in the system. You cannot have movement based on 1 minute turns and firepower ratings based on a five minute turn.

2. the nthere is the primary analysis of the mechanic on a board with miniature or on paper (or with substitues) if minis are lacking. Visual observation of unit turning, moving, command-control, etc is all better evaluated on a board with actual units.

3. Analysis of data and revising mechanics including dropping some and adding new concepts.

4. Complete system runthrough conducted with comment cards with the primary palytest group. Encourage rules lawyers and snipers to play as well as friends. Using multiple color coded cards for each mechanic makes it easier to group data on each mechanic. Then back to the drawing board to straighten out the kinks and unforeseen issues.

5.Repeat 4. unitl satified.

6. Convention demostrations and playtests with independent players. Use of the comment cards is important for rapid analysis. Be willing to discuss mechanic justification. Doing demos seesions with fewer players keep all of them active and happy and allows you to control the action and quickly answer questions.

Take comments to heart such as 'plays too slow', 'this activity is confusing' when analysising comments.

Game Design is a very long process if done right. Mainily a labor of love for a certain type of minis and/or era of study. So many people comment to me that they do not have the patience to spend on lengthy playtesting.

Personal logo Martin Rapier Supporting Member of TMP04 Sep 2009 3:10 a.m. PST

"It must be assumed that rules are play-tested before being unleashed on an unsuspecting world."

That is an interesting assumption. Does solo playtesting count?

Darryl at Faction Wars Inactive Member04 Sep 2009 4:31 a.m. PST

Once again, half of these 'play testers' never provide feedback. They are just looking for a free copy of the rules.

How true! I think "half" is an optimistic estimate though. ;-)

This has to be one of the most interesting threads on TMP, well to a game designer anyway.

Something that I would add is that it is helpful to observe other people play the game instead of being involved in the playtesting yourself. Also, get people who are real grognards to play the game… the sort of people you wouldn't normally want to play against yourself. They are likely to have a really different perspective on how the game plays to you.

-darryl

Personal logo Martin Rapier Supporting Member of TMP04 Sep 2009 4:48 a.m. PST

"half of these 'play testers' never provide feedback"

OTOH I have been involvoed in playtesting where despite the designer saying they'd like feedback, they only want to hear nice things and nothing negative at all.

Madan Mitra04 Sep 2009 6:03 a.m. PST

Ok,
I've finished writing High Ancients, it's been played by 2 independent groups not from my club, its bin played in my club and among friend a number times…they all want the copy but I won't give anyone a copy till 1 particular person I know has played it. He has an unnatural ability to find the flaws in any system and exploit them to the fullest, until 'he' has played it I won't accept it as finished.

Learnt my lesson from previous times when I have printed out rules and handed them to the group and begun the campaign only to call it off after ‘he' has discovered a fundamental flaw! He's nice that way…you all need some one like that…
Rgds
Madan

MajerBlundor Inactive Member04 Sep 2009 7:03 a.m. PST

I think that too many designers consider gaming among friends "play testing".

The problem is that after months or years of such "play testing" what seems obvious or easy to the friends is actually quite difficult to someone new to the rules.

This dynamic leads designers to bolt on more-and-more rules and to be blind to opaque or counter-intuitive rules.

I've also noticed a reluctance among some designers to break away from clunky rules regardless of player reactions. They become married to the rules that average players find difficult or counter-productive.

MB

RudyNelson04 Sep 2009 10:59 a.m. PST

MB in my list Stage 4 and five were the playtesting among friends. You are right . You will often get a totla different list of issues once you do playtesting among the publis (Stage 6).

I do not often do a lot of public by mail playtesting. For one thing even the basic concepts have not ben copyrighted yet. If designers are concerned over mailing playtest rules, then do not follow that route.

Playtesters deserve copies of the rules that they work on.

RockyRusso Inactive Member04 Sep 2009 11:26 a.m. PST

Hi

Or the opposite. Gamers who read a known phrase and somehow reach an odd conclusion. I have had various rules out there, and at conventions when someone argues with the GM, I usually start with "what does the rule say?". Often the player in his group forgot what the rule actually said for what they wanted it to say.

The silliest example happened a while ago in the Bay area. Big M&M game(I never play games at conventions, especially with rules I have worked on). Big fight, guy is yelling at a gamer then the GM.

They call on me.

Guy looks me in the eye and says "look I know what the rule says, but I am a close personal friend of Rocky Russo, and a member of the original playtest group. And what the rule is supposed to mean is ….."

And I said: "do you want to READ THE NAME TAG $%%^".

Yup, I am Rocky Russo, and I have never seen you before in my life!

Gaming groups commonly have a rules bully who invents something under an odd pretext to win a fight, and then it becomes DOCTRINE. Rules problems are not always the fault of the designers!

Rocky

MajerBlundor Inactive Member04 Sep 2009 12:15 p.m. PST

I do not often do a lot of public by mail playtesting. For one thing even the basic concepts have not ben copyrighted yet. If designers are concerned over mailing playtest rules, then do not follow that route.

Playtesters deserve copies of the rules that they work on.

I've not published any rules but to "play test" my home grown rules I do enjoy running them at our FLGS and conventions with strangers/new players.

I look to see how quickly the players grasp the rules, how quickly the game moves along, and to what extent the rules meet gamer expectations (eg firing an MG at a heavy Mech should be futile; MGs should suppress infantry better than RPGs; RPGs should be better at killing infantry in hard cover compared to small arms, etc.

If gamers express surprise at a game result, are confused by a mechanic or forget to implement it, or certain mechanics seem slow or cumbersome, I go home, ponder the problem, and seek advice on how to fix it here on TMP! :-)

MB

Personal logo McLaddie Supporting Member of TMP05 Sep 2009 2:55 p.m. PST

There are courses that address 'playtesting' among the requirements for a Bachelor's Degree in Game Design, as well as fieldtesting in degrees in simulation design.

They have a number of overlapping processes, but some of the main principles in both are:

1. Pre-planning to establish what is actually being tested.
2. Methodical and detailed records of issues found in a
series of tests before anything is changed to address
them.
3. A clear set of concrete game goals, so success can be
identified when it is achieved.
4. Clear instructions to the participants as to the
purposes of the playtest [specific, not general], and
the need to identify issues with the rules/play, but not
to involve themselves in re-design while they play.
5. Playtesters should, like witnesses to a crime, not be
allowed to communicate until all the feedback is
received, as 'impressions' from other players'
debriefing can color their own.

What this means is:
1. Play-testing has to have a specific design focus, not just everything at once. If the play-time and combat resolution are the issues, then focus on that. What can happen is during a series of games, the designer, without any focus, sees a problem in one game here, but another game there, tries to fix them even while the third game finds another issue. Never asked is whether the issues occur in all three games.

2. Several playtests need to be done before anything is changed. This is too often done in any kind of testing. It changes the game/design, which means the next play test could be a very different game because of that one test, and if not clearly identified, can impact parts of the design unexpectedly, or be overlooked. Change the combat resolution just a little, and the speed of the game is slowed, or the fact that now casualties are 20% higher are overlooked, and after several more tests, no one know when or why the design now creates more casualties.

3. I have seen designers, who wanted a strong command system, respond to playtesters issues, mostly tactical, to the point that the design is no longer about the command system. Without clear goals, information down the line can skew perceptions and you end up someplace you don't want to.

This is where a simple design can get weighted down with 'stuff' that play testers see as needs, becoming complicated, messing up what was a clean design.

Goals help. You can't get to where you want if you don't pay attention to the direction you're going.

4. Playtesters are there to make the design better, not to design their own game. Often while the purpose of the playtesting is to determine how the design plays as a game, the playtesters are focused on the history or representations in the game. Two different design issues, and combining them can create all sorts of problems in getting the design information needed. The designer has to make this clear to the playtesters: one or the other, but not both at the same time.

5. I was involved in a playtest situation a while back, where everyone playtested a rules set and then gave their responses on a website put up for that purpose. There was no focus to the playtesting, any and all issues were fair game. The first posters defined the issues and the rest, if they had had time to playtest yet, chimed in. Those who hadn't, of course focused on the issues brought up to the exclusion of any other impressions--or the playtesters didn't bother to play because others seemed to have 'got all the info the designer needed. It was half the process it could have been.

Most gamers don't want to go to all this trouble, which is understandable, but what you get are playtesting results with a lot of unperceived holes in them that only become obvious after the design is published. Often with in-house rules, those holes are overlooked by 'group think' or fixed while playing, as only in-house rules can be, and the 'fix' is never really 'playtested' outside the group--or even written down, becoming covert understandings rather than overt.

One that comes to mind in a published set of rules is Volley and Bayonet. The rules had units contact enemy units at whatever angle they were in at the point [and it was often a point] of contact. Then any retreats were straight back from that position. Well, the rules never really said that. So, on line it was mentioned once, and some folks got it and some didn't. Almost six years later, gamers in various groups were still 'squaring up' stands for combat, and everyone expressing surprise that some groups were doing it differently. I have several experiences of this in gamers' groups with rules, both on-line and face-to-face.

It is easy to over-look without blind playtesting.

Best Regards,

Bill H.

RudyNelson05 Sep 2009 4:44 p.m. PST

Mcladdie,
Concerning your initial comment on simulation design training, Actually in the field of 'Military Science' there were course curriculum that included simulation design. I got my BS in MilSci back in the 1970s so I am not sure but I would imagine that that is still the case.

Even in the US Military, when they first developed Simulation Centers for each post, officers from each Battalion attended work-shops on suimulation use and design. I was fortunate to be a rep for the 1/9 Cavarly Sqdn. At Fort Knox during the Armor Officers basic Training course, we used the Dunn Kampf often. My first exposure to micro-armor as each training set had GHQ tanks in them.

When I was the S4 for an MI Bn, I had the keys for the Fort Riley simulation center. Quartermaster use of simulations was a logistical focus rather than the tactical focus that I experienced as a combat line officer.

I am sure other TMP posters like Allen C. and Bill G who both were MI officers received a lot of exposure to simulation use as I did.

Mcladdie your comments on the overlapping nature and use were very good.

Phil Dutre07 Sep 2009 11:12 p.m. PST

Playtesting is very hard, and I concur with many of the comments made above.

Anyone who has seriously tried to test (house)rules, quickly realizes that most published rulesets probably didn't undergo any thorough playtesting at all.

It's one of the reasons I am very suspicious of army lists claiming to be a tool to set up fair games. I see the publishing policies of major companies (read: GW), by constantly twiddling and adapting their army lists as proof that not much playtesting went into their army lists in the first place.

Not many people understand the statistics and interdependencies of their own rules mechanics. Results are not always linearly dependent on the variables that are put in, but many designers and players treat them as such.

An interesting question in a given attack/defence scenario always is the following: suppose you want to play the scenario again, but you want the result to be such that the attacker would have caused twice the amount of casualties as he did the first time. How much stronger do you have to make the attacker's force to make this happen?
The answer is almost never 2…

Sorry - only trusted members can post on the forums.