How Do You Plan to Use 4th Edition?

[Olefin]

I am talking about the poll data here which had enough responses to be scientific as well as the information I am gathering based on responses on FB and Discord. As a person who uses data for predictive purposes as part of my job what you need is a scientifically valid sample - which we have here and then see how it applies overall.

That’s why you don’t need to survey all the backers to find out what they think of it. Unless you reject how scientific predictive data is gathered and insist that only 100 percent polling of a population is valid for gathering information and making predictions.

FYI polling data and quality data evaluations are made with about that percentage of the overall population or less all the time and as long as they aren’t overly weighted one way or the other are scientifically valid. Your typical political poll for instance tries to gauge the feeling of the American people as a whole and does so with a very small overall sample compared to the population as a whole.

And since FL hasnt done polling of their own what we have to base responses on, to date, is this poll here - which given the number of people who actually post here on a regular basis, represents a rather decent sized percentage of that posting population.

[Rainbow Six]

Originally Posted by Olefin

And since FL hasnt done polling of their own.

Do you have any evidence to support this claim?

[Raellus]

Olefin, I think you may be falling prey to the Fallacy of Small Samples, or the Auditor's Fallacy.

13 "ignore it completely" responses out of 42 poll responses is significant, but it's only a third of the responders here. You could also look at this as 2/3 of the responders here do plan to make some use of v4. The Discord is an echo chamber, so it's not representative of the wider T2k fanbase. Using the Discord subset to prove v4 will be a flop is like using the Free League forum membership to prove it will be a smash.

Regardless, as Rainbow Six pointed out, 13 out of over 8,000 KS backers is insignificant, statistically, so I don't know how accurate your predictive "analysis" is going to be.

-

[Olefin]

Originally Posted by Raellus

Olefin, I think you may be falling prey to the Fallacy of Small Samples, or the Auditor's Fallacy.

13 "ignore it completely" responses out of 42 poll responses is significant, but it's only a third of the responders here. You could also look at this as 2/3 of the responders here do plan to make some use of v4. The Discord is an echo chamber, so it's not representative of the wider T2k fanbase. Using the Discord subset to prove v4 will be a flop is like using the Free League forum membership to prove it will be a smash.

Regardless, as Rainbow Six pointed out, 13 out of over 8,000 KS backers is insignificant, statistically, so I don't know how accurate your predictive "analysis" is going to be.

-

Its actually 45 responses you have here so far and that polling data would be based on the actual number of members here who regularly sign on and post/answer polls. Thus the numbers is a statistically accurate sample of those people here. As you said you attempted to gather data. We have what, 120 or so people who post or comment here on a regular basis? Thus you have a good sample.

And of those 45 the overall positive answers for the V4 as a whole to where they intend to play an actual V4 game with the rules and the setting is very low. Thus this does not bode well for a reboot that states that the V1 and V2.2 are not the future of the game - instead the V4 is that future.

And I would love to see more data gathered elsewhere and intend to do so on my FB group and suggest that on the other one. That will give a larger sample and thus more data to make a better projection for the V4 as a whole. I also plan to contact Tomas and suggest he do a poll as well which may show the difference between the FL market and the existing T2K market as to acceptance and "liking" of the game.

It may actually help them to focus more on what areas people want to see more of - similar to questions that have been asked here of things people are interested in seeing in the fanzine.

[Tegyrius]

Originally Posted by Rainbow Six

According to the Kickstarter website Free League got 8,073 backers.

Not bad for what is basically a non-OSR appeal to nostalgia.

According to GDW's own production figures:

1e boxed set: 97,518 copies
2e core rulebook: 32,180 copies
2e boxed set: 5,240 copies
2.2 core rulebook: 7,757 copies

Don't quote me on this, but I want to say the print run for 2013 was 2,500 copies. PDF sales at the time 93GS closed up shop were Gold on DriveThruRPG, which put them between 501 and 1,000 copies; in the subsequent years, they've gone to Platinum, which I think is between 1,001 and 2,000 copies.

Bear in mind that those 8,073 backers are effectively pre-sales, a concept which didn't exist in the 1e/2e days - that's exclusive of the print run destined for the retail channel. We also don't know how many of those backers bought multiple copies as group purchases, gifts, or retail sales starters.

Originally Posted by Raellus

13 "ignore it completely" responses out of 42 poll responses is significant, but it's only a third of the responders here. You could also look at this as 2/3 of the responders here do plan to make some use of v4. The Discord is an echo chamber, so it's not representative of the wider T2k fanbase. Using the Discord subset to prove v4 will be a flop is like using the Free League forum membership to prove it will be a smash.

Regardless, as Rainbow Six pointed out, 13 out of over 8,000 KS backers is insignificant, statistically, so I don't know how accurate your predictive "analysis" is going to be.

Combining a statistically-insignificant sample with sampling bias is not a research method that would be defensible in my workplace. YMMV.

Some years ago, I rendered some advice to your correspondent. It seems an appropriate time to reiterate part of it:

Originally Posted by Tegyrius

It is an unfortunate geek tendency for individual fans or small local (or online) groups to project their own personal enthusiasm for a property onto the overall global population of potential customers, then make sweeping (and erroneous) generalizations about the commercial viability of their personal visions for said property. In some cases, this failed understanding goes so far as to drive catastrophically bad business decisions.

Intensity does not necessarily correlate to scale. In other words, being louder doesn't make anyone righter.

- C.

[unipus]

You'd have to be a pretty bad data scientist to think you could draw any truly useful conclusion from one poll on one fairly insignificant message board. It would be easy to make catastrophically bad missteps from doing so, though!

@Tegyrius - over what period of time are those sales figures? Some of them are pretty damn interesting.


(I'd also say that FL games in general, this one included, do have a strong if slightly obscured OSR essence to them, but that's another topic.)

[Tegyrius]

Originally Posted by unipus

@Tegyrius - over what period of time are those sales figures? Some of them are pretty damn interesting.

I believe they're for the lifetime of the company.

(I'd also say that FL games in general, this one included, do have a strong if slightly obscured OSR essence to them, but that's another topic.)

Fair point. I can see it with some, particularly Symbaroum and Forbidden Lands.

- C.

[unipus]

They definitely leaned all the way into it with Forbidden Lands, which is a beautiful, beautiful book.

Lifetime numbers make more sense. I wonder what % of the 1st edition sales were in year one, versus up until the release of v2, or maybe even after!

[Olefin]

Originally Posted by unipus

You'd have to be a pretty bad data scientist to think you could draw any truly useful conclusion from one poll on one fairly insignificant message board. It would be easy to make catastrophically bad missteps from doing so, though!

@Tegyrius - over what period of time are those sales figures? Some of them are pretty damn interesting.


(I'd also say that FL games in general, this one included, do have a strong if slightly obscured OSR essence to them, but that's another topic.)

Dont agree with you about insignificant message board - that is not what the forum juhlin is. And you can use the poll as a data starting point - FYI keep in mind that polling company's project political trends for the entire country based on samples of a few thousand people. As I said there needs to be more polling done.

Also the kickstarter numbers only show interest in the new version of the game when it was initially pitched to raise money. The question that needs to be determined now - especially before it gets codified into book form - is did the kickstarter population get what they paid for?

If the numbers here are indicative the answer is no.

That is why they need to do polling or have others do it on sites that have T2K players and fans who have seen the V4 and do it before they go to publication of the books.

PDF's are easy to change - I have done it, Raellus did it, etc. - that happens all the time on drivethrurpg.com releases - but its much harder to do it once you have printed books out there.

I know that for a fact - saw the fun one of my professors went thru in college having to send out corrections on a physics book he wrote that had three pages that had major errors on them that he had missed and another ten pages worth of minor corrections that were significant enough that they had to be fixed.

[Raellus]

Originally Posted by Olefin

Dont agree with you about insignificant message board - that is not what the forum juhlin is. And you can use the poll as a data starting point - FYI keep in mind that polling company's project political trends for the entire country based on samples of a few thousand people.

And they often get it wrong (e.g. 2016 US presidential election projections, both before and even on election day).

Originally Posted by Olefin

As I said there needs to be more polling done.

Absolutely. I commend you on your efforts to collect more data.

Originally Posted by Olefin

Also the kickstarter numbers only show interest in the new version of the game when it was initially pitched to raise money. The question that needs to be determined now - especially before it gets codified into book form - is did the kickstarter population get what they paid for?

Raw sales figures, and the poll here, aren't going to answer that question. To do so, you would have to specifically ask the KS backers, "did you get what you paid for?"

Originally Posted by Olefin

If the numbers here are indicative the answer is no.

Not necessarily. There are a few reasons why someone might have supported the v4 KS. Some supporters clearly wanted a game they could play right out of the box. Others may have always intended just to mine it for bits to use with earlier versions. Others may be completists, who backed v4 because the want to add it to their T2k collection, regardless of whether they intend to play it or not. Some backers may have just been curious. Additionally, some backers might not have had any expectations, vis-a-vis earlier versions (i.e. they just liked what they saw). This is purely anecdotal, but I've heard each of the first three afore-mentioned reasons cited by different forum members (or former members) who backed the v4 KS.

So, to sum up, more, better (i.e. more specific) data is definitely needed if you want answers to the questions you've posed, Olefin. In all sincerity, I wish you luck in collecting it. Please let us know what you learn (preferably by sharing the raw data).

Let's moot this. The OP is about how members here plan to use v4 (or not). The thread is starting to drift OT.

-

[Olefin]

I have seen nothing posted anywhere on the FL site to date that shows a poll or polling results or asking what people thought of the game in response to polling questions (and not just overall comments). Thus there is no evidence any such poll was or has been conducted to date.

They may be in the process of doing such a poll since they just released the pdf version but again nothing I have seen or heard people talking about - i.e. "hey FL has a poll running on the game"

Now that might change and if so then we will have another data point to go on. However using the responses on the FB sites for at least two major T2K groups and on the T2K Discord you can see plainly just from the posts that the overall response has been quite negative.

I will talk to the admins on my FB group about putting together a poll there similar to what Raellus did here to gather more information and data points. I could see what the admins over at the other FB group are doing and if they are gathering something like polling data similar to this.

Thus since nothing has been published by FL of any sort with polling information or questions - or an email saying please go here to answer a poll on the game - that would be evidence of that claim of there being a lack of such a poll, at least to date.

As such all we have to go on with any actual scientific data is this poll and this data.

[Rainbow Six]

Originally Posted by Olefin

And since FL hasnt done polling of their own

So you don't actually know whether they've done any polling or not? They may have done and have just chosen not to share their findings with you?

(Yes I know they haven't and here's the proof or no I don't know will do fine if you want to reply).

As for polling on other sites, I think you should probably bear some of your own comments in mind

Originally Posted by Olefin

Keep in mind the long term FB T2K and discord groups are even more negative than that as to its reception as anyone who has been on them can attest.

Originally Posted by Olefin

FYI polling data and quality data evaluations are made with about that percentage of the overall population or less all the time and as long as they aren’t overly weighted one way or the other are scientifically valid.

So by your own admission these groups are even more negative, which presumably is the same as being weighted one way over the other so as such would not be scientifically valid?

[Olefin]

Originally Posted by Rainbow Six

So you don't actually know whether they've done any polling or not? They may have done and have just chosen not to share their findings with you?

(Yes I know they haven't and here's the proof or no I don't know will do fine if you want to reply).

As for polling on other sites, I think you should probably bear some of your own comments in mind





So by your own admission these groups are even more negative, which presumably is the same as being weighted one way over the other so as such would not be scientifically valid?

The fact that I am on their email list as a buyer of the game, that no one who has bought the game has been posting that they have been asked to answer any poll and that no poll results have been mentioned anywhere by them shows they have not done such polling - i.e. between here, FB, FL's own site, the overall internet T2K community and the Discord there has been no discussion at all of any such poll being conducted.

Thus lacking any evidence at all of a poll being done shows no polls are being done. The community as a whole is pretty close knit - someone would have mentioned it by now - but there has been no mention.

And if the initial draft and Alpha and Beta leaked early than I highly doubt that a poll would be kept super secret that literally no one knows of one.

Thus using logical deduction no such poll, at this moment, exists.

As to the FB groups those groups have many more members than this group has and would offer a bigger sample for gathering data. Thus if the overall answer is negative it would show getting a bigger sample backs up the data that Raellus has gathered. Versus polling the Discord which I admit is highly negative - and thus would be better not polled.

And Tomas has his own FB group that is more positively inclined towards FL and its fans. Thus you get more data, a bigger sample and can see thru the "noise" and get a better appreciation for the actual situation.

Keep in mind for 8000 plus backers, if that is what you need to sample, then as little as 200 responses would give you a very valid sample size. For 140 or so posters here you are gathering data on what people, on this forum, think of the V4 and how they intend to use it.

What Raellus didnt ask and can ask again was of those who responded how many actually were part of the group that either backed the game or have otherwise seen it to date.

i.e. of the 45 have all of them seen it -either by backing it or otherwise seeing it? If say only 30 have actually seen it and 15 are giving responses without seeing it then the sample size is much smaller.

FYI if you want to have a discussion on validity of sampling data maybe we need a thread for that - this thread is supposed to be a discussion of how you would use the V4, not about sampling methods and validity and I dont want to be accused of jumping a thread.