The Survey Says

Now more than ever, small business owners rely on information to make sound business decisions.  Whether trying to determine the market potential of a new product or service, or to gain a better understanding of needs within their target market, this information is not only nice to have, but essential for small business success.  While there are many ways to obtain this valuable information, surveys remain a leading method due to their convenience and relatively low cost.  However, business owners must also be aware of the short comings when considering conducting surveys and interpreting the results.

 

Case Study

In an effort to teach students a real-world example of the scientific method and probabilities, we conducted a brief study on Facebook where we simply stated our hypothesis, that >10% of the population owns a small business. We then asked my network of over 600 individuals, “Do you currently own a small business?”

In the end, 35 responded “Yes” while 64 responded “No”.  There are plenty of interesting statistics and conclusions we can draw just on that alone.  First, nearly 35% of respondents own businesses. This appears to crush our hypothesis that a mere 10% would.  Further, we can also claim a 16% response rate (99 respondents / 607 friends).  If you’ve ever run a survey you may agree that a 16% response rate is very good!  But between the lines we’ll find the real insights of this little study that all business owners should understand!

 

Hitting the Target

Whenever we collect information for decision making purposes, we want to ensure the validity of the data to minimize the risk of making a bad decision.  The two aspects we are most interested in is accuracy and precision.  Accuracy defines how close our estimate is to hitting the target of the true value we would like to know.  Precision describes how close individual values may be surrounding our estimate.  Below is a classic depiction of accuracy and precision.  Of course, we would like to ensure we are both accurate and precise so we can hit our target and make a good business decision.

So how do we do this?

 

Representative Samples

To ensure accuracy in a survey, we want to reduce or eliminate any potential sources of bias.  Bias is defined as the distance the average estimate is from the true target value.  We want to ensure that there is nothing that will systematically cause our survey response to sway one way or another.  This is a lesson learned following the 2016 presidential election.  Polls suggested a democratic win, yet the republican candidate succeeded.  How was this possible?  The most plausible explanation was the inadvertent sampling bias among polls that favored the opinions of left-leaning, densely populated metropolitan areas but under sampling of the less densely populated rural areas that tend to be more right-leaning.

This brings up the discussion of a representative sample.  Just as in the polls, as small business owners we cannot survey every potential customer on their wants and needs so we attempt to identify a representative sample, a subset of our target market that we believe would represent all potential customers.  In our example, one of our participants commented that my social network is most likely not representative of the entire American population, and he is exactly right.  As such, any claims that we make from our data will be limited to the 607 Facebook friends that I have and cannot be extrapolated to the entire American population.  Thus, I can say that approximately 35% of all my Facebook friends are small business owners, or roughly 212 Facebook friends.  Or can I???

 

Response / Non-Response

There are two other sources of bias that can creep into our survey results – response and non-response biases. The first is response bias, which is defined as the bias that can occur if respondents do not reply truthfully.  This could be either intentional or unintentional, based on the question.  In the intentional case, respondents might feel a bit of peer pressure to respond in a certain way.  Consider if I instead asked for annual revenue figures.  Respondents may feel inclined to inflate their figures in an attempt to demonstrate their business success.  The inadvertent case may be more attributed to the survey design itself.  My simple question of business ownership may not be completely understood (at least as I intended) and respondents may answer differently due to a misunderstanding of the question asked.

Non-response bias is slightly different.  The non-response bias can be introduced if friends that tend to participate are different than those who tend not to participate.  In our current example, it is possible that small business owners may be more inclined to participate than non-business owners.  We are a different breed and take great pride in raising our hands and claiming, “I am a business owner”.  There is also the possibility that not all of my 607 Facebook friends are actively on Facebook, but the small business owners are in order to keep current and connected.  Speculation at best, but certainly another possible form of bias that could artificially inflate my 35% estimate.

 

The More the Merrier

Enough on accuracy and bias for now, what about precision?  Letting curiosity get the best of me, I was watching closely immediately after posting the survey.  As the first few responses rolled in, I felt I was on a bit of a survey rollercoaster since each response had the ability to swing the result by several percentage points.  It wasn’t until later in the week that the number seemed to stabilize. We claimed before a potential 35% estimate.  However, this single figure is what’s called a point estimate and only tells part of the story. Going back to our polling example, very often we will hear a stated poll percentage followed by a margin of error.  What is this margin of error?

While a point estimate provides a good estimate of the average response, the margin of error defines an interval that we can have a high level of confidence the true value will fall between.  In our example, the estimate of 35% includes just under a 10% margin of error (at 95% confidence).  What that means is that we can claim with 95% confidence that the proportion of my Facebook friends that are also small business owners is somewhere between 25% to 45%.  In this case, I can certainly say that my hypothesis of >10% has been demonstrated since the entire interval is well above 10% (barring the earlier biases discussed).

But that’s a pretty wide margin!  What if I wanted to make a decision where the costs incurred were relatively high and a wide margin could either make or break the budget?  In order to narrow our margin of error, we would simply seek to collect more survey respondents.  Qualitatively, the more information we have (via more respondents), the more confident we can be in the results we obtain (again, barring any bias issues).

Below is a simulation of the results we might expect to get if we were to repeat this survey 10 times. Each line represents the percentage of small business owners as calculated as the survey responses were to come in.  The red dashed line forming a bit of an envelope around the data represents the margin of error, which starts off wide with just a few responses and decreases as more responses are added to the result.  Note that a few points exceed the margin of error.  We stated earlier that we were 95% confident in our estimate.  These out of margin points represents the 5% error that we may make in our decision.

 

Conclusions

While the actual results of our case study are inconsequential, the insights we can gain through the data collection and data analysis is paramount for any small business owner. In the end, or final claim is that we are 95% confident that the percentage of my Facebook friends who own a small business is somewhere between 25% to 45% (roughly 150 to 270 friends).  If we were interested in a more precise estimate, we could increase the number of responses by allowing the survey to run longer.  However, this number may be overstated even for my group of friends due to the sampling methods involved.  Thus, if we were looking for a more accurate estimate for a population beyond my circle of friends, we would have to ensure we have a true representative sample for the population we are trying to learn about.

Small business owners who are using information and data analytics to drive key business decisions should definitely consider the accuracy and precision of their data collection methods prior to making any key decisions.  In many cases, it can prove to be crucial to understand these principles and how to apply them to help avoid making bad decisions.

 

Sharing Is Caring!