Know-How Exchange : Research/Metrics

Hi Brian,

Your model looks fine (with the available information), but let's interpret the relevant numbers:
1) the coefficient estimate is negative! So you do actually find a NEGATIVE relationship between x and y, consistent with your eyeballing the data
2) this coefficient estimate is significantly different from zeor, so you are correct in rejecting the null of no effect
3) the F-value of the total model is significant, so your model indeed helps you to explain the dependent variable
4) is the R-squared too low? It really depends on your perspective. In marketing academia, consumer behavior researchers typically get R-squares of less than 0.10 (if they would report them :-)), while quantitative researchers like myself get R-squares of over 0.80. The main reason is that quant researchers often have data on more variables (eg prices of the company and competitors, etc.), and thus can explain more variance in sales than models that only have e.g. the city market's average demo- and psychographics. By the way, it is easy to increase your R-square in time series models by including Y(t-1), e.g. past sales. Is your model now necessarily better?

In short, I do not think your model is garbage. The low R-squared does indicate it is a good idea to add more independent variables in order to better explain (and predict) sales. At the same time, check whether your data is far away from normal, in which case transformations may help you to get a close-to-normal distribution. You hint to this by saying there is an upper bound of 5 for units, which makes me think that other variables also have such bounds.

Brian,

let me ask one question - what type of variable is the dependent? Is it ordinal or continuous? I ask because if the data is ordinal, OLS regression may not be the best way to analyse it.

On the basis that the data is continuous and represents some measure of "sales", I've not seen R-squared measures this low in this type of model. At the very least, if you are explaining this in a commercial environment (as opposed to an academic one), you will face some challenges explaining the validity of what you have done here. Rightly or wrongly, very few people will feel instinctively comfortable with your results.

At the very least, the R-squared value should tell you that there are many more important factors which can be used to explain performance.
John

Hello Brian,

Is there a way to have a look at your data? E.g. can you post ~10 data points here? To protect your data it’s ok for me when you rescale them. I’m only interested in the visual impression you talk about.

A counter intuitive result can appear, e.g. when you try to fit a linear function (y = mx + b) to an inverse function (y = p/x + q) very strange numerical things can happen and usually have large (residual) errors and low R-values: both functions simply don’t fit well, at least over a wide data range. They can fit well over a short range.

Which brings me to the question what you are trying to accomplish? Are you looking for the best possible fit, i.e. the best possible description of findings? Then you should select a fit-function which exhibits the important properties of your observed data (don’t fit each and every peak). This can be a data transformation (y -> 1/y or x -> 1/x) as blanalytics states, it can be a shift, a polynomial (better: don’t) or other functions. Depending on your purpose you may even argue that a bad linear fit is still good enough.

These are fundamental questions you have to answer more from an engineering perspective. It’s an input, a decision you make. Statistics can’t answer this for you. - Identifying a variable as a significant contributor is a different story. A way to do it is to start with fitting the most prominent variable to your data and fit the residuals to other variables, step-by-step. However, this is not the best way to do it.

This brings me more to the research perspective. Is this a new research field? Then you can only look for a best, plausible fit as I described above. Do you have (let’s exaggerate it a little) ‘scientific’ reasoning available for this problem? E.g. in physics we know something about forces and gravity and we can expect a certain ‘law’ in measurements, e.g. y = m * x^2, where y is a distance and x is time and we would try to fit a good m-value (using y = m * x^2 + n * x + o would be a more careful approach). So comparing data to this expectation would be more than reasonable.

In other words: can you base your intuitive expectation on some, commonly accepted, reasoning and formulate it as some fit-equation? That would be a good starting point for refinements.

Other questions I have are about the quality of your data and about reliability of future predictions you may want to do. If your data are unreliable, statistics will report this to you in various high-error messages. No matter, how good your fit turns out to be today, how will you know how good it will be tomorrow, when applying to a different (data) situation? (I could give you the route to those results, but that is another story.)

Hope this helps,
Michael