Designing Interface Studies you can learn from

"do you like my interface?"  -- not very specific, giving a lickert scale, this is useful, agree/disagree, and there is a please the experimenter bias.
Developers are valuable testers.  Being the developer and the tester can be valuable. 
Getting beyond "do you like my interface?"

What's the comparison?
What's the yardstick?
When measuring effectiveness, even informally, good to have some comparison. 
Start out with a baserate question: how often does Y occur? requires mesuring Y
Correlations: Do x and y co-vary
requires measuring x and y
Causes: does x cause Y?
requires measuring x and y, and manipulating x
also requires somehow accounting for the effects of other independent variables

manipulations
independent variables--- indepenend of what the user does, in control of the experimenter
measures
dependent variables--what the user does measured-task completion time, create an acoount
accuracy, how many mistakes, how much do they remember afterwards, how do they feel emotionally afterward
precision
internal validity-if you ran it again, would you see the same results, to have a precise experiement you need to be able to remove the confounding factors and study enough people

generalizability
external validity--does this apply only to certain age group, or all?

ASK: is my approach better than the industry standard?

challenge--compare new cool, fidelity of implementation, and the new approach, or some combination
how to tease apart causal factors? when is it more relevant? to decide between 2 cameras-image quality or usability may be less relevant.  when you are a designer, you do have control over the variables
benefits and drawbacks of the QWERTY keyboard -iphone
manipulation; input style
measure-words per minute
however, imp to realize that it is important to realize that old phone users are expert at their older devices, to not be used to something the first time, may not be important. 
IS THIS DIFFERENCE SIGNIFICANT?
authors of user centric study-- did another study-- found that QWQERTY and iphone were about the same-- same speed, but iphone make many more errors.  however, in 2007, may have different kinds of people if you used different people for both comparison pieces of the study--potential for variation is greater.

strategies for fairer comparisons
insert your new approach into the production setting
recreate the production approach in your new setting
scale things down so you're just looking at a piece of a larger system
when expertise is relevant, train people up (give them the practice they need)

IS INTERFACE X BETTER THAN INTERFACE Y?
what dfoes better mean? in a complex system, several measures needed.  wht are you trying to improve, the answer is mostly it depends, and what does it depend on-gets to the goal of the interface, for what???
controlled comparison enables causal inference  (you can learn from what is causal to make a better decision going forward-