Judge by Real Data
"I find it good to use", "my friend says it's nice", "our user numbers are going up" — none of these are data. Only one thing settles whether a product works: the recorded behavior of real users in real scenarios.
What you'll run into
- 靠「自己人觉得」下结论，几个人觉得好就推上线
- 问卷收回来几十份，就当成规律往外推
- 功能上线了，但没有行为数据，说不清用没用、用到哪一步
Why small samples don't count
The problem with small samples and your own team's feelings isn't that they're "unreal" — it's that they lead you astray: they hand you the comfortable answer in the direction you were already hoping for.
- Insiders are people who "already know they'll use it". Whatever you make, they say it's good — because what they back is you, not the product. What you actually need is the people who still need convincing.
- Small-sample noise can support any conclusion. Two out of three people clicked a button and you read "sixty percent" — but that's just rolling the dice three times. Small enough, and numbers reflect luck, not rules.
- Asked face-to-face, people say what they think you want to hear. People tend to perform cooperation, rationality, helpfulness. "I really need this feature" doesn't mean they'll actually use it next month.
- A short-term bump from a campaign isn't the product's doing. Movement has to be attributable. If you can't tell which link in the chain caused it, a rise can't be repeated.
What counts as real data
Numbers are numbers, but some count as data and some are just noise. Four criteria:
- 看行为，不看表态。用户实际做了什么，比嘴上说了什么可靠得多。问「你需不需要」不如数「有没有人用、用了几次」。
- 看完整路径，不看单点。「浏览量涨了」没有意义，要看从进入到完成的整条链路里，人在哪一步停住。
- 样本要有量级。至少能区分「波动」和「趋势」——个位数的样本，除了暴露问题，给不了结论。
- 是真实场景。他自己用、没人在旁边盯、没有奖励诱导。测试环境、演示环境里的数据都不能算。
When you have no data
"I just don't have real data yet" — that's the most common situation, and it has an answer:
- 先埋点，后上线。没有埋点的功能，上线等于没上。把埋点当成功能的一部分，跟着一起验收。参见先埋点后上线。
- 先放一个小版本给真实用户。小流量的真实行为，好过一百个自己人的感受。窄受众、最小版本，都是为了让真实数据早点来。参见最小切片。
- 别用问卷代替行为。问卷能告诉你「他说他想要」，行为能告诉你「他到底用不用」。两者配合可以，但问卷不能当行为数据用。
- 先定要验证什么，再定看什么数。北极星定下来之后，才知道哪些数值得埋、哪些数只是好看。参见指标体系怎么搭。