The post has good questions for people to ask, but not specific advice about how to analyze results and decide when you have a conclusion. So I added a link to my tutorial in the comments section. ;-)
You guys suggest using Mixpanel to measure user engagement via split tests, but Mixpanel doesn't have a good way to split test!
I've resorted to hacking something together using properties. I use the property name to represent a segment/test name and the values to represent the splits. This leads to having a ton of properties, which the UI isn't designed for.
An A/B testing tool is on the roadmap, but we just haven't had time yet.
The best way to do tests right now is by utilizing funnel properties. You can test the whole length of your signup funnel and just add a 'button-color' property.
This lets you see the difference in conversion rates between your two treatments pretty easily.
I would also like to do this for one off events. The important thing for us is the ability to view the event metric across property values (intersecting segments):
If the community is engaging, people will be more likely to comment (like HN), but if you are a site, you really don't have two separate communities to swap in/out in the user's mind. Additionally, even if you did optimize your site design/functionality to increase comments, is that really desirable without an engaging community in the first place? Would everything end up being nothing that adds to the conversation? (see Youtube)
