How Your Privacy Policy Affects Sign-Ups – Surprising Data From 4 Different A/B Tests

How Your Privacy Policy Affects ConversionI recently tested four different privacy policies on a sign-up form on the home page of a betting community. The results were quite surprising as the variations had drastically different impact on sign-ups – from an 18.70% drop in sign-ups to an increase of 19.47%.

In this article I’ll show all four variations, run you through the test data, and give you concrete takeaways.

Privacy policy experiment 1:

Here’s a screen dump of the BettingExpert.com home page. As you can see, the control variant of the sign-up form does not feature a privacy policy, so I decided to test the impact that adding a privacy policy would have on conversions – in this case sign-ups.

Control Variant Sign-Up Privacy Policy

In the first experiment I went for an informal, slightly cute privacy policy:

100% privacy – we will never spam you

I totally assumed that the treatment with the privacy policy would perform significantly better than the control. In fact, I only ran this test in order to document how much of a lift it would generate. So I was pretty taken aback by the test results.

The treatment with the privacy policy actually hurt conversions and reduced the number of sign-ups by a staggering 18.70%!

Privacy Policy Experiment 1

Test data: I ran the test for 9 days, reached a sample size of 16152 visits and 297 conversions, and the control variant outperformed the treatment throughout the entire test period. The statistical confidence level was 96%, and the standard error was 0%.

Privacy Policy Experiment 1 Test Develpoment

The development of experiment one over the 9-day test period.

Takeaways from experiment 1:

However counterintuitive it may seem, adding a privacy policy does not guarantee more sign-ups. In fact, it can seriously hurt conversions.

My hypothesis is that - although the messaging revolves around assuring prospects that they won’t be spammed – the word spam itself give rise to anxiety in the mind of the prospects. Therefore, the word should be avoided in close proximity to the form.

Privacy policy experiment 2:

In the second privacy policy test, I stayed clear of the word spam but stuck with the short and sweet 100% privacy from the first treatment:

100% privacy. We keep all your personal information secret

Test data: I ran this test for 12 days, and after 15675 visitors and 279 conversions, there really was no significant difference between the control variant and the treatment. The treatment started out performing better, but tanked as the sample size grew over the 12 day test period.

Privacy Policy Experiment 2

The treatment started out performing better, but tanked as the sample size grew over the 12-day test period.

Takeaways from experiment 2:

The treatment in experiment 2 performed way better than the treatment from experiment 1, but it still didn’t improve conversion.

It would seem that removing the word spam and focusing on the aspect of keeping information secret had a positive effect. Nevertheless, this is still a vague policy that doesn’t say a whole lot.

Privacy policy experiment 3:

Ok, so experiment 1 and 2 gave me some valuable insights and helped me develop an idea for yet another treatment. I was still pretty surprised by the results of the first two tests.

For the third treatment I decided to go for a more authoritative and solid policy with no “cuteness factor”. Moreover, I wanted this treatment to be much clearer than the two previous ones. With all this in mind, I came up with:

We guarantee 100% privacy. Your information will not be shared

Bingo – this treatment was a home run! It increased conversions significantly and gave us 19.47% more sign ups.

Experiment 3 Privacy Policy

Test data: I ran the test for 12 days and reached a sample size of 20257 visitors and 380 conversions. The statistical confidence level was 96% and the standard error was 0%. From very early on in the test, the treatment outperformed the control variant.

Privacy Policy Experiment 3 Test Develpoment

The development of experiment 3 over the 12-day test period.

Takeaways from experiment 3: 

The most noticeable change in treatment 3 is the guarantee. In experiment 1 and 2, the policy only said, 100% privacy, whereas the policy in experiment 3 said, We guarantee 100% privacy.

Personally I find this wording much more credible. Moreover, the second part, Your information will not be shared, is way more clear and authoritative.

Credibility, clarity, and authority is really what you want in a privacy policy, and I believe that the combination of those three factors is what made this treatment perform so well.

Privacy policy experiment 4:

OK, so the first three experiments taught me a lot and helped me develop a few hypotheses – mainly that you need a guarantee on your privacy policy and that using the word spam can seriously backfire.

Now I was curious to see what would happen if I combined the “best” and “worst” from the previous experiments into one variant – which resulted in treatment 4:

We guarantee 100% privacy. We will never spam you!

Test data: I ran this test for 15 days and after 18959 visitors and 370 conversions, there was no significant difference between the control variant and the treatment policy. 

Privacy Policy Experiment 4

Takeaways from experiment 4:

Treatment 4 was a combination of the best performing variant, We guarantee 100% privacy, and the worst performing variant, We will never spam you.

As the test data revealed that there was no significant difference between the control and the treatment, it would seem that the “good” and the “bad” part parts cancelled each other out and therefore had no real impact on sign-ups.

Main take-aways from all 4 experiments

The privacy policy you use on your signup form can have major impact on your sign-up rate. However, just sticking one on there doesn’t guarantee more conversions – in fact, if your not careful about choosing the right wording, you could seriously hurt your conversion rate.

Experiments 1 & 3 Privacy Policy

But If you’re willing to put some time into researching what works on your specific form, a privacy policy can lead to a serious lift in conversions – talk a bout a major low-hanging fruit!

My research points to the fact that a credible, clear policy with a guarantee effectively assures prospects that it’s safe to fill out the form. Moreover, my data suggests that you should be careful with using the word spam – even if the intention is to guarantee against it – as it can backfire and create a higher level of anxiety.

The policy that did best in this series of tests was:

We guarantee 100% privacy. Your information will not be shared.

I’m currently testing it on my newsletter form here on ContentVerve.com, and at the moment the treatment with the policy is performing 6% better than the control – however, the data isn’t conclusive yet.

By the way – This wasn’t the first thing I tested on BettingExpert.com

I’ve done a bunch of other tests on BettingExpert.com. Among other things, I tested a few simple copy tweaks that resulted in an increase in sign-ups of of 31.54%.

Dramatic lift in conversion by tweaking form copy

Check out the full case study here >>

Comments

  1. Peter says:

    Dear Michael,

    Thanks for the great insights. I also love the fact that you allready implement the findings in your own newsletter update box above :)

    Best
    Peter

  2. Seo Martin says:

    looks like te user understand only SPAM YOU on the “we will never spam you” :-\

  3. yassin says:

    Awesome post. Thanks michael. I’ll put together a test and see what kind of results i’ll get. very interesting.

    Would love to see a post on how you develop your hypothesis in the first place and how you go about doing your quantitative and qualitative research (Maybe i’m asking for too much :) ).

    Always enjoying your posts.

    Make your days great
    Yassin

    • Michael Aagaard says:

      Thanks Yassin!

      And great tips for blog post topics. I think I’m going to do one on test hypotheses soon.

      - Michael

  4. Not surprising findings, but good information for others to reference.

    I’m betting you could get an even better conversion percentage by changing the “sign up” button text to something different. ;)

  5. Mike says:

    As usual, great stuff Michael! I wonder if a treatment using trust symbols would change sign ups at all (SSL, TRUSTe, etc.). Possibly a future ContentVerve experiment?
    –Mike

    • Michael Aagaard says:

      Thanks Mike! Testing trust symbols would definitely be a great case study for ContentVerve.com – great idea!

      - Michael

  6. Christian Møller says:

    Who would have guessed that it could hurt your conversionrate that much to add 1 line about privacy police.. oh man you have to be carefull about stuff with your copy :D Once again thanks for sharing Michael!

  7. Janean says:

    Good stuff. I’d be interested to see results of a similar test in the B2B world.

    • Michael Aagaard says:

      Thank you Janean – Yes a B2B case would be super interesting. I’d jump at the opportunity to experiment with B2B signup forms.

      - Michael

  8. mike says:

    Great share on your study! It sounds like the word, “guarantee” from treatment 3 was the difference maker. As with most products, people take comfort in knowing there is some sort of “guarantee” with their purchase and it seems that by utilizing that same word worked well with boosting conversions for you in this experiment.

  9. Christopher says:

    Great experiment. I never would have realized the impacts would have been so great, all from a couple of varying words. These tests are an excellent illustration of how slight little changes you may not think are important, can actually make all the difference in how well your sites and opt-in forms perform. I will be analyzing every detail on my websites more closely now. Thank you MIchael!

    • Michael Aagaard says:

      Hi Christopher – thank you very much for the kind words!

      Yes, these tests also taught me a lot and opened my eyes to a whole new area of low-hanging fruits in optimization.

      - Michael

  10. Greg Moore says:

    It’s excellent that you include the data about your tests – confidence level, etc.

    A University of California professor did studies showing that once you put an idea in someone’s head it’s difficult to get rid of it. My hypothesis is that saying you won’t “spam” people puts the idea of spam in their head, making them hesitate to fill in the form. Even though it says you won’t spam them, that’s not enough to remove the worry. So I guess your tests prove that. Apparently it’s best to simply avoid the mention of spam.

    If you can do another test, I would like to see the results for this phrase:

    “We value your privacy.”

  11. Bobby Hewitt says:

    Michael,
    Great meeting you in Boston recently.
    I agree with your analysis about increasing anxiety and am wondering if there is anxiety in your winner with the copy (Your information will not be shared.) The (We guarantee 100% privacy.) is VERY strong and therefore may be correcting for the anxiety in the second sentence but I can’t stop wonder how the winner would perform against just the first part?
    So…

    We guarantee 100% privacy. Your information will not be shared.
    Vs.
    We guarantee 100% privacy.
    I’m planning on running this one myself.

    • Michael Aagaard says:

      Hi Bobby – It was great meeting you too man!

      Really good point about the sharing information part! I hadn’t thought about that myself. I’ll have to test that too. Please let me about your results, I’d love to compare notes.

      - Michael

  12. Tam says:

    GREAT ARTICLE, GOOD TIPS

  13. I think it is a very interest A/B test, and i will implement this on my own site.

    http://grancanariaguide.dk

  14. Michael, great experiment, makes sense those results, that simple privacy policy passes confidence and users want it, nobody likes spams.

  15. Simon says:

    Hey Michael, I’ve just come across this article via a WhichTestWon posting. This is super interesting and I’ve shared the info via our own blog as well :)

    (https://www.iubenda.com/blog/2013/11/22/accept-privacy-policy-conversion-improvements/)

Speak Your Mind

*