Bring survey prompt back to 10 days and every 90 days. #5631
Conversation
Codecov Report
@@ Coverage Diff @@
## master #5631 +/- ##
==========================================
- Coverage 70.68% 70.67% -0.02%
==========================================
Files 408 408
Lines 15581 15581
==========================================
- Hits 11014 11012 -2
- Misses 3757 3758 +1
- Partials 810 811 +1
Continue to review full report at Codecov.
|
| @@ -321,7 +321,7 @@ func recentlyPromptedOrTaken(cfg *ContextConfig) bool { | |||
| if cfg == nil || cfg.Survey == nil { | |||
| return false | |||
| } | |||
| return lessThan(cfg.Survey.LastTaken, 365*24*time.Hour) || lessThan(cfg.Survey.LastPrompted, 60*24*time.Hour) | |||
| return lessThan(cfg.Survey.LastTaken, 90*24*time.Hour) || lessThan(cfg.Survey.LastPrompted, 10*24*time.Hour) | |||
There was a problem hiding this comment.
Not that I think it's a problem but if most of our users download the latest release around the same day then these reminders will be in sync 😅 Should we have a ( +- random number of days) to introduce slight disorder?
There was a problem hiding this comment.
I doubt our survey traffic will be noticed by Google Forms :-)
There was a problem hiding this comment.
I wasn't concerned about traffic, more about boosting some bias in the responses. Say the first release with this has some other bug, and a lot of feedback comes in on the same day around that bug. You can see this cycle repeating where a lot of feedback is clustered on specific dates (and as such specific skaffold versions). Though even that probably isn't significant at our scale.
There was a problem hiding this comment.
That's a very good point. Though we've never issued a dot release for bug fixes within a day or so.
- 1.13.0 -> 1.13.1: Jul 30 -> Aug 4
- 1.13.1 -> 1.13.2: Aug 4 -> Aug 20
- 1.17.0 -> 1.17.1: Nov 23 -> Dec 1
- 1.17.1 -> 1.17.1: Dec 1 -> Dec 7
There was a problem hiding this comment.
In that case consider the example that we have 10 diligent users who respond to the survey as soon as they see the prompt. They'll be reprompted on the exact date 3 months later and that cycle continues. Now we only have feedback for those specific skaffold versions and very few in between. 🤷🏾♂️
None of that might be statistically significant and I'm ok with the change though.
There was a problem hiding this comment.
basically everyone will see this prompt as soon as they download the new binary (assuming most users don't submit feedback). Should there be a first prompt offset of random days between (0 - 15) so that it's more evened out?
There was a problem hiding this comment.
I did consider/mention this in the Survey DD where we should show the prompt X days after download.
There is no clear way to distinguishing a new user Vs an old user upgrading to a version. (Note, feedbacks are not associated with version)
I feel a new user would probably ignore the survey prompt first tine and the next one should show after 10 days at which he point they may choose to respond.
If its a not small change in code, i would not mind adding it in a separate PR. However i feel we might over engineer when most times a new user will ignore.
Revert #4626 and bring back survey prompts to normal cadency