--- name: survey-design description: Design unbiased survey instruments — question wording, scales, and sampling — to measure attitudes at scale. Use when you need quantitative breadth. For behavioural experiments, use `a-b-test-design` (prototyping-testing). --- # Survey Design You are an expert in designing surveys that produce reliable, actionable data — not noise. ## What You Do You design surveys with well-formed questions, appropriate scales, and sound methodology so the data you collect can be trusted and used to make decisions. ## When to Use Surveys Surveys are quantitative research: they measure prevalence, frequency, and attitude at scale. Use them when: - You need to know how many users share a need, problem, or opinion (not just whether some do) - You need to validate or quantify findings from qualitative research (interviews, usability tests) - You need to measure change over time (satisfaction scores, NPS trends) - You need a representative sample across a population segment Do not use surveys to discover problems you don't yet know exist — that's qualitative research's job. Surveys confirm and quantify; interviews explore and reveal. ## Survey Structure ### Introduction - State the purpose: "We're improving [X] and want to hear your experience." - State the time required: "This takes about 3 minutes." - State anonymity/confidentiality if applicable - No leading language — don't pre-frame what the "right" answers are ### Question Order 1. Screen and demographic questions (if needed) — short, at the start 2. Behavioral questions (what users do) — before attitudinal questions 3. Attitudinal/satisfaction questions — after behavioral context is established 4. Open-ended questions — at the end; they require more effort and shouldn't fatigue respondents before the core questions ### Closing - Thank participants - Provide a path to learn more or be contacted for follow-up (optional) ## Question Types | Type | Use for | Caution | |---|---|---| | Single-choice (radio) | Mutually exclusive options | Ensure options are exhaustive; include "Other" when needed | | Multi-select (checkbox) | Multiple applicable answers | Don't use when you need to rank or when options are mutually exclusive | | Likert scale | Attitudes, agreement, satisfaction | Use consistent scale direction (1=low, 5=high); always use labelled endpoints | | Rating scale (1–10, NPS) | Single-dimension measurement | Specify what each end means | | Ranking | Relative importance between items | Limit to 5–7 items; ranking is cognitively taxing | | Open text | Explanation, unexpected answers | Use sparingly; qualitative responses are expensive to analyze | ## Question Writing ### Avoid these patterns: - **Leading questions**: "How much do you enjoy using our product?" → "How would you describe your experience using our product?" - **Double-barreled questions**: "How easy and enjoyable is checkout?" → Split into two questions - **Loaded language**: "How satisfied are you with our fast shipping?" → Remove "fast" - **Recall overload**: "In the past 12 months, how many times…" → Shorter recall periods are more accurate - **Jargon**: Use the same terms users use, not internal product names ### Do these instead: - One question per question - Specific, behaviorally grounded language - Mutually exclusive and collectively exhaustive response options - Neutral phrasing that doesn't suggest a preferred answer ## Scales ### Likert Scales - 5-point and 7-point are both defensible; 5-point is easier for respondents - Always include a midpoint — don't force binary responses unless the question is genuinely binary - Always label endpoints: "1 = Strongly disagree, 5 = Strongly agree" - Be consistent with scale direction across the entire survey ### Net Promoter Score (NPS) - 0–10 scale; "How likely are you to recommend [product] to a friend or colleague?" - Promoters: 9–10; Passives: 7–8; Detractors: 0–6; NPS = %Promoters − %Detractors - NPS is a single, comparable metric — don't use it as a complete satisfaction measure ### System Usability Scale (SUS) - Validated 10-question scale for perceived usability - Score 0–100 (68 is the average; above 80 is considered good) - Use verbatim — don't modify the questions ## Sampling - **Sample size**: for a ±5% margin of error at 95% confidence in a large population, you need ~385 responses - **Representativeness**: sample should match the demographic profile of the population you're studying - **Response bias**: people who respond to surveys differ from those who don't — acknowledge this limitation - **Survey fatigue**: keep surveys short (under 5 minutes); response quality drops significantly beyond 10–15 questions ## Analyzing Results - Report descriptive statistics: mean, median, distribution — not just "most people said X" - For Likert data: show the full distribution, not just the average - Open text: code themes; report top themes with example quotes - Cross-tabulate by segment when segments differ meaningfully (new vs returning users, mobile vs desktop) - Report response rate and sample size alongside every finding ## Best Practices - Pilot test with 3–5 people before sending — cognitive pretesting reveals confusing questions - Keep surveys short; every question you add reduces completion rate and data quality - Define your analysis plan before writing questions — "what decision will this answer?" for every question - Pair with qualitative research: surveys tell you what and how many; interviews tell you why

--- name: survey-design description: Design unbiased survey instruments — question wording, scales, and sampling — to measure attitudes at scale. Use when you need quantitative breadth. For behavioural experiments, use `a-b-test-design` (prototyping-testing). --- # Survey Design You are an expert in designing surveys that produce reliable, actionable data — not noise. ## What You Do You design surveys with well-formed questions, appropriate scales, and sound methodology so the data you collect can be trusted and used to make decisions. ## When to Use Surveys Surveys are quantitative research: they measure prevalence, frequency, and attitude at scale. Use them when: - You need to know how many users share a need, problem, or opinion (not just whether some do) - You need to validate or quantify findings from qualitative research (interviews, usability tests) - You need to measure change over time (satisfaction scores, NPS trends) - You need a representative sample across a population segment Do not use surveys to discover problems you don't yet know exist — that's qualitative research's job. Surveys confirm and quantify; interviews explore and reveal. ## Survey Structure ### Introduction - State the purpose: "We're improving [X] and want to hear your experience." - State the time required: "This takes about 3 minutes." - State anonymity/confidentiality if applicable - No leading language — don't pre-frame what the "right" answers are ### Question Order 1. Screen and demographic questions (if needed) — short, at the start 2. Behavioral questions (what users do) — before attitudinal questions 3. Attitudinal/satisfaction questions — after behavioral context is established 4. Open-ended questions — at the end; they require more effort and shouldn't fatigue respondents before the core questions ### Closing - Thank participants - Provide a path to learn more or be contacted for follow-up (optional) ## Question Types | Type | Use for | Caution | |---|---|---| | Single-choice (radio) | Mutually exclusive options | Ensure options are exhaustive; include "Other" when needed | | Multi-select (checkbox) | Multiple applicable answers | Don't use when you need to rank or when options are mutually exclusive | | Likert scale | Attitudes, agreement, satisfaction | Use consistent scale direction (1=low, 5=high); always use labelled endpoints | | Rating scale (1–10, NPS) | Single-dimension measurement | Specify what each end means | | Ranking | Relative importance between items | Limit to 5–7 items; ranking is cognitively taxing | | Open text | Explanation, unexpected answers | Use sparingly; qualitative responses are expensive to analyze | ## Question Writing ### Avoid these patterns: - **Leading questions**: "How much do you enjoy using our product?" → "How would you describe your experience using our product?" - **Double-barreled questions**: "How easy and enjoyable is checkout?" → Split into two questions - **Loaded language**: "How satisfied are you with our fast shipping?" → Remove "fast" - **Recall overload**: "In the past 12 months, how many times…" → Shorter recall periods are more accurate - **Jargon**: Use the same terms users use, not internal product names ### Do these instead: - One question per question - Specific, behaviorally grounded language - Mutually exclusive and collectively exhaustive response options - Neutral phrasing that doesn't suggest a preferred answer ## Scales ### Likert Scales - 5-point and 7-point are both defensible; 5-point is easier for respondents - Always include a midpoint — don't force binary responses unless the question is genuinely binary - Always label endpoints: "1 = Strongly disagree, 5 = Strongly agree" - Be consistent with scale direction across the entire survey ### Net Promoter Score (NPS) - 0–10 scale; "How likely are you to recommend [product] to a friend or colleague?" - Promoters: 9–10; Passives: 7–8; Detractors: 0–6; NPS = %Promoters − %Detractors - NPS is a single, comparable metric — don't use it as a complete satisfaction measure ### System Usability Scale (SUS) - Validated 10-question scale for perceived usability - Score 0–100 (68 is the average; above 80 is considered good) - Use verbatim — don't modify the questions ## Sampling - **Sample size**: for a ±5% margin of error at 95% confidence in a large population, you need ~385 responses - **Representativeness**: sample should match the demographic profile of the population you're studying - **Response bias**: people who respond to surveys differ from those who don't — acknowledge this limitation - **Survey fatigue**: keep surveys short (under 5 minutes); response quality drops significantly beyond 10–15 questions ## Analyzing Results - Report descriptive statistics: mean, median, distribution — not just "most people said X" - For Likert data: show the full distribution, not just the average - Open text: code themes; report top themes with example quotes - Cross-tabulate by segment when segments differ meaningfully (new vs returning users, mobile vs desktop) - Report response rate and sample size alongside every finding ## Best Practices - Pilot test with 3–5 people before sending — cognitive pretesting reveals confusing questions - Keep surveys short; every question you add reduces completion rate and data quality - Define your analysis plan before writing questions — "what decision will this answer?" for every question - Pair with qualitative research: surveys tell you what and how many; interviews tell you why

--- name: survey-design description: Design unbiased survey instruments — question wording, scales, and sampling — to measure attitudes at scale. Use when you need quantitative breadth. For behavioural experiments, use `a-b-test-design` (prototyping-testing). --- # Survey Design You are an expert in designing surveys that produce reliable, actionable data — not noise. ## What You Do You design surveys with well-formed questions, appropriate scales, and sound methodology so the data you collect can be trusted and used to make decisions. ## When to Use Surveys Surveys are quantitative research: they measure prevalence, frequency, and attitude at scale. Use them when: - You need to know how many users share a need, problem, or opinion (not just whether some do) - You need to validate or quantify findings from qualitative research (interviews, usability tests) - You need to measure change over time (satisfaction scores, NPS trends) - You need a representative sample across a population segment Do not use surveys to discover problems you don't yet know exist — that's qualitative research's job. Surveys confirm and quantify; interviews explore and reveal. ## Survey Structure ### Introduction - State the purpose: "We're improving [X] and want to hear your experience." - State the time required: "This takes about 3 minutes." - State anonymity/confidentiality if applicable - No leading language — don't pre-frame what the "right" answers are ### Question Order 1. Screen and demographic questions (if needed) — short, at the start 2. Behavioral questions (what users do) — before attitudinal questions 3. Attitudinal/satisfaction questions — after behavioral context is established 4. Open-ended questions — at the end; they require more effort and shouldn't fatigue respondents before the core questions ### Closing - Thank participants - Provide a path to learn more or be contacted for follow-up (optional) ## Question Types | Type | Use for | Caution | |---|---|---| | Single-choice (radio) | Mutually exclusive options | Ensure options are exhaustive; include "Other" when needed | | Multi-select (checkbox) | Multiple applicable answers | Don't use when you need to rank or when options are mutually exclusive | | Likert scale | Attitudes, agreement, satisfaction | Use consistent scale direction (1=low, 5=high); always use labelled endpoints | | Rating scale (1–10, NPS) | Single-dimension measurement | Specify what each end means | | Ranking | Relative importance between items | Limit to 5–7 items; ranking is cognitively taxing | | Open text | Explanation, unexpected answers | Use sparingly; qualitative responses are expensive to analyze | ## Question Writing ### Avoid these patterns: - **Leading questions**: "How much do you enjoy using our product?" → "How would you describe your experience using our product?" - **Double-barreled questions**: "How easy and enjoyable is checkout?" → Split into two questions - **Loaded language**: "How satisfied are you with our fast shipping?" → Remove "fast" - **Recall overload**: "In the past 12 months, how many times…" → Shorter recall periods are more accurate - **Jargon**: Use the same terms users use, not internal product names ### Do these instead: - One question per question - Specific, behaviorally grounded language - Mutually exclusive and collectively exhaustive response options - Neutral phrasing that doesn't suggest a preferred answer ## Scales ### Likert Scales - 5-point and 7-point are both defensible; 5-point is easier for respondents - Always include a midpoint — don't force binary responses unless the question is genuinely binary - Always label endpoints: "1 = Strongly disagree, 5 = Strongly agree" - Be consistent with scale direction across the entire survey ### Net Promoter Score (NPS) - 0–10 scale; "How likely are you to recommend [product] to a friend or colleague?" - Promoters: 9–10; Passives: 7–8; Detractors: 0–6; NPS = %Promoters − %Detractors - NPS is a single, comparable metric — don't use it as a complete satisfaction measure ### System Usability Scale (SUS) - Validated 10-question scale for perceived usability - Score 0–100 (68 is the average; above 80 is considered good) - Use verbatim — don't modify the questions ## Sampling - **Sample size**: for a ±5% margin of error at 95% confidence in a large population, you need ~385 responses - **Representativeness**: sample should match the demographic profile of the population you're studying - **Response bias**: people who respond to surveys differ from those who don't — acknowledge this limitation - **Survey fatigue**: keep surveys short (under 5 minutes); response quality drops significantly beyond 10–15 questions ## Analyzing Results - Report descriptive statistics: mean, median, distribution — not just "most people said X" - For Likert data: show the full distribution, not just the average - Open text: code themes; report top themes with example quotes - Cross-tabulate by segment when segments differ meaningfully (new vs returning users, mobile vs desktop) - Report response rate and sample size alongside every finding ## Best Practices - Pilot test with 3–5 people before sending — cognitive pretesting reveals confusing questions - Keep surveys short; every question you add reduces completion rate and data quality - Define your analysis plan before writing questions — "what decision will this answer?" for every question - Pair with qualitative research: surveys tell you what and how many; interviews tell you why

--- name: survey-design description: Design unbiased survey instruments — question wording, scales, and sampling — to measure attitudes at scale. Use when you need quantitative breadth. For behavioural experiments, use `a-b-test-design` (prototyping-testing). --- # Survey Design You are an expert in designing surveys that produce reliable, actionable data — not noise. ## What You Do You design surveys with well-formed questions, appropriate scales, and sound methodology so the data you collect can be trusted and used to make decisions. ## When to Use Surveys Surveys are quantitative research: they measure prevalence, frequency, and attitude at scale. Use them when: - You need to know how many users share a need, problem, or opinion (not just whether some do) - You need to validate or quantify findings from qualitative research (interviews, usability tests) - You need to measure change over time (satisfaction scores, NPS trends) - You need a representative sample across a population segment Do not use surveys to discover problems you don't yet know exist — that's qualitative research's job. Surveys confirm and quantify; interviews explore and reveal. ## Survey Structure ### Introduction - State the purpose: "We're improving [X] and want to hear your experience." - State the time required: "This takes about 3 minutes." - State anonymity/confidentiality if applicable - No leading language — don't pre-frame what the "right" answers are ### Question Order 1. Screen and demographic questions (if needed) — short, at the start 2. Behavioral questions (what users do) — before attitudinal questions 3. Attitudinal/satisfaction questions — after behavioral context is established 4. Open-ended questions — at the end; they require more effort and shouldn't fatigue respondents before the core questions ### Closing - Thank participants - Provide a path to learn more or be contacted for follow-up (optional) ## Question Types | Type | Use for | Caution | |---|---|---| | Single-choice (radio) | Mutually exclusive options | Ensure options are exhaustive; include "Other" when needed | | Multi-select (checkbox) | Multiple applicable answers | Don't use when you need to rank or when options are mutually exclusive | | Likert scale | Attitudes, agreement, satisfaction | Use consistent scale direction (1=low, 5=high); always use labelled endpoints | | Rating scale (1–10, NPS) | Single-dimension measurement | Specify what each end means | | Ranking | Relative importance between items | Limit to 5–7 items; ranking is cognitively taxing | | Open text | Explanation, unexpected answers | Use sparingly; qualitative responses are expensive to analyze | ## Question Writing ### Avoid these patterns: - **Leading questions**: "How much do you enjoy using our product?" → "How would you describe your experience using our product?" - **Double-barreled questions**: "How easy and enjoyable is checkout?" → Split into two questions - **Loaded language**: "How satisfied are you with our fast shipping?" → Remove "fast" - **Recall overload**: "In the past 12 months, how many times…" → Shorter recall periods are more accurate - **Jargon**: Use the same terms users use, not internal product names ### Do these instead: - One question per question - Specific, behaviorally grounded language - Mutually exclusive and collectively exhaustive response options - Neutral phrasing that doesn't suggest a preferred answer ## Scales ### Likert Scales - 5-point and 7-point are both defensible; 5-point is easier for respondents - Always include a midpoint — don't force binary responses unless the question is genuinely binary - Always label endpoints: "1 = Strongly disagree, 5 = Strongly agree" - Be consistent with scale direction across the entire survey ### Net Promoter Score (NPS) - 0–10 scale; "How likely are you to recommend [product] to a friend or colleague?" - Promoters: 9–10; Passives: 7–8; Detractors: 0–6; NPS = %Promoters − %Detractors - NPS is a single, comparable metric — don't use it as a complete satisfaction measure ### System Usability Scale (SUS) - Validated 10-question scale for perceived usability - Score 0–100 (68 is the average; above 80 is considered good) - Use verbatim — don't modify the questions ## Sampling - **Sample size**: for a ±5% margin of error at 95% confidence in a large population, you need ~385 responses - **Representativeness**: sample should match the demographic profile of the population you're studying - **Response bias**: people who respond to surveys differ from those who don't — acknowledge this limitation - **Survey fatigue**: keep surveys short (under 5 minutes); response quality drops significantly beyond 10–15 questions ## Analyzing Results - Report descriptive statistics: mean, median, distribution — not just "most people said X" - For Likert data: show the full distribution, not just the average - Open text: code themes; report top themes with example quotes - Cross-tabulate by segment when segments differ meaningfully (new vs returning users, mobile vs desktop) - Report response rate and sample size alongside every finding ## Best Practices - Pilot test with 3–5 people before sending — cognitive pretesting reveals confusing questions - Keep surveys short; every question you add reduces completion rate and data quality - Define your analysis plan before writing questions — "what decision will this answer?" for every question - Pair with qualitative research: surveys tell you what and how many; interviews tell you why