Redesigning a Survey Creation System to Reduce Errors by 45%
A 14-week end-to-end redesign of the survey builder — eliminating systematic errors through real-time validation, visual logic, and a progressive step structure.
45%
Fewer Errors
Enabled faster campaign launches and improved data reliability.
35%
Faster creation
22%
Higher data capture
18%
Higher completion
role
Lead Product Designer & PM
platform
B2B SaaS · Web App
duration
14 Weeks · Q3 2025
team
1 Designer · 3 Eng · 1 Senior PM
company
Karnival
Context & Problem
A capable tool generating its own errors
I joined as the first design hire after the engineering team had rebuilt the core survey builder. Three months of support ticket analysis revealed one thing clearly: errors were not user mistakes. They were built into how the workflow was structured.
"Store managers misconfigured branching logic on nearly 1 in 3 surveys with no way to detect the error until after the survey was already live."
"A single Save and Publish button conflated two different actions, causing teams to accidentally launch surveys they intended to save as drafts."
"No quality signal existed during the build process. Errors accumulated silently while users focused on content, invisible until the publish attempt."
"Rework became the norm. Teams averaged more than 2 publish attempts per survey, and treated the error modal as a standard step in the workflow."
"I just hit publish and hoped for the best. Half the time I got an error I had to Google to understand."
— Research Operations Manager · User interview participant
~1 in 3
Surveys had a logic error at publish
>2×
Average publish attempts per survey
~65%
Support tickets traced to the builder
Goals
Business goals & user outcomes
Business Goals
Reduce error-driven support volume
Lower builder-related support tickets to free up CS bandwidth for growth-stage conversations
Eliminate the re-publish cycle
Bring average publish attempts from 2 or more down to as close to 1.0 as possible
Improve data quality downstream
Errors in survey logic produced corrupted response data. Cleaner surveys meant cleaner analytics.
Reduce design and engineering debt
Replace the text-based rule editor with a maintainable, extensible visual system
User Goals
Understand what is wrong, immediately
Users should not have to click Publish to discover an error they could have fixed 20 minutes earlier
Configure logic without syntax knowledge
Non-technical team members in HR, ops, and research should own survey creation independently
Feel confident before publishing
A clear publish-readiness signal to replace the current guesswork at the end of the flow
Save work without going live
Separate saving a draft from publishing. These are two fundamentally different intentions.
Key Insights
What users taught us
12 moderated usability sessions · 5 contextual interviews · 3 months of support ticket analysis
Users didn't think in branching logic — they thought in steps.
Every user described their mental model sequentially: 'If someone says X, send them to the next relevant question.' The boolean syntax editor mapped to a completely different model, one only engineers found natural.
Cognitive model mismatch was the root cause, not user error.
An untrustworthy safety net is worse than none.
Preview mode existed, but around 60% of users skipped it entirely because it did not simulate branching logic. Users knew it was incomplete, so they stopped trusting it. A broken safeguard creates false confidence without providing real protection.
An incomplete preview created false confidence, not protection.
Error feedback must happen where the error was made.
A modal listing configuration errors at Publish was useless because users could not trace them back to decisions made 20 minutes earlier. Temporal distance kills the feedback loop.
Feedback co-located with the cause changes behavior. Deferred feedback does not.
Normalized workarounds mask the real severity of a problem.
Power users had developed a ritual: publish, read errors, fix, republish. They had adapted to failure so thoroughly it looked like competence. This suppressed the true error rate in analytics.
Ask: are users succeeding, or have they learned to manage failure?
Product Decisions
Key decisions that drove impact
Inline Validation Over End-of-Flow Modals
All errors were silently collected and surfaced in a single modal only at Publish. Users could not trace those errors back to the decisions that created them 20 minutes earlier.
Real-time, field-level validation that surfaces errors co-located with the action — not deferred to the end. Warnings are soft and non-blocking; errors are hard and publish-gated.
Added engineering complexity, but changed user behavior at the source.
Visual Logic Builder Over Syntax Editor
The rule editor required conditional expressions using field IDs and boolean operators. Most non-technical users skipped logic entirely, leaving surveys with broken branching paths.
An If → Then card builder connecting questions in natural language — no syntax, no field IDs. Retained an Advanced Mode escape hatch for power users.
Increased steps for power users, but removed the primary source of configuration errors.
Persistent Survey Health Panel
There was no ambient quality signal during building. Users focused on content had zero passive awareness that errors were accumulating.
A collapsible sidebar showing a live error and warning count with a color-coded indicator (green, yellow, red). Inspired by IDE linting: passive awareness without interrupting the active editing flow.
Reduced canvas space, but gave users ambient awareness that prevented errors before they compounded.
Splitting Save Draft from Publish
A single Save and Publish CTA conflated two fundamentally different intentions: saving progress (frequent, low-stakes) and launching a survey live (infrequent, high-stakes).
Two visually distinct actions: a ghost Save Draft button and a primary Publish Survey button gated by a health check. When errors exist, Publish shows a confirmation step explaining exactly what needs to be fixed.
One extra click, but eliminated the class of accidental publishes entirely.
Before vs After
Transformation across every touchpoint
Before
Accidental publishes from a single CTA
Save and Publish were one button. Teams routinely launched surveys they meant to save as drafts.
Errors only visible at Publish
Logic errors accumulated silently for 20+ minutes, then exploded in a modal nobody could parse.
Boolean syntax inaccessible to most users
Non-technical team members had to write field IDs and conditional expressions — or leave logic blank.
No health signal during building
The builder gave zero ambient feedback. Users had no idea how many errors they had accumulated.
After
Explicit Save Draft and Publish separation
Two visually distinct actions, each with different weight. Intent is clear and accidental launches are blocked.
Inline validation at the point of error
Errors surface on the field where they were created, co-located in time and space with the action that caused them.
Visual If → Then logic builder
Natural language card builder. No syntax, no IDs. Any team member can configure logic without training.
Persistent health panel with live count
Color-coded sidebar (green / yellow / red) gives passive awareness of error state throughout the build process.
Solution
A builder with ambient validation and visual logic
The core design principle was simple. Users should encounter errors during building, never only after publishing.
Wireframes only. Real UI not shown per NDA.
Ambient awareness without interruption
Users had no passive sense of error accumulation while building. This gives a live health signal without breaking the flow of content creation.
Co-located with the cause
A modal at Publish can list errors, but cannot teach users to avoid them. Errors surfaced at the field level change behavior.
Language-level configuration
Users think in steps, not conditional syntax. The If → Then card builder maps directly to how users naturally describe branching intent.
Intent-separated actions
One button for two different intentions caused accidental launches. Separating Save Draft from Publish restores user intent clarity.
Measured Impact
Results across every metric we tracked
"We went from dreading survey launches to actually trusting the tool. The health panel alone changed how our entire team approaches building. Nobody goes to Publish without a green status now."
— Head of Research Operations · Enterprise Customer (identity anonymized per NDA)
Measured over 8 weeks post-launch vs. an 8-week pre-launch baseline. Validated through analytics, support ticket volume, and post-launch user interviews.
45%
Fewer errors on publish
35%
Faster survey creation
22%
Higher data capture rate
18%
Higher completion rate
What this unlocked
Faster campaign launches: survey creation no longer blocked operational cycles waiting on error resolution
Cleaner data into analytics: fewer broken surveys meant downstream reporting improved without pipeline changes
Reduced CS burden: builder support tickets dropped by ~46%, freeing capacity for growth-stage customer work
| Metric | Before | After |
|---|---|---|
| Avg. publish attempts per survey | >2× | ~1.2× |
| Builder-related support tickets | ~65% | ~19% |
| Research team NPS score | Low 30s | Mid 50s |
| Preview skip rate | ~60% | ~12% |
"We went from dreading survey launches to actually trusting the tool. The health panel alone changed how our entire team approaches building. Nobody goes to Publish without a green status now."
— Head of Research Operations · Enterprise Customer (identity anonymized per NDA)
Iteration
What changed after launch
Power users hit a wall. The If → Then builder could not express nested AND/OR logic.
Added an Advanced Mode toggle exposing the syntax editor as an escape hatch, without affecting the default visual flow.
Power user satisfaction recovered. The non-technical flow was unchanged.
Users dismissed warnings too quickly. Several could not recall seeing them in post-launch interviews.
Increased warning visual weight: amber card border, sidebar icon, and a persistent health panel counter that stays visible after scrolling.
Warning acknowledgment rate increased significantly in session recordings.
Teams started saving drafts earlier and reviewing collaboratively before publishing — behavior we had not designed for.
Added a Share Draft link: one-click preview URL for async team review. Scoped to 2 engineering days.
Draft sharing became one of the highest-used features the following sprint.
Learnings
Reducing errors is more valuable than reducing steps in B2B systems.
- · In high-stakes workflows, one prevented error is worth more than saving 30 seconds.
- · The health panel added UI surface area but drove the largest behavior change in the project.
- · Efficiency is not always fewer clicks. Sometimes it is fewer mistakes.
Users follow mental models, not system logic.
- · The logic editor failed because it was built for boolean syntax, not sequential human reasoning.
- · Any interface requiring users to translate their intent into a different model will fail non-experts.
- · Always start from how users describe the task, not how the system processes it.
Clear feedback reduces reliance on training.
- · When errors are surfaced in context, users self-correct without needing documentation.
- · The most-impactful design intervention was not a feature — it was a feedback signal placed correctly.
- · Well-designed feedback loops replace onboarding. Poor ones require it.
Want to go deeper?
I can walk through research artifacts, the Figma file, or the PRD in a live conversation.