Oct 12


My recent WSJ Accelerators essay struck a nerve and after seeing a few comments posted, I want to clarify a few things — especially about Meebo’s culture and any perceived ill-will. I think some nuance may have been lost in the editing process and I want to make it clear that I love my team, Meebo, and what we built together. This is a personal story about a trying time in my professional life and I wanted to share a boots-on-the-ground perspective. So let me set the record straight:

* Meebo’s culture was as good as it gets and was undoubtedly female-friendly — I’ll never work with another group with so much talent, kindness, and commitment to being inclusive and for making the world a better place. You learn a lot about people after co-founding a company and I can’t say enough good things about my two co-founders, Seth and Sandy. Seth has the keenest strategic nose that you’ll ever encounter and Sandy will out-execute and out-charm anyone in the Valley.

Our first ten employees represented eight universities, three countries, and a 70/30 male/female ratio. Among our senior exec team, we had three women reporting to our CEO and co-founder. Our team’s multiple perspectives led to a stronger and more authentic product. This was an isolated incident that occurred outside of Meebo’s usual business. If anything, I think this shows that building an inclusive environment is hard and you always need to be willing to rethink your own personal views and assumptions. If these issues can arise during my time at Meebo, then I am no longer as quick to judge others either.

* There’s more work to do — Even within an amazing culture, I realize that these issues can just arise organically and subconsciously — in this case, during the hiring process. The most vivid part of this story for me is my coach’s feedback. We talked through a lot of possibilities about why someone would not join last-minute including considering gender, feeling out negotiating strategies, and of course, my own behavior. It’s entirely possible this was just a hard negotiation tactic or that any one of my many, many shortcomings played into this, but I’ll always be struck by the matter-of-fact “this is how the real world works” conversation. It’s the first time I had even the slightest idea that gender was top-of-mind for anyone and that I needed to be more aware of this part of my identity. Whether or not this ended up being the true root cause, I’ve learned to tread more carefully and not assume that people who come from other corporate cultures will inherently share my values and working style.

May 30

what i learned as an oompa loompa

Engineer (left) going Oompa Loompa (right)

Last fall, I heard the spousal call of duty, “Elaine, help me get these doors open.”

My husband, Todd, and his co-founder, Cam, had spent the last few years building chocolate machines and scouting the world for great cocoa beans in an effort to open a micro-batch chocolate factory in San Francisco’s Mission District. The construction crew finished converting the brick automotive garage to a food-safe bean-to-bar workshop and it was time to figure out the nitty gritty details like moving, merchandising, and designing the café & retail space. Todd needed all the help he could get and I was happy to pitch in during those critical months. Now that I’ve come up for air and returned to tech projects, I’ve had time to reflect on what I learned from brick-and-mortar operations:

Even the welds break

  1. For loops are a veritable miracle — At the chocolate factory, something breaks every single flippin’ day. Each morning I gave my evil eye to the roasters, melangers, temperers, wrapping machine, dishwasher, or anything with a screw, fuse, gear, glue, belt, or oil level and asked, “Okay, which one of you little buggers is going today?”

    In comparison, code brings tears to my eyes. If that for loop worked yesterday, then barring catastrophic hardware failures or someone checking in code they shouldn’t, it’ll likely work today. That type of, “if you don’t touch it, it’ll keep working” certainty seems divine. I’ve always loved the Web but I have renewed appreciation for redundancy, unit testing, and monitoring now.

    Lots of chocolate; no email.

  2. Want to release faster? Don’t cut features; cut email. Dandelion’s chocolate makers don’t check email throughout the day (the execs aren’t so lucky). If you can depend upon eight hours of focused time each day, then goals magically happen. With just a back of the napkin estimation, if the team says they’ll make 1000 bars, stop worrying — they’ve got it.

    There are still complications and distractions (e.g. tours, businesses with supply emergencies, daily check-in meetings, and see #1) but there are no late nights catch-up email marathons, no pleas for Agile-enabled predictability, no emergency requests to add two more weeks to the schedule, and no unsustainable sprints to a goal followed by “whoa, let’s never do that again” post-mortem analyses.

    Conservatively, I think that at least 30% of Meebo’s productivity was lost to folks staying current with voluminous Inboxes. Startups want to be hyper-communicative and transparent but those cc’s and long winding email threads add up. However, the first startup that balances getting stuff done despite an ever-expanding Inbox will have a formidable advantage.

    From left to right: future Pulitzer prize winner, Fulbright scholar, and Stanford grad

  3. Tech people, get over yourselves — Dandelion Chocolate has two Fulbright scholars, a Harvard law school grad, Ivy Leaguers, and even outside of formal academic pedigrees, is one of the most talented teams I’ve seen. Tech recruiters spend a lot of time bending over backwards looking for niche skillsets and as a result, we tend to think of ourselves as the center of the talent universe. However, if the folks at Dandelion Chocolate learned to code or design, they’d whoop most of our startup tooshes. Other industries are teeming with extraordinary, passionate people and it doesn’t take an army of recruiters and hundreds of LinkedIn emails to find them. The next time I build a team, I’ll look beyond industry borders.

    Brandon analyzing bean sorting efficiency

  4. Engineering principles provide arbitrage opportunities — When I tell tech people I’ve been pitching in at a chocolate factory, I spot the concerned eyebrows, “Elaine, what about intellectual stimulation?” or a VC will wander to another networking table presumably thinking, “Oy, small potatoes.” Fermentation processes, tempering crystallization structures, and modeling roasting & melanging profiles provide lots of mental fodder. Chocolate making isn’t a traditional startup but engineering principles like small achievable goals, metrics, and A/B testing are very applicable, opening new opportunities in existing markets. The tech world is a fantastic training ground for non-tech industries where there aren’t support networks of meetups, workspaces, and mentors.

    The #1 troublemaker at Dandelion Chocolate

  5. HCI is a digital thing — A little backstory… the women’s bathroom door at Dandelion Chocolate constantly breaks (see #1). From the mezzanine, I can hear the frustrated click-click-click when guests jigger the finicky door lock. At the beginning of the day, I paused when I heard the metal clicking to make sure the bathroom seeker didn’t need help. But by noon, the high-pitched metal-on-metal screeching made my hair stand on end, my stomach churn, and I was ready to do anything… anything… even standing vigilant outside the bathroom to personally escort patrons in and out of the loo… to prevent that irksome sound again.

    When I initially told my parents that I was interested in HCI (Human-Computer Interaction), they didn’t get it. And now I understand why. Technology is so virtual that it takes logging, data mining, insightful researchers, and persuasive experts to convince a team there’s a problem on the Internet worth fixing.

    However, when you’re working with tangibles, you are walking and breathing your own User Experience experiment. You see the half-drunk cups when you take out the trash. You can’t help but overhear under breath comments on the street. And if something is broken, you don’t have to do a cost-benefit analysis to figure out whether fixing the bathroom door is a priority. Someone’s going to fix the bathroom door or they’ll go crazy.

    Chocolate for sale

  6. Simple business models solve so many problems — With Meebo, our business model went something like this: we built a product that people like. We monetized a fraction of those eyeballs with brand advertising. We tracked the percentage of users who clicked an ad and logged their engagement times. However, since correlating brand advertising with bottom line revenue is nearly impossible online, we also monitored ad partnership renewals. If all of those metrics were healthy, we were happy. (~60 words)

    At Dandelion Chocolate, the business model is: we make and sell chocolate. (5 words)

    It’s so much easier to build a sustainable organization around a simple revenue model. There are no tensions between ad partners, distribution sites, engineering, and sales teams. There are fewer points of failure. Instead, everyone is aligned towards a simple goal: make something people want.

    The Dandelion Chocolate team goes to Hawaii after hitting a big goal

  7. Equity isn’t a great long-term motivator — In tech, salary is only one component of your compensation package. If your company does well, your stock options can be worth far more than all of your combined paychecks. Equity is “skin in the game.”

    However, the compensation at Dandelion Chocolate is traditional — wages and salaries. And when things get tough (e.g. the temperer breaks during the December rush — see #1), employees’ visions of becoming overnight millionaires aren’t shattered. Instead, it’s an even-keeled, “Okay, what do we need to do to get through this?” What motivates the team is working with interesting people, more opportunities to travel and grow, and building something pride-worthy.

    As a startup leader, you fear an employee exodus at the first sign of trouble. For engineers with lots of options, trouble means it’s time to diversify your equity portfolio and seek greener pastures elsewhere. In some cases, equity can have the opposite effect and even encourage short-term thinking.

    Valencia St crowded with café experts
    (Photo courtesy of Flickr:tofuart via Creative Commons)

  8. The real world has tougher critics — Startups can shield themselves behind a slow beta roll-out. Your mom and dad might follow your company blog but they probably didn’t critique last Friday’s release.

    In contrast, almost everyone in the world is a café expert. On his day off Todd gets calls and texts, “Todd, the bathroom door seems to slide funny…” or “I drove all the way from Sausalito and you just ran out of marshmallows!”

    The feedback is always, always, always appreciated. But your skin thickens a bit. The chocolate factory is a public venue of our closest friends, friends of friends, neighbors, and supporters — the people you want to make most proud. Every one of those visitor has seen hundreds of cafés and everyone has opinions about how they’d make it better. Unlike tech, you’re very exposed. And if we have an off day, we can’t follow-up with an email newsletter or pop-up notification, “We listened to your feedback! Here’s version 1.2 just for you!”

    Customers in the face blind retail mode

  9. Retail face blindness (i.e. prosopagnosia) — Hopefully there’s a psychology student searching for their next phD thesis among this blog’s readers.

    Either I’m very bland looking (totally plausible) or there’s an undocumented psychologic phenomenon for how people mentally categorize service professionals…

    As a door greeter, cashier, or farmer’s market volunteer, I love talking with visitors. A quick exchange can expand into an intimate 10-minute conversation about our families, our personal food preferences, frequently an exchange of names, and sometimes even promises to follow-up via email.

    Afterwards, I’ll say goodbye and walk next door for a sandwich where I coincidentally spot them again. They look at me when I join them in line, look back, and order a croissant. Nothing. Not even a spark of recognition. I’m a total stranger! This has happened frequently enough that I’m now fascinated by the brain’s ability to immediately delete people information — especially people we don’t think we’ll see again. If you end up researching this further or know anything about this, let me know!

    The chocolate chip cookie — a reliable favorite for seventy-five years and likely to be popular for decades to come

  10. Tech is still harder — It’s easier than ever to start a tech company. The food industry is envious of tech’s lack of permits and regulations, workspaces, meetups, access to capital, and the built-in network of angels & mentors. If you have a good idea, the entrepreneurial community will bend over backwards to help you.

    However, it’s much harder to keep that tech company going. The Internet reinvents itself every two years making longterm planning difficult. It’s safe to assume that people will probably eat chocolate in ten, twenty years. It’s hard to guarantee that any burgeoning startup will be relevant in six months. Building a longterm Internet business within a mercurial market is extraordinarily stressful. In addition to the normal trials of starting a business, you have to be hyper-aware of trends and prepared to pivot tomorrow. It’s likely that the team you hired 18 months ago signed up for a different vision that what you’re working on now and that requires further management. Years 0-2 are easier but years 2-8 are harder.

I’m winding down my Oompa Loompa days with more appreciation for brick-and-mortar businesses. If the toilet paper’s out at a restaurant, I’ll try to change it myself. I tip more and curse the evil people who steal tip jars and iPhones (grrrr)! When the Giants make it to the World Series, I tune in to the police radio chatter and listen for riots. I can add “skilled in the Tiffany bow tying method” to my resume.

Above all, I am grateful to have been a part of the Dandelion Chocolate story and to have learned so much from the team. Thank you again!

And of course, feel free to drop by Dandelion Chocolate at 740 Valencia St (at 18th) in San Francisco.

Dandelion Chocolate Door

Apr 27

100 Mistakes

As a startup founder, I wore a lot of hats and I made a ton of mistakes. I was a typical first-time, 20-something entrepreneur with tremendous pressure to scale with the team and business.

At night, I found myself awake agonizing over the mistakes I could see myself making — not letting go of ideas that I liked but that weren’t good for the business, not presenting my ideas effectively in group meetings, or not saying no to projects when I was already overwhelmed. However, I found that if I wrote down my mistakes in a bedside journal, I could return to sleep and revisit my mistakes in the daytime.

The journal grew and grew. And when I started hiring a team, I saw my team members make the exact same missteps. At first, I was relieved. I no longer felt like I was the worst contributor, manager, director, or VP! But I also wanted to compile my mistakes and share my perspective with them.

At first, I tried giving new managers and directors my bulleted list. However, that was horribly ineffective. No one wants to be handed a list from their manager of all the ways they’ll inevitably fail!

So instead, I started focusing on telling stories and setting the scenes for these mistakes. I sketched the scenes of all of the mistakes and started weaving them into a story that showed the professional journey that everyone makes from their first day on the job as a fresh grad to leading the company as a C-level executive.

It’s an illustrated story that I’ve been narrating and sharing with other startups and organizations. I presented the story at South by Southwest a few months ago. Since then, it’s been mentioned in the Wall Street Journal, Business Insider, and this morning, I presented a few of the mistakes on the CBS Morning Show.

Originally, this was a fun side project that I enjoyed sharing with other startups and the feedback that I’ve heard is that it should be a book. I’ve just started considering that in earnest. If you want to know where you can get a copy of 100 Mistakes, please bear with me — you’ll need to wait a little bit longer!

And at the end of the journey, I’m just grateful to have been a part of a team that allowed each other to grow and to learn from our mistakes. With a little bit more perspective, I am overwhelmed by how much everyone genuinely wants to do well by others and to create something meaningful together. However, old habits, misplaced exuberance, and role ambiguity sometimes get in the way.

Looking forward to sharing more soon!

Dec 09

Five UX Research Pitfalls

I wrote this for UX Mag a while ago and it remains one of my all-time favorite writing projects. However, I never posted it in its entirety on my blog so here it is now. Enjoy!

More and more organizations view UX as a key contributor to successful products, connecting teams with end-users and guiding product innovation within the organization. Though it’s fantastic to see this transition happen, there are growing pains associated with becoming a user-driven organization. These are the pitfalls that I see organizations grappling with most often.

Pitfall 1: It’s easier to evaluate a completed, pixel-perfect product so new products don’t get vetted or tested until they’re nearly out the door.

Months into a development cycle and just days before the release date, you realize that the UI has serious flaws or missing logic. If you’re lucky, there is enough flexibility in the schedule to allow grumbling engineers to re-architect the product. More likely, though, the PM will push to meet the original deadline with the intent to fix the UI issues later. However, “later” rarely happens. Regardless, everyone wonders: how could these issues have been caught earlier?

The UI is typically built after the essential architectural elements are in place and it can be hard to test unreleased products with users until the very last moment. However, you can gather feedback early in the process:

  • Don’t describe the product and ask users if they would use it. In this case, you are more likely testing your sales pitch rather than the idea itself. If you ask users if they want a new feature, 90% of the time they’ll say yes.

  • Test with the users you want, not the users you already have. If you want to grow your audience with a new product, you should recruit users outside your current community.
  • Validate that the problem you are solving actually exists. Early in the design cycle, find your future users and research whether your product will solve their real-world problems. Look for places where users are overcoming a problem via work-around solutions (e.g., emailing links to themselves to keep an archive of favorite sites) or other ineffective practices (e.g., storing credentials in a text file because they can’t remember their online usernames and passwords).
  • Verify your mental models. Make sure that the way you think about the product is the same as your user. For instance, if you’ve been pitching your product idea to your coworkers as “conversational email” but your actual users are teenagers who primarily use text messaging, then your email metaphor probably won’t translate to your younger users. Even if you don’t intend to say “conversational email” in your product, you will unconsciously make subtle design choices that will limit your product’s success until you find a mental model that fits that of your users, not of your coworkers.
  • Prototype early. Create a Flash or patched-together prototype internally as soon as possible. Even if your prototype doesn’t resemble a finished product, you’ll uncover and develop confidence in the major issues to wrestle down in the design process. You’ll also have an easier time seeing the areas of the product that need animations or on-the-fly suggestions which often go unscoped when the product is only explored in wireframes and design specs but can require significant engineering time.
  • Plan through v2. If you intend to launch a product with minimal vetting or testing, make sure you’ve written down and talked about what you intend for the subsequent version. One of the downsides of the “release early, release often” philosophy is that it’s easy to get distracted or discouraged if your beta product doesn’t immediately succeed. Or upon launch you might find your users pulling you in a direction you hadn’t intended because the product wasn’t fully fleshed out or dealing with weeks of bug-fixing and losing sight of the big picture. Once the first version is out the door, keep your team focused on the big picture and dedicated to that second version.

Pitfall 2: Users click on things that are different, not always things they like. Curious trial users skew the usage statistics for a new feature.

Upon adding a “Join now!” button to your site, you cheer when you see an unprecedented 35% click-through rate. Weeks later, registration rates are abysmal and you have to reset expectations with crestfallen teams. So you experiment with the appearance of your “Join now!” button by changing its color from orange to green, and your click rates shoot up again. But a few days later, your green button is again performing at an all-time low.

It’s easy for an initial number spike to obscure a serious issue. Launching a new feature into an existing product is especially nerve-wracking because you only have one chance to make a good first impression. If your users don’t like it the first time, they likely won’t try it again and you’ve squandered your best opportunity. Continuously making changes to artificially boost numbers leads to feature-blindness and distrustful users. Given all of this, how and when can you determine if a product is successful?

  • Instrument the entire product flow. Don’t log just one number. If you’re adding a new feature, you most likely want to know at least three stats: 1) what percentage of your users click on the feature, 2) what percentage complete the action, and 3) what percentage repeat the action again on a different day. By logging the smaller steps in your product flow, you can trace the usage statistics within all of these points to look for significant drop-offs.

  • Test in sub-communities. If you are launching a significant new feature, launch the feature in another country or in a small bucket and monitor your stats before launching more widely.
  • Dark-launch features. If you are worried that your feature could impact site performance, launch the feature silently without any visible UI and look for changes in uniques, visit times, or reports of users complaining about a slow site. You’ll minimize the number of issues you might have to potentially debug upon the actual launch.
  • Anticipate a rest period. Don’t promise statistics the day after a release. You’ll most likely want to see a week of usage before your numbers begin leveling.
  • Test the discoverability of your real estate. Most pieces of your UI will have certain natural discoverability rates. For instance, consider adding a new temporarily link to your menu header for a very small percentage of your users just to understand the discoverability rates for different parts of your UI. You can use these numbers as a baseline for evaluating future features.

Pitfall 3: Users give conflicting feedback.

You are running a usability study and evaluating whether users prefer to delete album pictures using a delete keystroke, a remove button, a drag-to-trash gesture, or a right-click context menu. After testing a dozen participants, your results are split among all four potential solutions. Maybe you should just recommend implementing all of them?

It’s unrealistic to expect users to understand the full context of our design decisions. A user might suggest adding “Apply” and “Save” buttons to a font preference dialog. However, you might know that an instant-effect dialog where the settings are applied immediately without clicking a button or dismissing the dialog allows the user to preview their font changes immediately and saves the user from opening up the dialog repeatedly to make small font style tweaks. With user research, it’s temptingly easy to create surveys or design our experiments so study participants simply vote on what they perceive as the right solution. However, the user is giving you data, not an expert opinion. If you interpret user feedback at face value, you typically end up with a split vote and little data to make an informed decision.

  • Ask why. Asking users for their preference is not nearly as informative as asking users why they have a preference. Perhaps they are basing their opinion upon a real-world situation that you don’t think is applicable to the majority of your users (e.g., “I like this new mouse preference option because I live next to a train track and my mouse shakes and wakes up my screen saver”).

  • Develop your organization’s sense of UI values. Know what UI paradigms (e.g. Mac vs. Windows, Web vs. Desktop, etc) and UI values (e.g. strong defaults or lots of customization, transparency or progressive disclosure) your team values. When you need to decipher conflicting data, you’ll have this list for guidance.
  • Make a judgment call. It’s not often helpful to users to have multiple forms of the same UI. In most cases it adds ambiguity or compensates for a poorly designed UI. When the user feedback is conflicting, you have to make a judgment call based upon what you know about the product and what you think makes sense for the user. Only in rare cases will all users have the same feedback or opinion in a research study. Making intelligent recommendations based upon conflicting data is what you are paid to do.
  • Don’t aim for the middle ground. If you have a legitimate case for building multiple implementations of the same UI (e.g., language differences, accessibility, corporate vs. consumer backgrounds, etc.), don’t fabricate a hodgepodge persona (”Everyone speaks a little bit of English!”). Instead, do your best to dynamically detect the type of user situation upfront, automate your UI for that user, and offer your user an easy way to switch.

Pitfall 4: Any data is better than no data, right?

You are debating whether to put a search box at the top or the bottom of a content section. While talking about the issue over lunch, your BD buddy suggests that you try making the top search box “Search across the Web” and the bottom search box “Search this article” to compare the results between the two. You can’t seem to place your finger on why this idea seems fishy though you can see why this would be more efficient than getting your rusty A/B testing system up and running again. Sensing your skepticism, your teammate adds, “I know it’s not perfect, but we’ll learn something about search boxes, right? I don’t see a reason not to put it in the next release if it’s easy?”

The human mind’s ability to fabricate stories to fill in the gaps in one’s knowledge is absolutely astounding. Given two or three data points, our minds can construct an alternate reality in which all of those data points make flawless sense. Whether it’s an A/B test, a usability study, or a survey, if your exploration provides limited or skewed results, you’ll most likely end up in a meeting room discussing everyone’s different interpretations of the data. This meeting won’t be productive and you’ll either agree with the most persuasive viewpoint or you’ll realize that you need a follow-up study to reconcile the potential interpretations of your study.

  • Push for requirements. When talking with your colleagues, try to figure out what you are trying to learn. What is the success metric you’re looking for? What will the numbers actually tell you? What are the different scenarios? This will help you determine the study you should run while also anticipating future interpretations of the data before running the study (e.g., if the top search bar performs better, did you learn that the top placement is better or just that users look for site search in the upper left area of a page?).

  • Recognize when a proposed solution is actually a problem statement. Sometimes someone will propose an idea that doesn’t seem to make sense. While your initial reaction may be to be defensive or to point out the flaws in the proposed A/B study, you should consider that your buddy is responding to something outside your view and that you don’t have all of the data. In this scenario, perhaps your teammate is proposing running the search box study because he has a meeting early next week and needs to work on a quicker timeline. From his perspective, he’s being polite by leading with a suggestion without realizing that you don’t have the context for his suggestion. However, after pushing him for what problem the above study will resolve, you can also help him think through alternative ways of getting the data he needs faster.
  • Avoid using UX to resolve debates. UX might seem like a fantastic way to avoid personal confrontation (especially with managers and execs!). After all, it’s far easier to debate UX results rather than personal viewpoints. However, data is rarely as definitive as we’d like. Conducting needless studies runs the risk of slowing down your execution speed and perhaps leaving deeper philosophical issues unresolved that will probably resurface again. Sometimes we agree to a study because we aren’t thinking fast enough to weigh the pros and cons of the approach, and it seems easier to simply agree. However, you do have the option of occasionally saying, “You’ve raised some really good points. I’d like to spend a few hours researching this issue more before we commit to this study. Can we talk in a few hours?” And when you do ask for this time, be absolutely certain to proactively follow-up with some alternative proposals or questions, not just reasons why you think it won’t work. You should approach your next conversation with, “I think we can apply previous research to this problem,” or “Thinking about this more, I realized I didn’t understand why it was strategically important to focus on this branding element. Can you walk me through your thinking?” or “After today’s conversation, I realized that we were both trying to decrease churn but in different ways. If we do this study, I think we’re going to be overlooking the more serious issue, which is…”

Pitfall 5: By human nature, you trust the numbers going in the right direction and distrust the numbers going in the wrong direction.

Hours after a release, you hear the PM shout, “Look! Our error rates just decreased from .5% to .0001%. Way to go engineering team! Huh, but our registration numbers are down. Are we sure we’re logging that right?”

Even with well-maintained scripts, the most talented stats team, and the best intentions, your usage statistics will never be 100% accurate. Because double-checking every number is unrealistic, you naturally tend to optimize along two paths: 1) distrust the numbers that are going in the wrong direction and, more dangerously, 2) trust the numbers that are heading in the right direction. To make matters worse, data logging is amazingly error-prone. If you spot a significant change in a newly introduced user activity metric, 9 times out of 10 it’s due to a bug rather than a meaningful behavior. As a result, five minutes of logging can result in five days of data analyzing, fixing, and verifying.

  • Hold off on the champagne. Everyone wants to be the first to relay good news so it’s hard to resist saying, “We’re still verifying things and it’s really early, but I think registration numbers went up ten-fold in the last release!” Train yourself to be skeptical and to sanity-check the good news and the bad news.

  • QA your logging numbers. Data logging typically gets inserted when the code is about to be frozen. Since data logging shouldn’t interfere with the user experience, it tends not to be tested. Write test cases for your important data logging numbers and include testing them in the QA process.
  • Establish a crisp data vocabulary. Engagement, activity, and session can mean entirely different things between teams. Make sure that your data gatekeeper has made it clear how numbers are calculated on your dashboards to help avoid false alarms or overlooked issues.

One of the main tenets of user research is to constantly test the assumptions that we develop from working on a product on a daily basis. It takes time to develop the skills to know how to apply our UX techniques, when our professional expertise should trump the user’s voice, or when to distrust user data. As a researcher, you are trained to keep an open mind and to keep asking questions until you understand the user’s entire mental picture. However, it’s that same open-mindedness and willingness to understand the user’s perspective that makes it easy to assume that because their perspective can make sense, that it should also justify changes within our product design. Or, because we are so comfortable with a particular type of UX research, we tend to over-apply it to our team’s questions.

While by no means a complete list, I hope these five pitfalls from my personal experience will be relevant to your professional lives and perhaps, provide some food for thought as we all strive to become better researchers and designers.