11436 SSO

User experience, data quality, and dealing with mistakes

Feb 24, 2012

As discussed in previous posts, it's important for app developers and API providers to understand how to protect users, apps, and APIs from abuse and how to deal with malicious attacks when they happen.

It's also important to think about how to design protection systems to optimize impact on abusers while minimizing impact on legitimate users.

Good design for online safety systems requires the ability to dial the aggressiveness of the protection, detection, and remediation mechanisms up and down. This is useful when dealing with a wave of attack that is temporary. That is to say, the system might need to be fine tuned to be aggressive against traffic originating from a suspicious source followed by a reduction in the aggressiveness when the attack has ceded.

The flip side of having the ability to dial up and down the aggressiveness of the system is the potential of denying high quality service to legitimate users. Mistakes in this realm are called “false positives” and happen when good entities (legitimate traffic, users, apps) are labeled as “not good” by classification systems. As a result, they receive low quality of service, or are even denied service. Any and all such mistakes can impose a huge cost in terms of users, usage, and profits.

Any online safety technology requires a system of checks and balances to ensure that the aggressiveness of the system is fine tuned to maximize true positives and minimize false positives. There are various mechanisms to achieve such quality. However they are a topic for a different blog post.

For a number of reasons it is impossible to achieve a 0% false positive rate. First, abusers and attackers do their best to appear as legitimate users to the system. Secondly, there is a “learning latency” between the time it takes for a system to learn new bad behavioral patterns and “publish” them to their detection systems. In addition, systems will never be able to correctly distinguish new bad users/apps from new good users/apps and aggressive systems can bundle new good and new bad users/apps together.

One possible remediation for the inevitable false positive rate is to build a user experience that is simple, intuitive and minimizes the intrusiveness of such technologies in to the user’s workflow. It is critical to realize that an extremely large majority of users, apps, or developers who will find themselves in a remediation flow will be legitimate users (because this flow will break the cost model of the abusers who are interfacing with the service programmatically). Anything that can be done to reduce the cost this flow imposes on legitimate users should be high priority.

One caveat to the above assertion is that it is worth the abuser’s time to experience a remediation flow first-hand, as it allows them to figure out how to (at scale) circumvent the remediation flow. A well designed flow should not assume otherwise.

Scaling Microservices