11436 SSO

Lessons from a Customer Security Incident

Jan 13, 2015

Building products used by demanding customers is hard. Running operations is hard. Safeguarding customer data and infrastructure is hard. We’ve benefitted from all who have traveled on this journey before us, and we’d like to share our experiences and lessons learned: what works, and what doesn’t.

In this post, we share a summary of a recent security incident and the lessons we learned from handling the attack.

In early December, an unauthorized party accessed credentials for two of our customers for accessing the Apigee Management Service. These credentials were stored in a third-party hosting service, which hosts our developer portal. The intruder accessed the credentials by exploiting a known Drupal vulnerability before the fix patch was applied, and then made a few configuration changes using the compromised credentials. Since these changes were unusual, they caught our attention. Following our standard first-response security procedures, we acted quickly to protect our customers by disabling public access to our Management Service.  

We then performed a thorough investigation to assess the extent of the attack and found it was limited to the two customers and the set of configuration changes made. We restored all the customers’ original configuration from backups and took steps to prevent similar attacks.

A painful lesson learned from this incident: we weren’t following the principle of "least privilege.” The credentials stored in the third-party service had more privilege than they should have. Had we been more diligent about this, the intruder would have had a much harder time accessing the credentials.

The incident raises several important security best practices to keep in mind:

  • Limit your user list to those who actually need to use the product

  • Periodically review your user access list to ensure an up-to-date user list; delete users no longer associated with your organization or the job function

  • Use RBAC (role based access control) to assign users the level of access they need to get their jobs done and no more—we’ve defined several roles to make it easy for you to do this

In our cloud offering, we run tens of billions of API calls per month that interact with our customers’ back-end systems. Our on-premises software handles an equally large number. The lessons we’ve learned and the myriad architectural choices we’ve made enable our customers to: scale their API traffic (both stateless and stateful); create isolation in a multi-tenant environment; provide flexibility to configure or code; support an analytical system that doesn't require tuning as apps and APIs change; and enforce security with access control and threat mitigation.

For more on security best practices, see the following articles and posts:

Apigee is hiring. If you’re an engineer excited about tackling challenging problems, visit our careers page.

Scaling Microservices