You just did a major update to your mobile application. A few days later, the Vice President of Marketing complains that he's starting to see a lot of 1-star (that is, low rating) reviews that the “app is not working”. The description of the problem is vague at best, and you don’t know if the problem is a bug in the mobile application, poor performance of the application that makes it seem that it's not working, or that one of your API dependencies is not working. You also know that you skimped on testing, so you're concerned that there are edge conditions regarding devices, carriers, and operating systems that have not been tested. Furthermore, you don't know if this is something that has a major impact on customers.
You can use Mobile Analytics to investigate what is stimulating these "app is not working" reviews, and assess its impact on customers.
You can start investigating the issue by monitoring the usage of your mobile application using Mobile Analytics.
- Sign in to Mobile Analytics if you haven't already done so. This displays the App Overview page where you can view information about your apps.
- Select Analytics in the main menu. This expands the App Usage menu item in the left navigation menu and opens Sessions. (You can also view app usage by clicking the monitor icon next to your app in the App Overview page.) You should see a graph that displays the number of active sessions and users for your application.
- Select a time span such as 1h (1 hour) over which to view the monitored data.
- Check the checkbox labeled "Compare with last week". Examine the display for spikes or drops in sessions.
You can also select Session Duration in the left navigation menu to see how long (on average) users are using your application. Significant drop-offs could indicate that users tried launching the application, were unable to complete their actions, and subsequently stopped using the app.
If you need more drill-downs to monitor usage, Apigee recommends using other products such as Google Analytics or Flurry.
Investigate application logs and network metrics
The next step is to examine your error logs. Doing this is probably the best way to determine if there are bugs in your application. You can view the error logs by selecting App Log Analysis in the left navigation menu. This expands the menu item and automatically selects Errors in the submenu. Select a time span over which to view the logged error data.
To determine if your new feature is the root cause of the bugs, check the “Compare with yesterday” or “Compare with last week” checkbox. This will compare current error counts with those of yesterday or last week. You can use that display to see if there is a increase in your bugs.
If your application is crashing, a special log message “Crash Occurred” will be displayed in your error log. Apigee will soon make the stack traces available for download, but for now use Google’s Play or Apple AppStore to download your stack traces (see Improve Stability and Eliminate Bugs and Distributing Apps to learn more).
Filter your log errors
Mobile Analytics provides a number of ways to filter your errors so that you can better isolate the source of a problem. For example, select Errors by Device Platforms in the left navigation menu. This displays errors encountered in each device platform, such as Android or iOS. Look for dramatic differences in the number of errors for one type of platform as compared to the others. In this way you can better determine if the problem is due to your software or due to a specific device platform.
Similarly, by selecting Errors by Device Models, and then comparing the number of errors on specific device models such as an Apple iPhone 4g or HTC Thunderbolt 4G, you can better determine if the source of the error is a specific device model.
Search for errors specific to the new function
Another way to filter your errors is to list only those errors that are specific to the new functionality. Each of the App Log Analysis views, such as Errors by Device Platforms, also lists the time stamp, severity level, and message for each logged error. If you've created specific messages for the new functions, you can search for those messages in the error log.
Expand error levels for log capture
You error log might not be recording things at a level for you to do the analysis you need. For example, your log might be capturing only errors, and not warnings. You can select a lower level of messages to be included in the log.
First, ensure that you integrated Configuration management into your application. See Add monitoring to your Android application for instructions on integrating the configuration APIs into an Android application. See Add monitoring to your iOS application for instructions on integrating the configuration APIs into an iOS application.
If you have integrated Configuration management into your application, select Configure in the main menu. This opens a configuration page, such as App Description. Then select Default Configs in the left navigation menu, and choose an appropriate level in the Log Capture Levels drop-down box.
Target a specific device
If you've isolated the problem to a particular device — suppose, for example, there is a customer who can reproduce the error — select Beta Test Configs, then enter the customer's telephone number to log errors only for that phone.
Examine the network
If the logs don't help you isolate the source of the problem, the next step is to look at your network. Often, application errors are caused by the interaction with backend systems, or even due to the mobile carriers. Poor network performance can also be perceived as a failure.
You can get an overview of your network by selecting Network Performance in the left navigation menu. This expands the menu item and opens the Overview. The Overview displays the number of requests, errors, and response time. Select a time span over which to view the network performance data.
The Overview also lists the average response time for each web service URL accessed by your app. Look for web services that are particularly slow.
Turn off new features
When you introduce a new feature, there is always the possibility that it may not work as expected or that the feature needs to be tweaked in production. By integrating Configuration management into your application, you can take advantage of the configuration management capabilities in Mobile Analytics to make needed changes in production.
If you have integrated Configuration management, select Configure in the main menu. Then select Beta Test Configs if you've isolated the issue as specific to a device or phone. Scroll down to the Application Specific Configs section. Turn the pertinent feature off by setting its value to false.
If one of your customers can systematically reproduce the bug, its recommended that you push the updated configuration to that specific device or phone. Make sure to enter the Apigee device Id for the customer's device in the Matching Apigee Device Ids box, or the telephone number (if the device is a phone running Android) in the Matching Telephone Numbers box. Enter each device Id or telephone number on a separate line.
Ask the customer to restart the application, and confirm that the configuration change has fixed the issue. After the issue has been fixed, push the updated configuration to all devices.
Analyze the effect of your changes
Confirm that your changes have a positive effect. Select Analyticvs in the main menu and look again at your application logs and network monitoring. If your configuration changes were effective, you should see a decrease in errors or improved network performance.
If you pushed an updated configuration to a particular device (using Beta Test Configs), examine the application errors or network performance on those devices compared to the other phones. Also, search your error logs to determine if errors related to the feature of interest are still occurring for that particular configuration.