Before we get started on our discussion of the Amazon Pinpoint service, we need to introduce some important concepts related to that technology.

Most programmers are probably familiar with the concepts of instrumentation, but may not be familiar with what that term means. The term instrumentation, when used in reference to computer programming, is used to describe code added for monitoring and measuring the performance of a product. It also involves code added for writing trace information to logs, for diagnosing issues. So, anytime a developer is writing complementary code to provide information designed for analysis and troubleshooting, they're implementing instrumentation.

A/B testing is a technique for comparing two or more different versions of a concept or product to determine which one performs better. Users are often randomly selected to use one of the versions, performance data for each version is gathered, and the data is analyzed to see which version is most effective.

Holdout testing is a type of statistical analysis that is used to determine if there is truly a lift from a marketing campaign. Lift is a marketing term used to describe any increase in revenue generated from a campaign. Marketers use holdout testing to verify that a perceived lift after a campaign is truly the result of the marketing effort. To institute holdout testing, a measurable segment of the users are defined as holdout users. These users will not receive the marketing email, push notification offer or other campaign material that the rest of the users are receiving. Once a campaign is over, the conversion rate of each group of users is compared to see if the promotion was worth the cost of running it. So, if your holdout users purchase the same amount or more than the recipients of the marketing push, then the lift was not a result of the campaign, and therefore the campaign was not successful. It means that the company gave away a percentage of profits when it was not necessary. Holdout testing is often used periodically to verify that money spent on marketing campaigns is adding value.

The term funnel analytics describes an analysis technique that's used to segment a group of users into categories, and measure conversion rates based on those categories. This practice can help companies determine the effectiveness they're achieving in certain vertical markets, and potentially increase or decrease marketing efforts based on the value of targeting a specific segment.