Customer Churn Analytics

I've had a great opportunity recently to work on a customer churn predictive analytics project. The goal is to predict which customers are likely to churn in the near future.

For tools I'm using all SQL Server 2008 applications; ssms, ssas, ssis, ssrs. What a great toolbox. After spending a little time learning about SAP, SPSS, and Statistica, I can honestly say that MS has a great stack for those interested in using predictive analytics to drive business decisions resulting in a very high ROI.

I'm not able to share any details on the project, but I will share the high level status. We are able to identify around 45% of the customers that will be considered lost in the next 2 months. This comes at a cost of a very manageable false positive rate.

Below is the lift chart showing how the model is performing. Training data consisted of a 2 year period ending in Jun 2008. This chart shows how well the model predicted lost customers for a 2 year period ending in June 2009. The business being analyzed is seasonal which led us to a monthly segmentation. The model has been verified in several ways including cross validation, lift analysis, classification, and decile performance.

There are plenty of ideas on the table for improving the model's accuracy, but the strides taken thus far have a clear business value which I'm hoping to be able to report on during upcoming posts. Model improvements could go on for quite some time in the future. On the top of my list include more attribute grooming, decision tree bagging, ensembles, over sampling, and exploring a few other algorithms in more detail.

PAW 2009 (Alexandria, VA) - Conference Summary

PAW was a great conference. Nearly every session/workshop I attended was full of either “how-to’s” or very relevant case studies facts. Below I’ll summarize my impression of the sessions attended.

Day 1: SAS Hands on workshop (Dean Abbott, President Abbott Analytics)

On day one I attended a full day workshop. The goal of the workshop was to get hands on experience building models. There was also an emphasis on using SAS’s Enterprise Miner. I have never worked with SAS before so it was great to play around with the tool and see what has made SAS the hands down leader in modeling software.

The workshop started with an overview of the data mining process. We talked about the CRISP modeling process in depth. A SAS representative attended the session and gave us all a brief introduction to SAS’s enterprise miner. Then we got started digging through the data.

Most of our time was spent transforming data. SAS has some very nice wizards and built in components that make data sampling, transformations, bootstrapping, etc… seamless to the end user. After getting the data prepared we reviewed descriptive statistics and began making modeling decisions. In the end we built several models using different algorithms. I chose to use neural networks and decision tree. In the end the ensemble of decision trees was the most accurate model.

Key Takeaways:
- SAS and most other modeling software pulls data out of the warehouse, stores it on a client machine or server, and then begins doing transformations & modeling. This results in additional management overhead. This began my thoughts on the advantages of in-database modeling such that SQL Server & Oracle offer.
- SAS only considers case level data. SQL Server offers the ability to look at nested data for each case. For example; consider a model where the case level data is a customer. We may want to evaluate customer transactions (nested). To do this in SAS we will need to summarize the transaction level data up to the case level data. In SQL Server we can simply included the nested transactions in the model structure.

Day 2 & 3: Lecture Sessions (Mostly case studies)

Keynote: Five Ways to Lower Costs with Predictive Analytics (Eric Siege, Ph.D.)

Great introduction to the series of case studies to follow. Eric Siegel was the conference Chair and a very nice guy. The keynote touched on many topics briefly and gave a good overview of data mining in general. An emphasis was placed on uplift modeling and the various ways that predictive analytics can add value.



Case Study: National Rifle Association; How to Improve Customer Acquisition models with Ensembles

There were several sessions and workshops that emphasized the power of ensembles. Dean Abbot (also taught Monday’s workshop) started the trend with this session. The concept is counter intuitive. Basically, what is happening is that we are running a model on a subset of randomly selected cases repeatedly and then simply averaging the results. In many cases the ensemble score is higher than any of the individual model scores and nearly always provides a better result long term. Great presentation.



Multiple Case Studies: Anheuser-Busch, Disney, HP, HSBC, Pfizer, and others; The High ROI of Data Mining for Innovative Organizations (John Elder, Ph.D.)

John Elder is a great mixture of academics and business skills. In this session he listed a series of data mining situations and concluded with the results. Key point – Not all of the data mining examples were considered successful. Reasons for failure were typically business process or politics related.

Keynote: Predictive Analytics over On-line and Social Network Data (Usama Fayyad, Ph.D.)

There was an emphasis on using social network data as a behavioral or attitudinal input to mining models. The concept is very interesting. Marketing departments realize that friends and family members have much more influence that advertising. Finding a way to leverage social networking is one way to impact behavior. In this presentation Usama (former CIO of Yahoo) talked about how yahoo presents ads to users. Yahoo is tracking individual search request to identify trends. They also are evaluating the longevity sensitivity. How long is a search request relevant to the user? Very interesting stuff. Some of this again raises privacy issues that have yet to be hashed out.

Case Study: Target Challenges of Incremental Sales Modeling in Direct Marketing (Andrew Pole)

Target has done a lot of modeling around customer uplift. Andrew is focused on customer level uplift. The key problem here is how to determine if marketing has provided uplift for an individual. Typically models look at groups of people or profiles and measure results at an aggregate level. At the individual level, it is much more difficult to quantify results because we don’t have a hold out set. In other words, we can’t test the result of a mailer because a single customer can’t receive and not receive a mailer. Kind of a narrowly focused issue, but interesting.

Case Study: Optus (Austrailian Telcom) Know your Customers by Knowing Who They Know and Who They Don’t (Tim Manns)

Tim was a great speaker. He is working for Optus who is very aggressively trying to use relationships to improve customer retention as well as drive customer growth. In short, they are looking at who customers are calling and how often they are calling to create relationship networks. When the networks are created they can be used in marketing efforts to impact customer retention. For example, when a customer churns, then related customers are at an increased risk of churning. Marketing can take action to prevent customer loss.



KDDCup 2009 Competition Results: Orange Labs (France Telcom)

Again, an emphasis on Decision Tree ensembles was presented. Below is a graph that was used to show the uplift that the model could provide.

Case Study: Citizens Bank; Building In-Database predictive Scoring Model: Check Fraud Detection (Jay Zhou, Business Data Miners)

Dr Zhou was not the best speaker, but his presentation was great. He was focused on in-database modeling which I am also very interested in. Why pull the data out, transform, model, and then try to find a way to get all of this back in the database? There was not a lot of in-database talk, and neither Oracle nor Microsoft was at the conference. Additionally, mining languages were not discussed in much depth at all. I asked several people if they have any exposure to DMX and not one person even know the language existed. I’m not sure if that is a good or bad thing or if it is just a result of being at a conference where SAS and SPSS were the main sponsors.

Keynote: Opportunities and Pitfalls: What the World Does and Doesn’t Want from Predictive Analytics (Stephen Baker, Business Week Author)

Again, the presentation is not available for this session which is too bad because Stephen Baker did a great job of presenting some of the issues with PA. Stephen is an author of Business week as well as the author of The Numerati. He talked quite a bit about the risk associated with PA as well as the creepiness factor that being too accurate at predictions causes. I have not read his book yet, but I did order it from Amazon this week.

Case Study: The Financial Times, The New York Times, Sprint-Nextel – Predicting Future Subscriber Levels (Michael Barry, Data Miners, Inc)

Michael Barry presented a subscriber demand level forecasted method that should be a little more accurate than traditional methods. He calls the method Hazard Probability. Basically, he is looking at the likelihood that a customer is going to churn at particular points in the future. This likelihood is then applied to existing and new customers. Fairly simple method, but clearly provides more insight into future demand as well as how marketing efforts impact future demands.

Case Study: Coke – A Predictive Approach to Marketing Mix Modeling (Ram Krishnamurthy, Coke)

Coke has spent a lot of time and effort on marketing channel optimization. Is it better to spend money on TV advertising, Print, Radio, or Billboards? There are many variables at play and with Coke’s many brands this issue becomes quite complex. The presentation does a good job of presenting the issue as well as the way Coke has tackled it.



Case Study: Lifeline Screening- Segmented Modeling Application in Health Care Industry (Ozgur Dogan, Merkle)

Merkle’s Ozgur Dogan presented a case study based primarily on segmentation analysis. Instead of looking at all cases in a single analysis prospects are segmented based on some logical grouping. Uplift for the individual segments can be much higher in by segmenting prior to modeling.

Lessons that we Learned from the Netflix Prize (Istvan Pilaszy, Gravity R&D)

This guy was a pure genius. I really did not understand most of the presentation because the equations were over my head. It would take me a lot more time to go through each equation to truly understand the details of the results. However, I do know that ensembles were used along with a weighting strategy to more accurately predict the movies that customers would like to view. Below is my favorite equation from this presentation.



Day 4: Full Day Workshop – The Best and the Worst of Predictive Analytics: Predictive Modeling Methods and Common Data Mining Mistakes (John Elder, Ph.D.)

I can’t say enough about how good John Elder’s workshop was. Please do read through his presentation. I copied a few of his slides here just to give a little view into the concepts he covered. Below is a great pictorial that describes how different algorithms attempt to fit a dataset. We talked about the advantages/disadvantages of each of these models.



The next two slides show the power of ensembles. First we looked at a slide to see how algorithms performed over several datasets. Below you can see that neural networks was probably the best overall performer, but there is none of the algorithms were the best for every dataset.



Next we looked at the impact of ensembles. Each algorithm was included in an ensemble (multiple algorithms as opposed to boostraped). Different methods were used to combine the results. Look at the huge improvement. Clearly ensembles need to be considered in any modeling process.



Another view of and ensembel of trees resulting in higher uplift than any of the individual trees could offer.



John Elder is one of the author of Handbook of Statistical Analysis and Data Mining Applications. I have yet to finish the book, but it is going to be on my desktop as a reference for a long time to come. He combines statistics, data mining, and applications to give a uniquely complete view of the modeling process. John has not lost sight of the ROI goal that DM should offer and is a great speaker.

Overall this workshop was a great. We hit on many topics some of which could take days to get a deep understanding. I’d recommend John’s workshops to anyone interested in learning the nuts and bolts of data mining and hope to attend more of his sessions in the future.

Conclusion

Predictive Analytics World was a great success. I learned a lot and feel like I at least know what the industry is doing in the DM space. LiveLogic has some key advantages as we currently have a high level of expertise in handling large amounts of data as well as in-database transformations. We also have a very nice platform to work with in SQL Server. Our focus should be on improving statistical knowledge, algorithm usage, modeling experience, and in building our own case studies that we can leverage.

Predictive Analytics World 2009

It's only 2 weeks away and at this point I'm about 75% sure that I'll be able to attend the full 4 day event which is pretty exciting. A couple changes at work have lightened the end of project time crunch a little.

PAW's agenda is full of keynotes, case studies, and workshops that I can't wait to attend. This should be a very good event to network with like minded people as well. I'll be sure to post updates here throughout the event.

Below is a micrpoll displaying what people are most interested in attending at PAW.

Website Polls Powered By MicroPoll

Gartner CRM Summit

What a summit! The Gartner CRM Summit was packed with many interesting and informative lectures and workshops. Going into the conference Jon (LiveLogic founder) and I were primarily interested in surveying the level of interest in customer analytics. We also hoped to gather success stories in the form of case studies focuses on analytics as applied to CRM. Gartner delivered on these expectations and much more.

While there were a few software references the overall tone of the conference was not tool centric. The magic quadrants were referenced in several of the lectures I attended, but this was always followed by a disclaimer. Instead, the focus was on services (internal and external).

The ongoing message was around adding value through analytics. Several lectures were directly focused on this topic and many others included analytics as part of their discussions as well. It is great to hear the emphasis on adding measureable value at a marketing summit. There were also many references made to the need for quality data sources. MDM and data quality were discussed in detail. The industry is clearly changing.

Great job Gartner.

http://www.gartner.com/it/page.jsp?id=685208

The Never Ending Tools Debate

Rarely does a week go by without some sort of BI tools debate. You don’t need to look very far. Linkedin discussions, Gartner, BEYE Network, etc… all expose the intense emotions that many of us have about the tools we use daily. Those of us that have chosen to specialize in the Microsoft suite of products are a common target. Here are a few of the most common remarks related to Microsoft BI tools.

1. Not capable of enterprise level deployments due to lack of metadata control
2. Dumbed down BI with extensive wizards and point-and-click features
3. Lack of industry focused solutions
4. Depreciate the value of BI professional services
5. Newcomer to the BI industry therefore not on the same level as traditional players
6. MS is not a serious tool because the license cost is so low

The list goes on and on, but what it comes down to is that Microsoft is seen as a serious threat because of the comparative value their BI stack offers. Some of the criticism has basis, but much of it is based on people’s feelings. Although I feel that each of these points are false, I'm not going to take the time to debate them because my intent is to recommend a change of focus away from tools.

I propose a truce. Let’s shift the discussion from the tool’s features to the ability to deliver a value added solution to our clients. IBM, Oracle, SAP, MS, etc… all have tools that are capable of getting the job done. By constantly focusing on tools we are creating an impression that the tool is the most important part of a BI solution. Instead, let’s discuss the methods that we, BI professionals, use to add value for our clients. It would be great to see more discussion around concepts such as data mining and decision support solutions.

Don't get me wrong, I am not saying that tools aren't part of the equation. Features that our tools offer directly impact the final solution's ability to add value. I am suggesting that we keep the tool portion of the total solution in perspective.

External Decision Support Systems (EDSS)

Having spent a few years in the BI industry working with quite a few clients it has become very clear that many companies are taking on similar BI projects. It makes perfect sense that companies belonging to a particular industry will need to make very similar decisions. How these companies approach a particular problem (decision point) may differ widely. Some may invest heavily in the best decision support system imaginable, and others may leave these decisions to the intuition of individual contributors.

There is opportunity to add value by providing highly focused External Decision Support Systems (EDSS) to targeted verticals. The model for such of a service would be based on the below principles.

1. EDSS is owned, housed, operated, and maintained by the vendor

2. A common decision (highly focused) among multiple organizations is identified

3. The EDSS will be provided input data to include sets provided by clients as well as sets of external data common amongst all clients

4. Algorithms are created and executed by the EDSS engine which convert the inputs into the desired output (decision)

This model provides the client with very clear advantages.

1. No need to build or maintain a BI project required to support a particular decision (no cost of ownership)

2. Client benefits from the investment of other companies dealing with the same problem/decision point.

3. External data that would be too costly for individual companies to acquire becomes cost effective for a pooled resource to make available.

The vendor that provides EDSS becomes a true specialist in the BI system that is being offered. This level of expertise does not exist in today’s distributed BI project model as each company is limited to the expertise that can be developed in house.

Target EDSS’s will need to focus on decision support systems that require high levels of analysis. By designing and delivering a tool that overcomes the many obstacles associated with building a truly analytical BI system, EDSS providers will add value.

Focusing on the Core

To be competitive in today’s business environment companies must have a market advantage. Clearly the days of opening up the only company serving a starving demand are over. If a company is innovative enough to identify or create a new or underserved market, then it will not be long until a slew of undercutting competition raises the stakes. This makes it extremely important that companies focus on what they do best; i.e. their core competency. By maintaining focus on the core businesses stand a much better chance of maintaining the market advantage which was the foundation for existence at startup.

Given that there is value in maintaining focus on the core, let’s examine how a business might determine which functions to keep in-house and which should be outsourced. The basic formula should not change.

If in-house total cost is greater than outsourcing total cost, then outsource.

This is a very simple guideline, but the key is in the word “cost”. Below is a list of costs that should be evaluated.

Complexity – What is the costs of adding complexity to a business? If a company is in the business of manufacturing a product, then what is the impact of adding the burden of managing a BI system in addition to maintaining the core business? Will focus be lost or lessened? If so, what are the risks associated with this shift of focus?

Management – What is the cost of managing a BI project? Are there resources currently available in-house with the expertise needed to manage a BI project? If not, will the project be added to an existing manager’s function? What are the risks associated with managing a BI project with under skilled personnel?

Product Selection – All BI projects utilize a core set of software products to include ETL, database, and front end. Which products will be purchased? Will these products provide the desired functionality? Is there in-house expertise available with broad enough product experience to make a product decision? What are the costs associated with developing the BI project on software the fails to deliver the desired results?

Architecture – Perhaps the most important decisions that are made during the design phase of a BI project are architecture related. Is there enough in-house expertise to make these decisions? What are the associated costs with architectural mistakes?

Value – What level of value will be added by an in-house BI project development team? How does this compare to an outsourced BI project? Time to delivery should be considered. What is the time value of the BI project? Annual value of BI project / 12 months = Cost per month BI project is delayed

This short list of considerations is meant to give some insight into the costs that need to be evaluated when considering whether to complete a BI project in house or to outsource. It is important to the BI project’s success as well as the company’s success that this decision is made with care.

BI Vendor Categories

The Business Intelligence Industry, like all others, is made up of vendors that provide products, services, or combinations of both. Each of these vendors offers value in a unique way. This blog categorized BI vendors into one of three categories from a value added perspective.

Software Sales
Companies that off software products as their primary line of business fall into the software vendor category. Software vendors can be further categorized by the type of product being offered. Database vendors include Microsoft, Oracle, SAP, etc... Front-end vendors include Panorama, Qliktech, Cognos, etc… Software sales BI vendors produce raw software packages that buyers then installs, configures, and customizes to meet their needs. Some software sales companies are offering their product in software-as-a-service format as a low cost alternative.

Packaged Products
A second group of BI vendors provide customized software packages which deliver focused functionality to a particular client group. Many times packaged product vendors build their products on top of software sales vendor products. Software such as enterprise resource planning often falls into this category. Customer relationship management may also fall into this category. Oracle Hyperion and SAP BPC will also fall into this category. All of these products provide clients with a solution to a specific problem as opposed to a software sales product. For example, a database is an tool that client use to build a solution while a CRM package provides a solution for the clients specific need. Some packaged product vendors offer their product in the software-as-a-service format.

Body Shops
Finally, a third category of BI vendors provide services as their primary offering. These companies are hired by their clients to consult, design, build, modify, maintain, etc BI software for their client’s needs on a custom basis. While software sales and packaged product vendors typically offer services to their client, this is a secondary line of business for these vendors. Body shops provide these services as their primary line of business. Many times partnerships will exist between body shops and software sales vendors providing software sales vendors the ability to focus on their core product (software) while body shops specialize in a particular software segments.

Each of these three categories of BI vendors attempt to add value in a very unique way. It will be the topic of future blogs to discuss the pros and cons of each category’s approach.

My New BI Blog

Welcome to my new blog - BiVal. I've decided to start writing short essays on some of the lessons that I have learned (or will learn) while working in the BI field. The subject of all of my writing will revolve around business intelligence applications in some way. While some post may be technical I plan to spend most of my time writing about the value added side of BI. Specifically, my focus will be on adding value through improved data backed decision making.

Clearly this is not the blog that the masses will want to subscribe to or add to their feed list. However, I do hope to reach a few BI professionals as well as business minded individuals that may be pondering the same thoughts.

Till next time...