AnalyticsOps: A Natural Extension of DevOps

By Vishwa Kolla, AVP & Head of Advanced Analytics, John Hancock Financial Services

Vishwa Kolla, AVP & Head of Advanced Analytics, John Hancock Financial Services

In 1997, when Deep Blue beat Kasparov, it had to crunch through atleast 5K scenarios on each move. In 2016, when AlphaGo beat Lee Sedol, it had to crunch through at least 5M. It is not surprising to see why Analytics professionals take pride in building the most sophisticated models. That said, the real value of Analytics is in the sophistication of its implementation.

In his book, Deep Thinking, Grand Master Garry Kasparov says “Weak human + machine + better process is superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.”

DevOps is a culture that enables IT functions to significantly reduce time to value. Organizations that embrace this culture create value in an iterative, incremental, collaborative and a holistic manner. Analytics functions too, can and should embrace such a culture. Doing so helps resources to be better utilized in solving the right problem well, as opposed to solving any problem. According to revenue figures from Capital IQ, firms that have embraced an AnalyticsOps mindset have enjoyed a sustained Cumulative Average Growth Rate (CAGR) of 7-12 percent over 1.3 decades. Their CAGRs handily beat respective industry averages of 1-5 percent. An operating model for AnalyticsOps might have some of the following elements with overlapping DevOps principles.

“Feature automation helps with creating a structured process of mining gold from the data ore. DevOps principles: Automated”

First: Think Shared Value: It is rare that products are built in silos. Cross-functional, -regional and-organizational collaboration is paramount for a successful launch. Making Product, Business, and IT organizations partners in solution development will help with building it right the first time. DevOps principles: Holistic, Collaborative.

Second: Answer So-What: Assume we have perfect information (and models). So what? Answering this question helps one think through success criteria and gives a decision framework. For example, if a person targeted for an offer responds, so what can the firm sell her? What elements will help personalize the buying experience? Such a construct helps with scoping Analytics projects so that they go beyond building propensity models to building for example a next best action model. These models work in conjunction to improve conversion rates and add to the bottom-line. This principle may be more unique to AnalyticsOps.

Third: Overweight speed over accuracy: Often, Data Scientists will want to stitch all available sources and frequently trade marginal returns of extra precision for the enormous amount of time that goes into engineering that value. A parallel from agile, Minimum Viable Process (MVP), can help. Analytics professionals should think minimum viable set of inputs or MVI. An MVI construct can help prioritize feature integration into sprints. Should an Analytics project be paused for any reason, the project (and models) will still have had demonstrable value. DevOps principles: Iterative, Continuous.

Fourth: Emphasize repeatability: Analytical models extensively create and leverage synthetic features. Often, these are latent drivers of a predictive model. For example, in absence of crash history, auto-insurers use Credit Score as a proxy for characterizing the crash-risk of their insured. These features are generally more complex and need to be iterated on to shape them. If one does not automate and document lineage, such synthetic features can easily get lost between model iterations. As new data comes in, models will likely produce different results leaving Analytics professionals spending more time explaining the deviations as opposed to building better models. Feature automation helps with creating a structured process of mining gold from the data ore. DevOps principle: Automated.

Fifth: Visualize, Illustrate and Tell Stories: Lastly, as the saying goes, a picture can speak a thousand words. In several of my projects, putting context around inputs, transformations and outputs significantly helped with improving model adoption. Additionally, walking through a predicted output and the top contributory reasons made it real for insight consumers. In our experience, a supervised shallow learning model performed at least 10x better than a sophisticated deep-learning model to detect suspicious activities. The enabler was a self-serviced visual association between inputs and outputs. For example, a human would better associate a lower NPS score to word and phrase triggers such as “not happy, unhappy, dislike, I’m frustrated” etc. The illustrative tool helped both the accuracy and rate of algorithmic learning process. DevOps principle: Self-Service.

In summary, AnalyticsOps is a natural extension of DevOps thinking. Firms that successfully embrace this culture will reap a greater ROI on their Analytics investments.

Regulatory Technology Special