Advancements in Search and the Promise of AI
In 1998, Google, along with a handful of other search engine companies, unlocked the doors to the Worldwide Web, by allowing users to ask all sorts of* factual *look-up questions. For example:
- "Who is the first president of the United States?"
- "How do I make a Gimlet?"
- "What’s the capital of China?"
Building on these advancements, in 2011 Apple Introduced Siri, a natural language user interface to answer questions, make recommendations, and perform various actions by delegating request to a set of Web services. For example:
- "Schedule a meeting for tomorrow at 1PM with Bill"
- "Remind me to register to vote tomorrow"
- "Play me some Beatles songs"
For many users, Siri is their initial foray into world of artificial intelligence. But is Siri really intelligent? Or is she just knowledgeable? If we define intelligence as the ability to solve problems and figure things out, then the answer is “sort of." Siri can figure out what you’re saying (most of the time), but it has to relay what you’re saying to a set of web services to then look up some fact. Siri gives us the illusion that she’s all knowing, but in reality her main trick is figuring out (*inducing) *what you’re saying, then looking up the question on the Web to *deduce *an answer. For Siri to be really intelligent she would be able to *induce *both what you are saying, and then *induce *an answer. Induction, or inductive reasoning, is what is needed to answer all sorts of inferential questions. Remember, the best deductionist is the calculator, which is hardly “intelligent. The promise of AI requires a system that can quickly and efficiently *induce *answers to queries. Such a system would allow users to ask inferential questions, and receive immediate answers. For example:
- "What food should I order here? (preferably something delicious)"
- "Who in the bar should I ask out on a date? (ideally, someone that likes me in return)"
- "How much should I charge Bill (to maximize my revenues)"
Currently, there isn’t any system that can answer all forms of inductive queries. Instead, there exist thousands of different companies, each dedicated to answering a specific inferential question, such as:
- Dating sites, which answer: “Whom should I date?”
- Food Site, which answer: “Where should I eat?”
- Travel Sites, which answer: “Where should I travel?”
On the enterprise end of things, numerous companies exist that answer similar, inductive questions:
- Which of my leads will likely convert to a deal?
- How much rent should I charge for my apartment?
- When will our customers quit using our service?
Each of these questions requires a whole company to answer for two simple reasons:
- There is no universal system that can answer real-time inferential questions
- The current state of AI does not allow for building inferential systems that are extensible across multiple domains.
If there were a single system that could answer all sorts of inferential questions, Siri for example, we would delegate all our queries to SIRI and numerous companies would either have to augment their offering, or go out of business.
Data Analytics vs. Search
Since there isn’t a universally pervasive AI system that can answer all sorts of inferential questions, companies rely on their internal staff of mathematicians and scientists (collectively, data scientists) to build models from heaps of data to predict outcomes, which the companies can then bundle and sell to consumers as scores, recommendations, matches, forecasts, and insights.
Data analysis can be a very complex undertaking, and is generally reserved for people with a postgraduate education, if for no other reason than these people have experience analyzing data to support their academic theses.
So, it seems common sense to hire academics from top-tier institutions, and move them into private sector jobs to analyze vaults of data and help companies build models that can predict various outcomes. But, approaching model of development from this perspective can be very costly, and can often yield poor results. The reason for this lies in the structural complexities associated with data analysis and subsequent model development.
What’s more, the skills necessary to build a predictive model are still not enough to deploy a model and make use of it in a production environment. For this, you need even more specialized skills ranging from systems administration, database administration, natural language data processing, and software engineering.
So why hire these hard-to-find, expensive, resources? From a business perspective, the answer is quite simple: because the people doing the hiring don’t have the background to manage disparate sets of skill sets. One person possesses all the necessary skills required to build a solution, then communication cost approach zero, at the expense of the cross-disciplinary resource, of course. However, if the hiring manager has the ability to manage a complex process, and can disaggregate the skill sets necessary to build *and deploy *a predictive model, then the costs to build and deploy a model decrease by order(s) of magnitude.
Of course the standard venn diagram does not speak to the distributions (market availability) for these skills. To better represent the distributions, we edit the sizes of the Venn circles. Normally, the distributions would not matter, but they are very important, if we are to disaggregate the skills into their own sets of skills, and hire individuals from each of those sets, and daisy-chain them together to form a virtual data scientist. “Why would we want to do this,” you may be wondering; the answer: you gain comparative advantage, absolute advantage, economies of scale, and bottom line: you spent way less money, and get much better results!
- Comparative Advantage: Each person does the task for which they have the lowest opportunity cost. Doesn’t it seem wasteful to have a PhD mathematician writing database scripts? Terrible!
- Absolute Advantage: Each person you hire is an expert their own field, not a jack-of-all-trades, and in hiring this way, you can find people that have a very nuanced expertise, something that we will find is critically important when building predictive models; statistical programming is all about nuance!
- Economies of Scale: Your unit cost to produce an additional model decreases exponentially for each additional model you build at a rate of 1/x^2 on average.
- Bottom Line:, You can benefit from dramatic differences in cost across skillsets. A systems administrator is much cheaper to hire than a PhD in math, and you don’t want to have your math PhD fiddling with linux server issues. For God’s sake!
A Virtual Conveyer-Belt
Building a predictive model requires several steps. It doesn’t matter if you’re building recommendation engine, a churn algorithm, a speech recognition system, etc., whatever the outcome, there are a series of steps that are necessary to follow, which will ensure your success.
At high-level, to build a predictive model, you need to create what’s called a training set. A training set is a matrix of data that contains features about that what you want to predict. For example, let’s say you want to predict whether or not someone will buy an item from your store. The training set for this type of prediction would invariably contain data about users that buy from your store, such as their age, gender, occupation, etc., and as well (and equally important) data about users that do not buy from your store (i.e., people that browse, but do not buy).
Since we are discussing the structural issues associated with model development, we won’t go into the details of how you create a model. In general, though, developing a model requires:
- Obtaining disparate sets of data from which to create a training set
- Merging the disparate data together to form one large matrix (aka ETL)
- Specifying the variable of interest (your dependent variable)
- Training a model
- Validating the model
- Finally Generating predictions
Without addressing the structural issues associated with model development, and simply hiring a data scientist to ingest data and build a predictive model, you run the risk of allocating tens or even hundreds of thousands of dollars to a project and not seeing the results you expect, either because the data are not predictive, the data scientist to ask the right questions, the models took too to long to run, etc. There are many reasons as to why you can have a failed project. Fundamentally, you need to think through an appropriate data pipeline: a mechanism by which you ingest new data, and run it through some processes to generate a set of useful predictions. You can think of this as a virtual conveyor belt, where you have different individuals manipulating the data as it moves along from beginning to end of the development process.
A Key Insight (Automated ETL)
If you can set up processes to automate these six steps you will have automate the process by which you produce a predictive model. The most difficult step in this process, and arguably the reason for which there is no all-serving AI system is the inability for companies to automate step 2 in this process: the ETL step. Automated ETL is the proverbial keystone in the Roman arch, linking the ingestion of raw, disparate data, to the development of an analytical model, which can generate predictions.
At Serial Metrics, our Orion platform has done just this. Principally, we have built a framework to automate the ETL process, thereby reducing the steps necessary to create a predictive model. We have the ability to build and deploy a model in under 15 minutes, thus reducing the length of time for which a person would obtain a pretty good model from weeks or months down to minutes.
By automating the process by which you can create a model, which can in turn generate inferences, and doing it all in 15 minutes, we have laid the foundation for a universal AI system that can produce answers to questions that are inductive in nature.
This is a substantial leap in data analytics.
More importantly, however, as continue to reduce the lead times associated with model development - from minutes down the seconds - we start to blur the lines between data analytics and search.
It is therefore our goal to revolutionize search and bring forward a new form of querying capability: inductive search, and unlock the potential of Artificial Intelligence for all.