As deep learning becomes more commonplace there are a few trends that are worth calling out. First, with a glass half empty view. Even though we are noticing large improvements in quality of results, we will start noticing saturation in some of these jumps for classical tasks in the next 1-2 years. By classical tasks, I am collectively referring to a large set of machine learning problems which were developed in the pre-deep learning era. Things like search, classification, entity tagging, etc. Most of these have already benefited from deep learning improvements. However, we would start hitting saturation due to limited headroom.
On the other hand, the trends for large, strong models would, very likely, continue going strong with two specific outcomes – unified models and stateful AI experiences.
Large models will start converging across modalities. Currently, we have English language pre-trained models that outperform universal models for English tasks. We are already observing some initial results that universal models outperform English specific models in English tasks. Similarly, multi-modals models are currently built to support multi-modal tasks. However, we will see convergence there as well. Ultimately, we should have a distilled, compact and ideally a single, pre-trained model which is then adapted for various applications.
On the application side, we should see stateful AI experiences. If we think about it, most of the experiences today are state-less. If one takes search as an example, every time you type a query, it goes through the same process over billions of documents of retrieval, ranking, localization, freshness etc. We humans are stateful. Most likely, our first search query and the second one has some relationship with one-another. Things like personalization or localization introduce some of notion of state, but it is bare bones. In future, as models become powerful, we will see new applications where users interact with machines in a highly interactive way. The emergent behavior that we observe in large language models at particular sizes show trends relating to this more and more. It will open up a new set of opportunities for applications and services to be built. Really exciting and early days in this space.