/

What is GPT-3? Search and AI Perspectives

What is GPT-3? Search and AI Perspectives

Second in a Three-Part Blog Series on Conversational Search

In this blog, we address the question “what is GPT-3”, and how might it impact the search engine market and search applications. The blog is the second in a three-part series on ChatGPT, GPT-3.x and Large Language Models.  We hope the blogs can summarize these fast-changing developments in AI and offer pragmatic advice on the evolution of search interfaces to incorporate “conversational search.”

We will also touch on GPT-3.5 Turbo which was released during the writing of this blog, and adde a post-script on GPT-4, which was announced for limited preview immediately after this blog was published. Told you things move fast!  Most of the advice on GPT-3 mentioned here also applies to GPT-3.5 Turbo and GPT-4.

What is GPT-3? How is it different than ChatGPT?

GPT-3 (3rd Generative Pre-Trained Transformer) is an example of a Large Language Model (LLM) that uses advanced AI techniques called neural network machine learning (or deep learning). Developed by OpenAI and released in 2020, the model was trained using internet data and is able to generate large volumes of human-like text from a natural language text query as input.  ChatGPT, the wildly viral (and somewhat controversial) generative AI tool, is essentially a variant of GPT-3 specifically designed for chatbot applications.

ChatGPT resembles an easy-to-use end-user application (specifically a very good chatbot), which is what contributed to its wild popularity.  While you can also “play” with GPT-3, it is more like a developer tool that can be used for more general natural language processing tasks. GPT-3 is essentially the “next generation” of natural language processing tools.

What is GPT-3.5 Turbo?

While this blog was being written, OpenAI introduced the new ChatGPT and Whisper models. This “new” version of ChatGPT runs on an updated language model OpenAI calls GPT-3.5 Turbo. 

While it’s too early to make any definitive comparisons, it seems that GPT-3.5 Turbo is a tweaked “faster” version of GPT-3 accompanied by a 90 percent price drop (more on pricing later).  In the OpenAI discussion forum, we did find one example and specific use case where a developer preferred an older OpenAI model to the new GPT-3 Turbo.  So even with this new release, we believe most of the rest of the blog and our advice are valid for the GPT-3.x releases. You can play with GPT-3.5 Turbo (and all the other models) in the OpenAI Playground, after signing up for a free account.

GPT-3 Use Cases: Next Generation Natural Language Processing

Natural Language Processing (NLP) has been around for over 50 years.  Before the recent maturation of more sophisticated tools centered around AI and deep learning, the available tools used advanced statistical models to crudely “understand” text, or at least its inherent structure.  This included word frequency, word proximity, grammar models, and other linguistic techniques. 

The text processing use cases for these “classic” NLP tools coincides with the use cases for GPT-3.  You can sign up for an account and experience these various use cases in the OpenAI Playground, which we will use to illustrate the use cases.

Semantic search (i.e. natural language queries) – GPT-3 is capable of parsing and understanding a natural language query or prompt in multiple languages, which can be submitted via an API or a simple text interface.  Unlike good search bar interfaces, however, GPT-3 doesn’t automatically implement features such as autocorrect or autocomplete.

This is an example from the GPT-3 Playground asking a natural language query – in French – asking the age of former NBA player Patrick Ewing.  The model recognizes the French query and replies in French that Patrick Ewing, born on 5 August 1962, is 58 years old.

While answering convincingly in flawless French, the answer is WRONG.  Patrick Ewing is 60 years old (as of March 2023).  This illustrates that recency is one of the inherent flaws in the OpenAI models, most of which were trained on information up to 2021.

Copywriting – GPT-3 can generate high-quality text for copywriting tasks, including product descriptions, marketing materials and headlines.  It should be noted however that the out-of-the-box tool is limited by the Internet data on which it was initially trained.  If your products or services did not exist in that domain, you would have to further “train” the model on that content or provide more information in the prompt.

The example below from the GPT-3 playground is a Product Description example of a Tesla Model Y.  The information was probably gleaned from the Internet and the response is cut off for this free tool.

Here we ask GPT-3 to provide a product description for the non-existent 2030 Ford Explorer.  In the absence of real information, the tool actually “makes up” an answer from what it can gather of the current car.  However, if you provide the tool with more information in the prompt, it actually provides a reasonable results.

Summarization – GPT-3 can summarize long texts by extracting the most important information, presenting it in a clear and concise manner.  The example below is rather simple, but the results are virtually indistinguishable from a human.  GPT-3 could prove to be better at this task than “classic” NLP tools for large documents such as contracts, proposals, research papers, or other long-form documents.

Text Parsing – GPT-3 can parse text by analyzing and extracting specific information or patterns from large amounts of unstructured text data.  The example below illustrates this capability.  At first, the results seem unremarkable compared to what “classic” NLP tools might be able to produce.  However, these tools usually have developer interfaces that require programming to submit content and generate an answer.  OpenAI’s natural language interface to the same application is much more elegant and useful to non-programmers.

Classification – GPT-3 can be used for classification tasks by analyzing text and accurately assigning it to categories, such as sentiment analysis, spam detection, or topic classification.  This example is fairly rudimentary and could have been accomplished easily with “classic” NLP solutions.  However, GPT-3 was trained on how to automatically classify these animals via machine learning.  No ontology or taxonomy was programmatically developed and used (this raises another controversial question about machine learning and knowledge management.)

Translation – GPT-3 can translate text between languages, making content accessible to and from a global audience.  There are excellent “classic” translation tools available in the market today.  However, some can have problems with literal translation versus idiomatic translation.  One online translation tool translated “bucket list” into “listed de souhaits” which means “wish list”.  GPT-3’s answer means “list of things to do before you die”, which demonstrates a slightly more nuanced understanding of the term.

As tools like GPT-3 mature, they may replace “classic” NLP tools in the following applications:

  • Website search
  • Document search (e.g. patent search, academic research, legal discovery)
  • Customer service chatbots
  • Question Answering systems
  • Market intelligence (e.g. text summarization of website, social media, etc.)
  • Healthcare dictation and clinical documentation
  • Clinical trial matching
  • Computer aided coding
  • Clinical diagnosis
  • Credit scoring
  • Insurance claims management
  • Fraud detection
  • Resume evaluation / resume-to-job matching
  • Sentiment analysis

How does this all relate to search? 

Search applications are more than just a Google or Bing search bar.  All of the applications above deploy search engine and NLP technologies. And wherever NLP is currently deployed, deep learning-based AI tools like GPT-3.x have the potential to create a sea-change in how well they work.

GPT-3 Can Amaze – It Can Also Generate Rubbish

On the subject of technology maturity, one of the cautions with generative AI tools like GPT-3 is that, just as easily as they can amaze, they can generate rubbish.

The 2025 MINI Cooper does not exist.  If you search the Internet, you will find recent speculative articles about an electric version of the 2025 MINI Cooper.  GPT-3 seems to have fabricated an answer – but in a convincing manner. 

In the Chat interface of the OpenAI Playground, using the GPT-3.5 Turbo model, it seems that OpenAI is already trying to address the issue of “made up” content.  The Chat version of the answer explicitly states that it does not have access to “future information” and, therefore, the answer it gives is speculative.

What Next? Deploying, Extending, and Using GPT-3

Once you decide to commit to OpenAI as a vendor (even just for a pilot project), the next step isn’t as simple as “let’s start using GPT-3.x”. 

The next challenge includes answering questions like “What model should I use?” (yes, there’s more than one), or “Do I need to customize or extend the model?” to “What are the development and operational costs?” (for using GPT-3). 

Things are changing so fast that they may likely be out of date as soon as we hit “publish” on this blog, but we will try to elaborate on these and other questions below.

Which model do I use?  What are the operational costs?

OpenAI currently lists four language models to choose from for their AI tools.  According to OpenAI, when deciding which model to use, there is a tradeoff between how fast a model is, and how powerful a model is.  This also correlates to the usage cost for the model, which is utility like – based on the amount of content processed (measured in tokens).  These four models are part of the third generation (-003) of OpenAI models. GPT-3.5 turbo is a variation, but essentially has the same roots.

This is the OpenAI model pricing as of March 7, 2023.  OpenAI describes Ada as the fastest performing model, while Davinci is the most powerful (trained on the largest amount of data). For those unfamiliar with the concept of a token, it is essentially a “piece” of a word, or approximately 4 characters.  As a point of reference, the collected works of Shakespeare are about 900,000 words or 1.2M tokens.

OpenAI does provide some guidance on model selection, and a very technical model overview to help with the decision. Once you have picked a model, operational costs will depend significantly on your use case and how frequently you need to use the OpenAI services.  You may incur significant up-front costs for the models to ingest a library of documents, and only minimal operational costs later, depending on query volumes. 

The market is still evolving and we are already seeing dramatic drops in pricing as vendors try to establish market share and test what prices the market will bear for AI services.  On March 1, 2023, OpenAI announced a 90% reduction in the pricing for the ChatGPT API at $0.002 per 1k tokens (750 words). 

In the near term, we think customers will have a difficult time trying to navigate the model selection and cost forecasting process.  This is one area in which Pureinsights can offer advice.

What will it cost to customize your model? Who “owns” the improved model?

As we demonstrated with the product description example, the out-of-the-box GPT-3 model may not have information pertinent to your application.  OpenAI claims you can “customize” the model using the API with a single command and as few as 100 examples of training data.  But there are some practical questions that remain unanswered or vague that need to be considered.

The cost of developing models like GPT-3 is prohibitive for many companies, and a deep customization of the model could require considerable computational resources – and money.  Your cost for customizing and using OpenAI models will depend, again, on which model you choose, how much data you use to train your model, and how much you use the model.

So when you submit data to OpenAI through the API, how is your data used by OpenAPI?  Who “owns” the smarter model?  

Previously, OpenAPI could use the data you submit for its own purposes in the general language model.  After user feedback, OpenAI recently changed its data policy so that your data is only used to improve the model if you explicitly opt in. 

What happens if you don’t opt-in?

At this time, we believe that gives you exclusive use of the model you have fine-tuned with your own resources.  Similar to how extensions of Google BERT work, OpenAI likely runs an incremental model tuned on the training data you provided. 

It’s fascinating to think about the cloud architecture necessary to moderate the “master” AI models while taking into account what could be millions of paid fine-tuned models, and doing this at a manageable cost for the AI service provider.

How does OpenAI handle proprietary or sensitive data in custom models?

Data security is always a huge concern when we build sensitive search applications for our clients.  You certainly don’t want everyone to have access to salaries and HR data on a corporate intranet, and you don’t want to expose Personal Identifiable Information (PII) or proprietary intellectual property in public search applications.

So what are the ramifications of using tools like OpenAI and sensitive data?

Using the OpenAI API in an application or for fine-tuning the model would require the transmission and storage of that data outside of your control.  Until its clear how this data could be secure, and short of having your own expensive “private” version of an OpenAI language model, applications that involve sensitive or proprietary data may not be implementable.

Pureinsights also offers advice in this situation, including cost-effective ways to create your own private language models and infrastructure using open source options.

OpenAI Foundry and GPT-4

[POST-SCRIPT: Updated 14 March 2024]

So what comes after GPT-3 and GPT-3.5 Turbo? While our heads are still spinning on a swivel, information has leaked out already about OpenAI Foundry and GPT-4.

OpenAI Foundry

According to interpretations of a leaked product brief, Foundry will allow “cutting-edge customers” to run OpenAI models at scale with dedicated capacity.  Whether or not this means a “private” instance of a model that can address data security, we’ll have to wait and see.  You can read a summary the Foundry leak in this CMSWire article: Foundry: OpenAI’s New Developer Program Set to Launch (cmswire.com)

GPT-4 

Before we talk about GPT-4, ponder this: the entirety of English Wikipedia constitutes just 0.6 percent of GPT-3’s training data.  On the heels of its announcement about GPT-3.5 Turbo, OpenAI continued its onslaught on the market by announcing on 14 March, 2023 that it would be rolling out GPT-4 to API users

Prior to the launch, OpenAI CEO, Sam Altman played down expectations , noting in podcasts interviews that:

  • GPT-4 is not an Artificial General Intelligence (AGI), meaning a rival to human intelligence.
  • GPT-4 doesn’t have 100 trillion parameters (compared to GPT-3’s 175 billion parameters) – i.e. it will not know 500x more data than its predecessor.
Not much more can be confirmed at this date except for the fact that GPT-4 is multi-modal, meaning that it can it can accept images as well as text input (outputs are still only text.)  OpenAI states that early adopters include Microsoft for Bing search, Duolingo, Morgan Stanley, and Khan Academy. OpenAI also claims that the new model has been adjusted to not respond to inputs that generated potential harmful content in previous iterations.
GPT-4 doesn't answer negative prompts

AI Wars: Alternatives to OpenAI

We are entering the age of the “AI arms race” with OpenAI and Microsoft (and their new $10 billion investment) getting the bulk of the attention.  But there will be alternatives, and aside from a few notable startups, all eyes are on Google.

Google actually published initial research on Transformers (the T in GPT) in 2017, and soon after that, Google BERT was born in 2018.  Google surprised everyone by also making BERT open and free to use (for now).  In 2021, Google also hinted at a conversational search interface called Google MUM. Since then, however, Google has been fairly quiet, even as it has slowly improved semantic search and question answering capabilities in its Google internet search engine.

With the sensational launch of ChatGPT in November of 2022, however, Google had to respond by announcing Google Bard.  Google was definitely caught off guard when it rushed and botched the launch of Bard.  But Google has more tricks up its sleeve with DeepMind, a British AI company that it acquired in 2014.

Some other notable names to watch:

  • Stability AI
  • Anthropic
  • AI21 Labs
  • Cohere
  • Amazon Web Services
  • Baidu

Summary: Pragmatic AI and Search Application Perspectives

To wrap up, we want to first remind everyone of how AI services like GPT-3 and technologies like LLMs matter in search. 

  • In the past, search engines leveraged “classic” NLP technology to parse and process queries, and to crawl, index and deliver a list of results, ranked by relevance.
  • Search now doesn’t just live on internet or website search bars, it lives inside applications that people use every day – to decide what to watch, order take-out food, plan a vacation, find a new job, hire a new employee. Search is
  • LLMs, AI services, and even Knowledge Graphs will eventually revolutionize how we interact with these applications, and how they respond. We call this “conversational search.”

But this won’t happen overnight.  Traditional keyword-based search still has its place, and hybrid search also – a first step towards fully AI-driven search applications.

So, what should you do?

It’s not time to panic and rush into things, but neither is it the time to just ignore this technology as a “fad.”  We suggest that you take time to:

  • Learn more about the tools – the OpenAI playground is currently free (with registration) and you can experiment with a Chatbot or standard text input interface (future blog from us on the UI, perhaps).
  • Think about your specific use cases – what problem are you solving?
  • Consider a pilot project that will help you better learn about how much data will pass through pay-for-use APIs for AI services, and what this will mean in terms of development and operational costs.

We are still early in the AI services war.  To help you navigate the battlefield, we hope to host educational webinars and write blogs that cover topics such as Large Language Models (part 3 of this blog series), and how to get started with AI and search, including open source tools that are available today.

As always, please CONTACT US if you have any comments or questions, or to request a free consultation to discuss your ongoing search and AI projects.

Twitter
LinkedIn

Stay up to date with our latest insights!