If speed and price are concerns, use the FOSS models available on the Hugging Face Hub: https://hf.co/models. Thousands of models, different sizes and tasks. Download locally and fine-tune, if necessary.
Normally I'd point out how these are a lot less capable than GPT3. But the article ends up fine tuning GPT Babbage, and multiple free models can outperform Babbage, so this is very solid advise.
How about fine tuning testing w/ Davinci and then scaling it down for the other models or HF once you've proven it works. I believe the openai docs propose this approach (minus HF of course)
That's up to you. Many don't want to open an account and pay in order to explore what's possible. There are LLMs available on the HF Hub, such as google/flan-t5-xl.
The medium article you posted does analysis on OpenAI's old embeddings. OpenAI has a single new embedding model [0] that replaces all the old models and it's also super cheap ($0.0004 / 1K tokens).
The first comment in that article has details on the new model. Not the original author but per their testing they said they paid $70 to encode 1M records. The embeddings are 1536 dimensions, which require a lot of vector storage. The HF hub has open models for 384 dimensions or 768 dimensions that work well for a lot of use cases.
Based on my understanding of the article, it sounds like OP has re-discovered/invented Knowledge Distillation[0]. Which is reasonably well research technique, and frequently applied to get models to run on compute constrained devices. Or taking very deep models, and using them to train very shallow, but low-latency models.
Glad to know that's what it's called! For avoidance of any doubt I'm not trying to claim I 'discovered' any of this - just wanted to share some useful learnings :D
When you independently discover commonly used techniques (often without even realizing it), it's usually a good sign you're working on and thinking about cool stuff! Eventually one of those ideas will be entirely novel (you've probably already had many of them), and that's how scientific progress is made.
I came here to say this. This entire article should have been 5 lines, had they known about Knowledge Distillation.
Anecdotally, a similar thing happened to me when I "independently invented" synthetic data generation for deep learning back in 2016... Turns out my brilliant idea was not that original after all
The more I learn, the more I realize how rarely I've thought of a "new idea." In fact, I'm not convinced such a thing even exists. An idea is generally a set of connections between pre-existing ideas, so perhaps a novel subset of those connections would qualify as a "new idea," but one could argue it's more of a rearrangement of existing concepts. (What was the first idea? The first thought? Does thought require language, and if so, is consciousness an emergent property of language?)
When I realized how rare it is to have a unique thought, I reframed my approach to evaluating my own ideas. Instead of worrying about novelty, I worry about relevance.
If I can't find a pre-existing similar idea, I assume I'm on the wrong track unless I can identify a compelling reason why I'm in a unique position to have an idea that nobody has ever thought or at least felt like communicating publicly.
If I can find such a pre-existing idea, then I know I'm on the right track, and often it validates a pattern I identified between a subset of connections in my previous knowledgebase, solidifying the pattern in my mind. Regardless of whether I can label it a "new idea," I can use it to work toward the next step in whatever line of thinking caused me to notice the pattern in the first place.
I have tried this method some time ago, it does work but it is not consistent. Sometimes you’d get good results identical to davinci as the article mentioned but you have the risk of showing random text not related to the prompt.
But I haven’t fine tuned on a large number of records, so that might solve the consistency problem.
This is great, thanks for sharing. I've been pretty reluctant to use davinci in production due to costs, but the other models seem significantly less capable. It seems like there's still a lot to learn about working with these models at scale.
Yes definitely give this a go, for us with our use case davinci is prohibitively expensive in production so we literally cannot use it given the number of requests we make - interestingly I saw some OpenAI documentation the other day at the YC event which basically said that with a large enough dataset (I recall >= 30k examples) all of the models (yes even ada) start to behave with similar performance, so bear that in mind!
Any other obvious low hanging fruit tricks for GPT3? Currently using Davinci003 for my use case bc price didn’t seem incredibly expensive. Was planning on fine tuning a model down the road, but didn’t get to it for MVP bc it seemed a prohibitively large amount of work.
Does OpenAI provide guarantees now around fine tuned models being loaded 24/7? Last time I fine tuned Curie models they took forever to load and that killed it for me as we’re building an interactive application.
If you're using it to the point you need it guaranteed up 24/7, you may be better off not treating it as a SaaS API and instead building up your own model. It seems somewhat likely OpenAI has some pretty insane changes going on with access and pay structure in the future, so I'd be cautious becoming reliant on them (at least for the next 3 months)
We've had some demand for this, if there's enough we'll definitely consider it when we hire more engineering resource, which is the current constraining factor
Definitely give them a go, we use fine-tuned ada a bunch for classification work for example; I personally think the smaller models are overlooked and don't get enough love - if OpenAI increased the context window of a model like babbage to 8k tokens I feel like that would be as much of a big deal as making a marginal improvement to davinci, purely because so many use cases rely on low-latency, many request models.
Is this just fine tuning a cheaper model like Curie on the outputs of a more expensive model like Davinci? I proposed this as a question/idea to an OpenAI employee on Twitter and he talked me out of it.
I think it depends on how the answers from davinci are used. If the answers are used verbatim to fine-tune a cheaper model, then that could be problematic. However if it is used to generate a fine-tuning dataset which is then corrected manually, I don't see the problem.
He didn't give a great explanation. Just kinda kept dismissing it as a bad idea and suggesting other paths.
I don't see why using the Davinci outputs verbatim would be a problem in certain situations. The goal is just to get a fine-tuned cheaper model (like Curie) closer to Davinci performance in some narrow problem domain. Of course it's never going to be as good or broad as Davinci with this approach, but the lower cost may outweigh that. Just surprised more people haven't tried and benchmarked this approach...but I'm no expert here so there is probably a good reason.
they're talking about current googlle's response which everyones been clearly suggesting is no longer actual response time to a search but a entirely truncated and curated deal.
Google is primed to be a rear-view mirror technology if it keeps reducing it's actual use.
Wow some of these comments seem harsh. The author isn't claiming to have invented these techniques and it's cool to see them being applied in the "real world" to save money.
Agreed. Fine-tuning GPT-3 is often called out as expensive and sometimes unnecessary given K-shot & embeddings. I'm glad they shared their success story even if it's not ground-breaking.
Thanks - we used to use a bunch of k-shot prompts (particularly with our previous idea before we pivoted when we got into YC), but with the davinci model we were sending ~3.5k tokens per invocation, which in the long term was costing far more than finetuning!
Checkout https://text-generator.io for embeddings for search, they are better because they are cheaper/faster/take into account linked images (actually embed text images and code in the same space)
Your training trick is neat/great innovation but also keep in mind it is likely overfitting meaning when you get a bit of new data you need to index and search that model isn't going to do well at all at embedding it, said differently that training works well if you can cover the types of data you're going to see in production really well at training time. If not there's a big accuracy drop for unseen data due to overfitting
I'm clearly not versed in AI but at least two of their examples are quite obviously wrong. Their Study Notes example asks for five key facts about Ancient Rome -- one of them is borderline incorrect and only applies to the Roman Empire, the other one is an overgeneralization, and they're two, not five. The "Receipts" example gets the total sum wrong.
the title in question is misleading and likely the negative feedback received you received is aimed at that.
The primary objective of ChatGPT is not to serve as a finished product answering queries, but rather to undergo fine-tuning with respect to given prompts. This model is capable of utilizing a variety of sub-models and executing queries as necessary.
Let's consider an example where a user inputs the sentence "What is the capital of France?"
This is how ChatGPT will answer you:
> 1 Preprocessing: The chat interface processes the user's input and recognizes that the user is asking for the capital of a country.
2 Generation of prompt: Based on this information, the chat interface generates the following prompt: "The capital of France is __."
3 Input to GPT-3: The prompt and the user's input are fed into the GPT-3 model.
4 Generation of response: The GPT-3 model then generates a response based on the prompt and input, for example: "The capital of France is Paris."
5 Postprocessing: The response generated by the GPT-3 model is postprocessed by the chat interface to ensure that it is grammatically correct and in the correct format. The final response returned to the user is: "The capital of France is Paris."
What i mean by that is that of course you will be able to use smaller more efficient models to get faster, quicker results and what ChatGPT really is, is the world training it what prompts look like. The "GPT Part" is only used after heavy preprocessing i would not be surprised if it already knew how to use its smaller models itself =)
This does not at all sound how all LLMs I know work. Eg no model the size of ChatGPT should need grammar correction. Having that run client side sounds even more weird. Where did you take this information from?
That is a horrendously bad source. ChatGPT doesn't know how ChatGPT works, because nothing about how ChatGPT works was published on the internet at the time when ChatGPT was trained. Frankly, there's still nothing on the internet about this.
> Of course im looking in from the outside with no validity to my claim but if i were to write a bot from scratch thats also how i would do it.
ChatGPT wasn't "written" as much as trained. Writing a bot and training one require completely different mentalities.
For those specifically interested in text embeddings, here is a good analysis: https://medium.com/@nils_reimers/openai-gpt-3-text-embedding...