Reported by Machine Heart
It seems that paper search tools are starting to become competitive!
For those who search for papers every day, having a good search tool is simply delightful, and efficiency definitely increases.
But the reality is that either the search tool is ineffective, or the keywords you input are not working. In any case, the papers you are trying to find in your mind and the results that come up from the search can hardly be said to be related; they are simply worlds apart.
The website we are going to introduce below can help you solve the problems encountered in paper searching. The website is called arXiv Xplorer, specifically designed for semantic searches of papers on arXiv.According to the project author, the internal algorithm of the site uses OpenAI’s latest embedding model to conduct search queries for users, thus finding the most relevant papers.
arXiv Xplorer address: https://arxivxplorer.com/
The project author stated that OpenAI’s new embedding API left a deep impression on him, so he wanted to see how this embedding could be used in practice. Therefore, he spent a few days building this project, and so far, it works very well. In addition, he also used ChatGPT to write 80% of the UI, used Pinecone to store the vector database, and used Google Cloud Functions to embed queries and execute searches.
If you want to learn more about embedding models, you can check the website.
Embedding model: https://openai.com/blog/new-and-improved-embedding-model/
With arXiv Xplorer, you can find the papers you need, even if your description is very vague, or even just inputting “interesting ML papers” as a non-informative description, the engine can still help you complete the task. As shown below, after entering a few keywords, the website displays the query results.
In this process, compared to traditional search tools like Google or arXiv’s own search, you can discover interesting papers that you’ve never seen before; it seems that arXiv Xplorer performs even better.
You can also directly search for similar papers by pasting the arXiv URL. For example, the input in the image below is the address of the paper “A Generalist Agent”; the search results show (in red box) that the similarity of A Generalist Agent is 100%, and the other search results are displayed in order of score.
In addition, you can click the small triangle in the red box above, and then the interface changes to the one shown below, displaying the paper participants and abstract. Below there are also two functions:“More Like This” will show more similar papers;“View” will link to the paper’s homepage on arXiv.
Seeing this feature-rich website, netizens’ curiosity could not be contained, asking, “You used OpenAI’s embedding technology, but this technology is charged, how much did you pay for it?” The project author replied: “Embedding all papers in the CS category (about 500,000 papers) cost $40.”
Some netizens also raised a series of questions about the technology, such as:“Does this site embed all arXiv titles?” The project author replied:“He embedded all the titles and abstracts of the papers, initially manually calculating cosine similarity and sorting, but Pinecone made it super easy!”
Another netizen suggested: “This tool’s search function is excellent! If it could sort by publication date, that would be awesome.” Regarding this, the project author stated that he would continue to optimize and strive to achieve more perfect functions.
Reference link: https://twitter.com/tomtumiel/status/1611729847700570118?s=20&t=sW31zy64CvhMH81ntcxzXw
© THE END
For reprints, please contact this public account for authorization.
Submissions or inquiries: [email protected]