TasksWithCode @TasksWithCode

@pfau Jakub's paper contributions authorswithcode.org/researchers/?a…

@sama Jakub Pachocki's paper contributions. authorswithcode.org/researchers/?a…

Jacob Pachocki, (OpenAI's new chief scientist's) paper contributions. authorswithcode.org/researchers/?a…

@DrJimFan There is a startup @Vapi_AI that addresses these challenges above and offers a working system with developers building on top of their open transparent platform (developers can choose the voice, AI agent etc.)

We have been running authorswithcode.com service for over a year now. The number of paper authors contributing to machine learining has grown to 433k+ and code authors to 170k+. More features spotlighting researchers to come.

This approach builds on the recent method of using 3D Gaussians to model scenes, enabling the quick and accurate creation of high-quality 3D images from photos or videos, even for large and complex scenes @xiaolonw's contributions authorswithcode.org/researchers/?a…

Xiaolong Wang @xiaolonw

3 years ago

3D Gaussian Splatting is great, but can it work without the pre-computed camera poses? Introducing: COLMAP-Free 3D Gaussian Splatting Our recent work shows not only it can, but 3D Gaussians make camera pose estimation easy (compared to NeRF) along with reconstruction. 👇🧵

4 66 303 77K 127

It has been 10 years since the release of the word2vec paper. Word2vec is arguably the first model (if we can even call it that, considering its simplicity—it consists of just two arrays of vectors) to demonstrate the power of distributed representation learning. The training process is surprisingly straightforward: it involves pulling each word's vector closer to those of other words in the same sentence, while simultaneously pushing it away from a few unrelated words not present in that sentence. By applying this process to a large corpus, we end up with vectors for semantically related words that roughly point in the same direction within a high-dimensional space. It's worth noting that all Large Language Models (LLMs) generate context-independent vectors, similar to those learned by word2vec, as part of their training. The models use these vectors to represent any input. Subsequently, they transform these into context-dependent vectors based on the sentence. Tomas Mikolov's contributions authorswithcode.org/researchers/?a…

Google AI @GoogleAI

3 years ago

Congratulations to Jeff Dean, Greg Corrado, & co-authors of the paper “Distributed Representations of Words and Phrases and their Compositionality”, for winning the #NeurIPS2023 Test of Time Award! This prize recognizes a highly impactful paper published at NeurIPS 10 years ago.

15 101 739 358K 56

This paper introduces a new objective function for self-supervised learning (SSL) that leverages the geometric properties of manifolds to which input classes are mapped. The focus is on maximizing manifold capacity, which essentially measures the number of object categories that can be linearly separated. This approach aims to enhance the efficiency and effectiveness of SSL in creating representations that are more aligned with the way biological systems process visual information. The paper presents a novel model for visual representation that is competitive with current SSL frameworks and potentially more closely mimics the functioning of the primate visual system. @tedyerxa

Teddy @tedyerxa

3 years ago

Excited to share our results on Efficient Coding of Natural Images using Maximum Manifold Capacity Representations, a collaboration with @KuangYilun @EeroSimoncelli and @s_y_chung to be presented at #NeurIPS2023 1/n

2 25 134 37K 67

@yangzhou301 Its our pleasure. Thank you for all your contributions

This work shows that Large Language Models (LLMs) aligned with fine-tuning or Reinforcement Learning from Human Feedback (RLHF) are still susceptible to prompt attacks that can reveal the training data memorized by the model. It is a known fact that base models tend to memorize their training data. This study reveals that aligning models does not conceal this vulnerability. This finding holds significance, particularly for businesses that do not publicly release models trained on sensitive data but provide access to these models through an interface. not-just-memorization.github.io/extracting-tra… First author Milad Nasr's contributions authorswithcode.org/researchers/?a… The prompt contains repeating words - GPt3.5 turbo model's output for such a prompt revealing training data (this was confirmed by matching model output with content downloaded from internet)

Andrej Karpathy continues to be the few biological agents whose world model is worth sampling from for nuggets of insights on LLMs youtu.be/zjkBMFhNj_g?si…

A thoughtful answer to the question - LLMs have been practically exposed to the entire corpus of human knowledge and still have not made connections that have led to a discovery, at least not yet. Eric's contributions to research authorswithcode.org/researchers/?a…

Eric J. Michaud @ericjmichaud_

3 years ago

@dwarkesh_sp tl;dr: Maybe learning simple things (basic knowledge, heuristics, etc) actually lowers the loss more than learning sophisticated things (algorithms associated with higher cognition that we really care about), and the sophisticated things will eventually be learned as scaling

14 52 823 151K 873

Jack mentions in his tweet that his walkthrough was inspired by the walkthroughs done by @NeelNanda5. Needless to say, if this trend picks up, it would be immensely beneficial to everyone. There is nothing quite like hearing an author discuss the details, nuances, and limitations of their own work

A paper walkthrough by @jackm2003 shares an intriguing finding: grokking has been observed in non-neural architectures. Originally noted in transformers working with algorithmic datasets, grokking is a phenomenon where the accuracy on the validation set increases, but significantly after the point at which training would typically be halted due to early stopping. In early stopping, training is stopped to prevent overfitting when the model's performance on a validation set ceases to improve. However, in the case of grokking, this dramatic increase in validation accuracy occurs much later, suggesting a different dynamic at play. The authors also hypothesize that the interplay between the error and the regularization term (which they refer to as 'complexity') in the loss function might be a crucial factor in triggering grokking. Jack Miller's contributions to research authorswithcode.org/researchers/?a…

Neel Nanda @NeelNanda5

3 years ago

I'm excited to see more paper walkthroughs exist!

4 2 63 12K 16

For anyone planning to efficiently fine tune a LLM, this article by @rasbt on LoRA could be helpful. He explains with clarity the trade-offs to consider such as choice of quantizing pretrained weights, choice of optimizers (Adam vs SGD), impact of schedulers etc. What is LoRA? For simplicity, imagine all the weights of a model as one large matrix W. The key insight of LoRA is that, unlike in pretraining, during fine-tuning we can approximate the gradient update matrix (which is the same shape as W) with two smaller matrix thereby achieving savings in both compute and memory. magazine.sebastianraschka.com/p/practical-ti… Sebastian's contributions authorswithcode.org/researchers/?a…

An approach for distributed training of LLM where pretraining is performed in parallel on multiple nodes and can have a large number of training steps (compared to typical Federated averaging). The model parameters are sent to a central server that finds the average change in model parameters. This is then used to update the global model using a different optimizer (Nesterov instead of AdamW used for local updates). Note also it is the model parameter and not the gradients that are used to update global model parameters. This is then dispatched to all nodes and the cycle is repeated. Arthur Douillard @Ar_Douillard 's contributions - (first author) arxiv.org/pdf/2311.08105…

This paper explores a combination of approaches to filter the input to a generator to improve RAG performance Zora's (@ZhiruoW) contribution to research. Code is also released (models used FLAN-T5 and LLAMA-2) It suggests a combination of methods to filter irrelevant content to a generator authorswithcode.org/researchers/?a…

Zora Wang @ZhiruoW

3 years ago

Everyone is using RAG, but most of the retrieved context is noisy! 🚨 Introducing FilCo: “Learning to Filter Context for Retrieval-Augmented Generation” TL;DR: Get rid of the irrelevant content using FilCo, and you'll get better outputs. Preprint: arxiv.org/abs/2311.08377

6 54 282 51K 216

This unforgiving retrospective analysis on the anniversary of Galactica release, by its first author, @rosstaylor90, sets a high bar for researchers - it underscores the importance of releasing work for feedback and continual improvement, even if it is a work in progress. Interestingly, the initial almost vitriolic criticism of Galactica for its creative responses is now largely accepted, viewed as either an opportunity or a limitation depending on the use case. More significantly, the limitations of base LLMs, like Galactica, are now universally acknowledged, with their tendency towards hallucination recognized as a fundamental reality. Galactica was, in essence, just a base model. Here is a link to Ross Taylor's contribution to open source ( incidentally this app is created in part from the data @paperswithcode publishes - a startup he cofounded ) authorswithcode.org/researchers/?a…

Ross Taylor @rosstaylor90

3 years ago

I am the first author of the Galactica paper and have been quiet about it for a year. Maybe I will write a blog post talking about what actually happened, but if you want the TLDR: 1. Galactica was a base model trained on scientific literature and modalities. 2. We approached

84 301 3K 961K 729