高雄醫學大學圖書館 |

Odors as ''Natural Language'': Sparse Neural Networks in Mammalian Olfactory Systems and Large Language Models /

Record Type:	Language materials, printed : Monograph/item
Title/Author:	Odors as ''Natural Language'': Sparse Neural Networks in Mammalian Olfactory Systems and Large Language Models // Bo Liu.
Author:	Liu, Bo,
Description:	1 electronic resource (155 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 86-12, Section: B.
基督教聖經之智慧書導讀 :	The studies of physics, neuroscience, and artificial intelligence (AI) have a long intertwined history. Particularly, sparse connectivity is a common feature of the brain neural networks and a key focus in AI for efficient computation; notably, pruning trained networks for sparse connectivity has a long history, partially inspired by neuroscience. This thesis explores sparse neural networks through two linked research topics: one focused on the brain (bilateral alignment in olfactory systems), and the other on AI (pruning large language models for on-device AI assistants).For the first topic, inspired by mammalian dual nostrils creating two cortical neural representations of odors, in Chapter 1, we studied how to construct the inter-hemispheric projections aligning these representations. We hypothesized that this construction originates from online learning since mammals are constantly breathing. With a local Hebbian rule, we found that sparse interhemispheric projections suffice for bilateral alignment and discovered an inverse scaling that more cortical neurons allow sparser projections. Also, the local Hebbian rule was found to approximate the global stochastic gradient descent (SGD) rule since their update vectors align, suggesting that biologically plausible learning rules can approximate global learning rules if they contain the gradient information of the latter.The next chapter extends Chapter 1 from four perspectives: an analysis of the update vector alignment between Hebbian and SGD rules and how it depends on the network parameters; a simple theory that recurrent connections in olfactory cortex may improve the bilateral alignment, inspired by the Hopfield Networks (associative memory) 1 and similar to the design of Google Titans model that combines recurrent neural networks with Transformers; the dynamical properties of Hebbian learning; and finally, the geometric landscape of Hebbian learning.A similar inverse scaling has been discovered in the Transformer attention matrices used in large language models (LLMs), which motivated the second topic. Concretely, we pruned pretrained Meta Llama-2 and Llama-3 models to obtain models with fewer parameters and develop on-device AI assistants, explored their sparsity limits, and compared their performance at the limits. We found that more than 50% of the parameters in both models could be pruned, and Llama-3 produced fewer factual errors at the sparsity limit but required more parameters presumably due to its training settings and dataset.In summary, by studying sparsity in both biological and artificial neural networks, this thesis may provide valuable insights into the general bilateral alignment problem in neuroscience (across different modalities and brain regions such as the frontal cortex responsible for short-term and motor response and the medial entorhinal cortex for spatial memory), open the door to interesting theoretical questions, and inspire more efficient AI algorithms or applications.
Contained By:	Dissertations Abstracts International86-12B.
Subject:	Neurosciences. -
Online resource:	http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=32002155
ISBN:	9798280717947

Odors as ''Natural Language'': Sparse Neural Networks in Mammalian Olfactory Systems and Large Language Models /
Liu, Bo,

Odors as ''Natural Language'': Sparse Neural Networks in Mammalian Olfactory Systems and Large Language Models /Bo Liu. - 1 electronic resource (155 pages)

Source: Dissertations Abstracts International, Volume: 86-12, Section: B.

The studies of physics, neuroscience, and artificial intelligence (AI) have a long intertwined history. Particularly, sparse connectivity is a common feature of the brain neural networks and a key focus in AI for efficient computation; notably, pruning trained networks for sparse connectivity has a long history, partially inspired by neuroscience. This thesis explores sparse neural networks through two linked research topics: one focused on the brain (bilateral alignment in olfactory systems), and the other on AI (pruning large language models for on-device AI assistants).For the first topic, inspired by mammalian dual nostrils creating two cortical neural representations of odors, in Chapter 1, we studied how to construct the inter-hemispheric projections aligning these representations. We hypothesized that this construction originates from online learning since mammals are constantly breathing. With a local Hebbian rule, we found that sparse interhemispheric projections suffice for bilateral alignment and discovered an inverse scaling that more cortical neurons allow sparser projections. Also, the local Hebbian rule was found to approximate the global stochastic gradient descent (SGD) rule since their update vectors align, suggesting that biologically plausible learning rules can approximate global learning rules if they contain the gradient information of the latter.The next chapter extends Chapter 1 from four perspectives: an analysis of the update vector alignment between Hebbian and SGD rules and how it depends on the network parameters; a simple theory that recurrent connections in olfactory cortex may improve the bilateral alignment, inspired by the Hopfield Networks (associative memory) 1 and similar to the design of Google Titans model that combines recurrent neural networks with Transformers; the dynamical properties of Hebbian learning; and finally, the geometric landscape of Hebbian learning.A similar inverse scaling has been discovered in the Transformer attention matrices used in large language models (LLMs), which motivated the second topic. Concretely, we pruned pretrained Meta Llama-2 and Llama-3 models to obtain models with fewer parameters and develop on-device AI assistants, explored their sparsity limits, and compared their performance at the limits. We found that more than 50% of the parameters in both models could be pruned, and Llama-3 produced fewer factual errors at the sparsity limit but required more parameters presumably due to its training settings and dataset.In summary, by studying sparsity in both biological and artificial neural networks, this thesis may provide valuable insights into the general bilateral alignment problem in neuroscience (across different modalities and brain regions such as the frontal cortex responsible for short-term and motor response and the medial entorhinal cortex for spatial memory), open the door to interesting theoretical questions, and inspire more efficient AI algorithms or applications.

English

ISBN: 9798280717947Subjects--Topical Terms:

240374
Neurosciences.
Subjects--Index Terms:

Neural networks

Odors as ''Natural Language'': Sparse Neural Networks in Mammalian Olfactory Systems and Large Language Models /
LDR:04465nam a22004333i 4500 001 391493
005 20251124054806.5
006 m o d
007 cr|nu||||||||
008 251208s2025 miu||||||m |||||||eng d
020 $a 9798280717947
035 $a (MiAaPQD)AAI32002155
035 $a AAI32002155
040 $a MiAaPQD $b eng $c MiAaPQD $e rda
100 1 $a Liu, Bo, $e author. $0 (orcid)0000-0002-2819-608X $3 524055
245 1 0 $a Odors as ''Natural Language'': Sparse Neural Networks in Mammalian Olfactory Systems and Large Language Models / $c Bo Liu.
264 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2025
300 $a 1 electronic resource (155 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-12, Section: B.
500 $a Advisors: Murthy, Venkatesh N.
502 $b Ph.D. $c Harvard University $d 2025.
520 $a The studies of physics, neuroscience, and artificial intelligence (AI) have a long intertwined history. Particularly, sparse connectivity is a common feature of the brain neural networks and a key focus in AI for efficient computation; notably, pruning trained networks for sparse connectivity has a long history, partially inspired by neuroscience. This thesis explores sparse neural networks through two linked research topics: one focused on the brain (bilateral alignment in olfactory systems), and the other on AI (pruning large language models for on-device AI assistants).For the first topic, inspired by mammalian dual nostrils creating two cortical neural representations of odors, in Chapter 1, we studied how to construct the inter-hemispheric projections aligning these representations. We hypothesized that this construction originates from online learning since mammals are constantly breathing. With a local Hebbian rule, we found that sparse interhemispheric projections suffice for bilateral alignment and discovered an inverse scaling that more cortical neurons allow sparser projections. Also, the local Hebbian rule was found to approximate the global stochastic gradient descent (SGD) rule since their update vectors align, suggesting that biologically plausible learning rules can approximate global learning rules if they contain the gradient information of the latter.The next chapter extends Chapter 1 from four perspectives: an analysis of the update vector alignment between Hebbian and SGD rules and how it depends on the network parameters; a simple theory that recurrent connections in olfactory cortex may improve the bilateral alignment, inspired by the Hopfield Networks (associative memory) 1 and similar to the design of Google Titans model that combines recurrent neural networks with Transformers; the dynamical properties of Hebbian learning; and finally, the geometric landscape of Hebbian learning.A similar inverse scaling has been discovered in the Transformer attention matrices used in large language models (LLMs), which motivated the second topic. Concretely, we pruned pretrained Meta Llama-2 and Llama-3 models to obtain models with fewer parameters and develop on-device AI assistants, explored their sparsity limits, and compared their performance at the limits. We found that more than 50% of the parameters in both models could be pruned, and Llama-3 produced fewer factual errors at the sparsity limit but required more parameters presumably due to its training settings and dataset.In summary, by studying sparsity in both biological and artificial neural networks, this thesis may provide valuable insights into the general bilateral alignment problem in neuroscience (across different modalities and brain regions such as the frontal cortex responsible for short-term and motor response and the medial entorhinal cortex for spatial memory), open the door to interesting theoretical questions, and inspire more efficient AI algorithms or applications.
546 $a English
590 $a School code: 0084
650 4 $2 96060 $a Neurosciences. $3 240374
650 4 $2 96060 $a Biophysics. $3 264265
653 $a Neural networks
653 $a Stochastic gradient descent
653 $a Hebbian learning
653 $a Artificial neural networks
653 $a Spatial memory
690 $a 0786
690 $a 0317
690 $a 0800
710 2 $a Harvard University. $b Biology, Molecular and Cellular. $e degree granting institution. $3 524056
720 1 $a Murthy, Venkatesh N. $e degree supervisor.
773 0 $t Dissertations Abstracts International $g 86-12B.
790 $a 0084
791 $a Ph.D.
792 $a 2025
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=32002155