Ampere Computing

Ampere Computing

Ampere Computing LLC is an American fabless semiconductor company that designs ARM-based central processing units (CPUs) with high core counts for use in cloud computing and data center environments. Founded in 2017 by former Intel president Renée James, the company is headquartered in Santa Clara, California, and operates as an independent subsidiary of SoftBank Group since November 2025. == History == Ampere Computing was founded in fall 2017 by Renée James, ex-President of Intel, with funding from The Carlyle Group. James acquired a team from MACOM Technology Solutions (formerly AppliedMicro) in addition to several industry hires to start the company. Ampere Computing is an ARM architecture licensee and develops its own server microprocessors. Ampere fabricates its products at TSMC. In April 2019, Ampere announced its second major investment round, including investment from Arm Holdings and Oracle Corporation. In June 2019, Nvidia announced a partnership with Ampere to bring support for Compute Unified Device Architecture (CUDA). In November 2019, Nvidia announced a reference design platform for graphics processing unit (GPU)-accelerated ARM-based servers including Ampere. In the first half of 2020, Ampere announced Ampere Altra, an 80-core processor, and Ampere Altra Max, a 128-core processor, without the use of simultaneous multithreading. In March 2020, the company announced a partnership with Oracle. In September 2020, Oracle said it would launch bare-metal and virtual machine instances in early 2021 based on Ampere Altra. In November 2020, Ampere was named one of the top 10 hottest semiconductor startups by CRN. In May 2021, the company announced a partnership with Microsoft. In April 2022, Ampere said that it had filed a confidential prospectus with the U.S. Securities and Exchange Commission, signaling its intent to go public. In June 2022, HPE announced their Gen11 ProLiant system would use Ampere Altra and Ampere Altra Max Cloud Native Processors. In July 2022, Google announced T2A instances using Ampere Altra in the Google cloud and in August 2022 Microsoft announced their instances of Ampere running in Azure. On March 19, 2025, investment holding company SoftBank Group announced it will acquire Ampere Computing for $6.5 billion. The deal finalized in November 2025, with Ampere remaining as an independent subsidiary with its headquarters in Santa Clara, California. == Products == Ampere develops ARM-based computer processors and CPU cores under their Altra brands. These are used in databases, media encoding, web services, network acceleration, mobile gaming, AI inference processing, and other applications and programs that need to scale. On February 5, 2018, Ampere announced the eMAG 8180 featuring 32x Skylark cores fabricated on TSMC's 16FF+ process. It supports a turbo of up to 3.3 GHz with a TDP of 125 W, 8ch 64-bit DDR4, up to 1 TB DDR4 per socket, and 42x PCIe 3.0 Lanes. The Skylark cores were based on AppliedMicro's X-Gene 3. Packet offers servers with the eMAG 8180 and 128 GB DRAM, 480 GB SSD, and 2x 10 Gbit/s networking. On September 19, 2018, Ampere announced the availability of a version featuring 16x Skylark cores. === 2020 === On March 3, 2020, Ampere announced the Ampere Altra featuring 80 cores fabricated on TSMC's N7 process for hyperscale computing. It was the first server-grade processor to include 80 cores and the Q80-30 conserves power by running at 161 W in use. The cores are semi-custom Arm Neoverse N1 cores with Ampere modifications. It supports a frequency of up to 3.3 GHz with TDP of 250 W, 8ch 72-bit DDR4, up to 4 TB DDR4-3200 per socket, 128x PCIe 4.0 Lanes, 1 MB L2 per core and 32 MB SLC. Ampere also announced their roadmap with Ampere Altra Max (2021) in development and AmpereOne (2022) defined. === 2021 === The 128-core Altra Max was released in 2021 and targeted hyperscale cloud providers. It uses the same server socket and platforms as Ampere Altra, and both products have one thread per core. The Altra Max CPUs provide 128 Arm v8.2+ cores per chip and run up to 3.0 GHz. They also support eight channels of DDR4-3200 memory and 128 lanes of PCIe Gen4. Also in 2021, Oracle launched its Oracle Cloud Infrastructure (OCI) using Ampere Altra processors. === 2022 === In February 2022, Ampere and Rigetti Computing announced a strategic partnership to create hybrid quantum-classical computers. The companies will combine Ampere's Altra Max CPUs with Rigetti's Quantum Processing Units (QPU) in cloud-based High-Performance Computing (HPC) environments. In April, Microsoft previewed its Azure Virtual Machines running on the Ampere Altra. The VMs run scale-out workloads, web servers, application servers, open source databases, cloud native .NET applications, Java applications, gaming servers, media servers, and other processes. In May, Ampere announced the sampling of AmpereOne CPUs, 5 nanometer chips based on its in-house Ampere-developed core. AmpereOne will add support for DDR5 main memory and PCIe Gen5 peripherals. On June 28, 2022, HPE became first tier-one server provider to offer compute with optimized cloud-native silicon for service providers and enterprises embracing cloud-native development with new line of HPE ProLiant RL Gen11 servers, using Ampere® Altra® and Ampere® Altra® Max processors, delivering high performance and power efficiency. === 2023 === During April 2023, Ampere released the Altra developer's kit, an IoT Prototype Kit based on Ampere Altra, aimed at cloud developers, available in 32-core, 64-core, and 80-core formats. === 2024 === In May 2024, Ampere updated its AmpereOne roadmap to 256 cores and announced a joint effort with Qualcomm on CPUs and accelerators. == Customers == Ampere's customers include Microsoft Azure, Tencent Cloud, Oracle, ByteDance, Hewlett Packard Enterprise (HPE), Cloudflare, Equinix, Kingsoft Cloud, Meituan, Scaleway, UCloud, Foxconn Industrial Internet, Gigabyte, Inspur, Cruise, Hetzner, Project Ronin, Wiwynn and Google Cloud Platform Cruise uses an Ampere Altra variant for its autonomous driving unit. The CPU was selected because of its throughput and low power consumption. In 2021, Oracle, Microsoft, Tencent, and ByteDance committed to using Ampere's customized chips, first announced in May. In April 2022, Microsoft previewed Ampere Altra processors in its new Azure D-and E- series virtual machines. The Dpsv5 series is built for Linux enterprise application types, and the Epsv5 series is for memory-intensive Linux workloads. They provide up to 64 vCPUs, include VM sizes with 2GiB, 4GiB, and 8GiB per vCPU memory configurations, up to 40 Gbit/s networking, and high-performance local SSD storage. In 2022, Microsoft's Ampere Altra-based Azure servers became the first cloud solution provider server to be Arm SystemReady SR certified. The Azure VMs, powered by Altra processors, were also the first to be SystemReady Virtual Environment standard certified. SystemReady defines a set of firmware and hardware standards as a baseline for system development for software developers, original equipment vendors, and chipmakers.

Trebel (music app)

Trebel is an on-demand music download and discovery platform developed by M&M Media Inc. The company's business model aims to combat digital music piracy by giving users access to on-demand music at no cost while delivering fair compensation to artists and music rights holders. Trebel has a patent that allows it to market itself as the only international music service in which users can legally download music and listen to it offline for free. As of March 2023, Trebel has a catalog of 75 million songs from record labels such as Universal Music Group, Sony Music Entertainment, Warner Music Group and hundreds of independent labels. Trebel is based in Stamford, Connecticut. with additional locations in Mexico City, Jakarta, Bogota, Los Angeles and Miami. The app is available in the Apple App Store, Google Play Store, and Huawei AppGallery. == History == Trebel was founded in 2014 by Gary Mekikian, who was previously the co-founder of answerFriend, Inc., which commercialized web based question-answering technologies and merged with Electric Knowledge, forming InQuira. This company was eventually acquired by Oracle Corporation in 2011. His co-founders at Trebel include Stanford classmates Corey Jones and Luis Soto Durazo, as well as his daughters Grace and Juliette. Mekikian envisioned Trebel as an alternative to music piracy after a high school classmate of his daughters was targeted by cyberattackers while illegally downloading music online. Trebel was initially released in 2015 under the name Project Carmen to students at Ohio State, Santa Monica College, Cal State Fullerton, UCLA and Long Beach State. In its original incarnation, the service planned to target students at 3,000 universities and 30,000 high schools in the United States. A beta version of the app was introduced in 2016 with content from Universal Music Group and Warner Music Group. Trebel launched commercially in the United States and Mexico in 2018. In 2018, Mexican mass-media corporation Televisa also became a minority investor in Trebel. In May 2020, during the early months of the COVID-19 pandemic, Trebel was a digital broadcast partner for Se Agradece, a concert produced in Mexico by Televisa to honor frontline COVID workers that featured artists such as Rosalia, J Balvin, Maluma and Ricky Martin. In June 2021, Trebel reached 3 million monthly active users. In October 2021, Trebel signed a music licensing agreement with Merlin Network, the licensing agency for the independent music sector that controls an estimated 12% of the global digital recorded music market. In January 2022, Trebel announced a strategic alliance with MNC Corporation, an Indonesian media conglomerate, which also became a minority backer of the company. In March 2022, Trebel reported 5.2 million monthly active users as a result of growth in Latin America. In the same month,, Latin music star Maluma became a backer of Trebel and an advisor to Gary Mekikian, helping expand the service throughout Latin America. On April 18, 2022, Trebel launched in Indonesia during the finale of the music competition show X Factor Indonesia. Trebel also signed a deal that month with Soccer Media Solutions, a sports and entertainment marketing agency in Mexico, to sell Trebel’s premium advertising inventory through Soccer Media. In May 2022, Guillermo Ochoa, goalkeeper for the Mexican national soccer team, invested in Trebel and became an ambassador for the company. On October 2, 2022, Trebel collaborated with Musica Studios, one of the largest music companies in Indonesia, on the production of a music festival in Jakarta titled Trebel Music Fest. The event featured performances by top Indonesian music artists such as Noah, Nidji, and d'Masiv. In October 2022, Trebel launched in Colombia. The service reached 1.2 million monthly active users in Colombia six months after launching. In December 2022, Trebel collaborated with KFC in Indonesia on the release of a KFC digital music program using a product called Trebel Max. As part of the program, KFC customers who bought the Crazy Superstar Combo package at KFC received a subscription to Trebel Max for 30 days. Trebel announced the launch of Trebel AI in May 2023. Trebel AI uses ChatGPT-powered technology to generate playlists based on natural language queries from users. In Indonesia, the Trebel AI feature was announced during a broadcast of the show Indonesian Idol XII that took place on May 8, 2023. In July 2023, Trebel reached more than 13 million monthly active users. In November 2023, Trebel became a featured app on the Discord app directory. Discord users that add the Trebel bot to their servers have access to Trebel's on-demand music library and have the exclusive privilege of being DJ's during server sessions with up to 150 concurrent listeners. == Platform == === Features === Trebel has a patent that allows it to market itself as the only international music service in which users can legally download music and listen to it offline for free. As of March 2023, Trebel has a catalog of 75 million songs from record labels such as Universal Music Group, Sony Music Entertainment, Warner Music Group and hundreds of independent labels. Trebel offers unlimited music downloads that are playable in the app by registered users only. Offline listening is free to all users and not blocked by a paywall. Users can search for music based on song, artist, album, browsing friends' recent activity, and through other users' playlists. The app also offers free cloud storage for downloaded songs. Trebel also contains a feature called SongID, which identifies music being played nearby using a short sample, then offers it for download on the service. Podcasts are available for free listening on the service as well. === Business model === Trebel uses a business model that generates revenue from the sale of digital advertising as well as user interactions with branded experiences, and consumption of virtual goods within the app (akin to mobile games). The app also features a brand takeover feature called Trebel Max, which offers unlimited access in exchange for users engaging with experiences offered by specific brands. Trebel’s brand partners include Uber, KFC, Walmart, Coca-Cola, Amazon and P&G. === Content === In September 2022, Trebel secured an exclusive release of the song “Suara Hatiku” by Indonesian actress Amanda Monopo. As of March 2023, Trebel offers 75 million songs through licensing agreements with Universal Music Group, Sony Music Entertainment, Warner Music Group and global indie rights agency Merlin. == Awards == In 2023, Trebel won three Google Play awards including "Best App of 2023", "Best Everyday Essentials" and "Users' Choice".

Teacher forcing

Teacher forcing is an algorithm for training the weights of recurrent neural networks (RNNs). It involves feeding observed sequence values (i.e. ground-truth samples) back into the RNN after each step, thus forcing the RNN to stay close to the ground-truth sequence. The term "teacher forcing" can be motivated by comparing the RNN to a human student taking a multi-part exam where the answer to each part (for example a mathematical calculation) depends on the answer to the preceding part. In this analogy, rather than grading every answer in the end, with the risk that the student fails every single part even though they only made a mistake in the first one, a teacher records the score for each individual part and then tells the student the correct answer, to be used in the next part. The use of an external teacher signal is in contrast to real-time recurrent learning (RTRL). Teacher signals are known from oscillator networks. The promise is, that teacher forcing helps to reduce the training time. The term "teacher forcing" was introduced in 1989 by Ronald J. Williams and David Zipser, who reported that the technique was already being "frequently used in dynamical supervised learning tasks" around that time. A NeurIPS 2016 paper introduced the related method of "professor forcing".

Stochastic block model

The stochastic block model is a generative model for random graphs. This model tends to produce graphs containing communities, subsets of nodes characterized by being connected with one another with particular edge densities. For example, edges may be more common within communities than between communities. Its mathematical formulation was first introduced in 1983 in the field of social network analysis by Paul W. Holland et al. The stochastic block model is important in statistics, machine learning, and network science, where it serves as a useful benchmark for the task of recovering community structure in graph data. == Definition == The stochastic block model takes the following parameters: The number n {\displaystyle n} of vertices; a partition of the vertex set { 1 , … , n } {\displaystyle \{1,\ldots ,n\}} into disjoint subsets C 1 , … , C r {\displaystyle C_{1},\ldots ,C_{r}} , called communities; a symmetric r × r {\displaystyle r\times r} matrix P {\displaystyle P} of edge probabilities. The edge set is then sampled at random as follows: any two vertices u ∈ C i {\displaystyle u\in C_{i}} and v ∈ C j {\displaystyle v\in C_{j}} are connected by an edge with probability P i j {\displaystyle P_{ij}} . An example problem is: given a graph with n {\displaystyle n} vertices, where the edges are sampled as described, recover the groups C 1 , … , C r {\displaystyle C_{1},\ldots ,C_{r}} . == Special cases == If the probability matrix is a constant, in the sense that P i j = p {\displaystyle P_{ij}=p} for all i , j {\displaystyle i,j} , then the result is the Erdős–Rényi model G ( n , p ) {\displaystyle G(n,p)} . This case is degenerate—the partition into communities becomes irrelevant—but it illustrates a close relationship to the Erdős–Rényi model. The planted partition model is the special case that the values of the probability matrix P {\displaystyle P} are a constant p {\displaystyle p} on the diagonal and another constant q {\displaystyle q} off the diagonal. Thus two vertices within the same community share an edge with probability p {\displaystyle p} , while two vertices in different communities share an edge with probability q {\displaystyle q} . Sometimes it is this restricted model that is called the stochastic block model. The case where p > q {\displaystyle p>q} is called an assortative model, while the case p < q {\displaystyle p P j k {\displaystyle P_{ii}>P_{jk}} whenever j ≠ k {\displaystyle j\neq k} : all diagonal entries dominate all off-diagonal entries. A model is called weakly assortative if P i i > P i j {\displaystyle P_{ii}>P_{ij}} whenever i ≠ j {\displaystyle i\neq j} : each diagonal entry is only required to dominate the rest of its own row and column. Disassortative forms of this terminology exist, by reversing all inequalities. For some algorithms, recovery might be easier for block models with assortative or disassortative conditions of this form. == Typical statistical tasks == Much of the literature on algorithmic community detection addresses three statistical tasks: detection, partial recovery, and exact recovery. === Detection === The goal of detection algorithms is simply to determine, given a sampled graph, whether the graph has latent community structure. More precisely, a graph might be generated, with some known prior probability, from a known stochastic block model, and otherwise from a similar Erdos-Renyi model. The algorithmic task is to correctly identify which of these two underlying models generated the graph. === Partial recovery === In partial recovery, the goal is to approximately determine the latent partition into communities, in the sense of finding a partition that is correlated with the true partition significantly better than a random guess. === Exact recovery === In exact recovery, the goal is to recover the latent partition into communities exactly. The community sizes and probability matrix may be known or unknown. == Statistical lower bounds and threshold behavior == Stochastic block models exhibit a sharp threshold effect reminiscent of percolation thresholds. Suppose that we allow the size n {\displaystyle n} of the graph to grow, keeping the community sizes in fixed proportions. If the probability matrix remains fixed, tasks such as partial and exact recovery become feasible for all non-degenerate parameter settings. However, if we scale down the probability matrix at a suitable rate as n {\displaystyle n} increases, we observe a sharp phase transition: for certain settings of the parameters, it will become possible to achieve recovery with probability tending to 1, whereas on the opposite side of the parameter threshold, the probability of recovery tends to 0 no matter what algorithm is used. For partial recovery, the appropriate scaling is to take P i j = P ~ i j / n {\displaystyle P_{ij}={\tilde {P}}_{ij}/n} for fixed P ~ {\displaystyle {\tilde {P}}} , resulting in graphs of constant average degree. In the case of two equal-sized communities, in the assortative planted partition model with probability matrix P = ( p ~ / n q ~ / n q ~ / n p ~ / n ) , {\displaystyle P=\left({\begin{array}{cc}{\tilde {p}}/n&{\tilde {q}}/n\\{\tilde {q}}/n&{\tilde {p}}/n\end{array}}\right),} partial recovery is feasible with probability 1 − o ( 1 ) {\displaystyle 1-o(1)} whenever ( p ~ − q ~ ) 2 > 2 ( p ~ + q ~ ) {\displaystyle ({\tilde {p}}-{\tilde {q}})^{2}>2({\tilde {p}}+{\tilde {q}})} , whereas any estimator fails partial recovery with probability 1 − o ( 1 ) {\displaystyle 1-o(1)} whenever ( p ~ − q ~ ) 2 < 2 ( p ~ + q ~ ) {\displaystyle ({\tilde {p}}-{\tilde {q}})^{2}<2({\tilde {p}}+{\tilde {q}})} . For exact recovery, the appropriate scaling is to take P i j = P ~ i j log ⁡ n / n {\displaystyle P_{ij}={\tilde {P}}_{ij}\log n/n} , resulting in graphs of logarithmic average degree. Here a similar threshold exists: for the assortative planted partition model with r {\displaystyle r} equal-sized communities, the threshold lies at p ~ − q ~ = r {\displaystyle {\sqrt {\tilde {p}}}-{\sqrt {\tilde {q}}}={\sqrt {r}}} . In fact, the exact recovery threshold is known for the fully general stochastic block model. == Algorithms == In principle, exact recovery can be solved in its feasible range using maximum likelihood, but this amounts to solving a constrained or regularized cut problem such as minimum bisection that is typically NP-complete. Hence, no known efficient algorithms will correctly compute the maximum-likelihood estimate in the worst case. However, a wide variety of algorithms perform well in the average case, and many high-probability performance guarantees have been proven for algorithms in both the partial and exact recovery settings. Successful algorithms include spectral clustering of the vertices, semidefinite programming, forms of belief propagation, and community detection among others. == Variants == Several variants of the model exist. One minor tweak allocates vertices to communities randomly, according to a categorical distribution, rather than in a fixed partition. More significant variants include the degree-corrected stochastic block model, the hierarchical stochastic block model, the geometric block model, censored block model and the mixed-membership block model. == Topic models == Stochastic block model have been recognised to be a topic model on bipartite networks. In a network of documents and words, Stochastic block model can identify topics: group of words with a similar meaning. == Extensions to signed graphs == Signed graphs allow for both favorable and adverse relationships and serve as a common model choice for various data analysis applications, e.g., correlation clustering. The stochastic block model can be trivially extended to signed graphs by assigning both positive and negative edge weights or equivalently using a difference of adjacency matrices of two stochastic block models. == DARPA/MIT/AWS Graph Challenge: streaming stochastic block partition == GraphChallenge encourages community approaches to developing new solutions for analyzing graphs and sparse data derived from social media, sensor feeds, and scientific data to enable relationships between events to be discovered as they unfold in the field. Streaming stochastic block partition is one of the challenges since 2017. Spectral clustering has demonstrated outstanding performance compared to the original and even improved base algorithm, matching its quality of clusters while being multiple orders of magnitude faster.

Yooreeka

Yooreeka is a library for data mining, machine learning, soft computing, and mathematical analysis. The project started with the code of the book "Algorithms of the Intelligent Web". Although the term "Web" prevailed in the title, in essence, the algorithms are valuable in any software application. It covers all major algorithms and provides many examples. Yooreeka 2.x is licensed under the Apache License rather than the somewhat more restrictive LGPL (which was the license of v1.x). The library is written 100% in the Java language. == Algorithms == The following algorithms are covered: Clustering Hierarchical—Agglomerative (e.g. MST single link; ROCK) and Divisive Partitional (e.g. k-means) Classification Bayesian Decision trees Neural Networks Rule based (via Drools) Recommendations Collaborative filtering Content based Search PageRank DocRank Personalization

Instance-based learning

In machine learning, instance-based learning (sometimes called memory-based learning) is a family of learning algorithms that, instead of performing explicit generalization, compare new problem instances with instances seen in training, which have been stored in memory. Because computation is postponed until a new instance is observed, these algorithms are sometimes referred to as "lazy." It is called instance-based because it constructs hypotheses directly from the training instances themselves. This means that the hypothesis complexity can grow with the data: in the worst case, a hypothesis is a list of n training items and the computational complexity of classifying a single new instance is O(n). One advantage that instance-based learning has over other methods of machine learning is its ability to adapt its model to previously unseen data. Instance-based learners may simply store a new instance or throw an old instance away. Examples of instance-based learning algorithms are the k-nearest neighbors algorithm, kernel machines and RBF networks. These store (a subset of) their training set; when predicting a value/class for a new instance, they compute distances or similarities between this instance and the training instances to make a decision. To battle the memory complexity of storing all training instances, as well as the risk of overfitting to noise in the training set, instance reduction algorithms have been proposed.

Rectified linear unit

In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function is an activation function defined as the non-negative part of its argument, i.e., the ramp function: ReLU ⁡ ( x ) = x + = max ( 0 , x ) = x + | x | 2 = { x if x > 0 , 0 x ≤ 0 {\displaystyle \operatorname {ReLU} (x)=x^{+}=\max(0,x)={\frac {x+|x|}{2}}={\begin{cases}x&{\text{if }}x>0,\\0&x\leq 0\end{cases}}} where x {\displaystyle x} is the input to a neuron. This is analogous to half-wave rectification in electrical engineering. ReLU is one of the most popular activation functions for artificial neural networks, and finds application in computer vision and speech recognition using deep neural nets and computational neuroscience. == History == The ReLU was first used by Alston Householder in 1941 as a mathematical abstraction of biological neural networks. Kunihiko Fukushima in 1969 used ReLU in the context of visual feature extraction in hierarchical neural networks. In 1998, Gregory Woodbury demonstrated that the rectified linear function could account for a broad range of emergent properties in the visual cortex. His work showed that a single unified model could drive the joint development of refined retinotopic maps, ocular dominance columns, and orientation selectivity. By utilizing the rectifier's "cutoff" property, Woodbury achieved a close quantitative fit to biological data, matching the spatial periodicities and topographic refinement patterns observed in macaque and cat cortical maps. Furthermore, he extended this framework to adult plasticity, accurately replicating the spatial and temporal dynamics of lesion-induced cortical reorganization. This research established that the rectified linear response was a necessary mechanism for the stable self-organisation and maintenance of complex, multi-feature neural maps. In 2000, Hahnloser et al. argued that ReLU approximates the biological relationship between neural firing rates and input current, in addition to enabling recurrent neural network dynamics to stabilise under weaker criteria. Prior to 2010, most activation functions used were the logistic sigmoid (which is inspired by probability theory; see logistic regression) and its more numerically efficient counterpart, the hyperbolic tangent. Around 2010, the use of ReLU became common again. Jarrett et al. (2009) noted that rectification by either absolute or ReLU (which they called "positive part") was critical for object recognition in convolutional neural networks (CNNs), specifically because it allows average pooling without neighboring filter outputs cancelling each other out. They hypothesized that the use of sigmoid or tanh was responsible for poor performance in previous CNNs. Nair and Hinton (2010) made a theoretical argument that the softplus activation function should be used, in that the softplus function numerically approximates the sum of an exponential number of linear models that share parameters. They then proposed ReLU as a good approximation to it. Specifically, they began by considering a single binary neuron in a Boltzmann machine that takes x {\displaystyle x} as input, and produces 1 as output with probability σ ( x ) = 1 1 + e − x {\displaystyle \sigma (x)={\frac {1}{1+e^{-x}}}} . They then considered extending its range of output by making infinitely many copies of it X 1 , X 2 , X 3 , … {\displaystyle X_{1},X_{2},X_{3},\dots } , that all take the same input, offset by an amount 0.5 , 1.5 , 2.5 , … {\displaystyle 0.5,1.5,2.5,\dots } , then their outputs are added together as ∑ i = 1 ∞ X i {\displaystyle \sum _{i=1}^{\infty }X_{i}} . They then demonstrated that ∑ i = 1 ∞ X i {\displaystyle \sum _{i=1}^{\infty }X_{i}} is approximately equal to N ( log ⁡ ( 1 + e x ) , σ ( x ) ) {\displaystyle {\mathcal {N}}(\log(1+e^{x}),\sigma (x))} , which is also approximately equal to ReLU ⁡ ( N ( x , σ ( x ) ) ) {\displaystyle \operatorname {ReLU} ({\mathcal {N}}(x,\sigma (x)))} , where N {\displaystyle {\mathcal {N}}} stands for the gaussian distribution. They also argued for another reason for using ReLU: that it allows "intensity equivariance" in image recognition. That is, multiplying input image by a constant k {\displaystyle k} multiplies the output also. In contrast, this is false for other activation functions like sigmoid or tanh. They found that ReLU activation allowed good empirical performance in restricted Boltzmann machines. Glorot et al (2011) argued that ReLU has the following advantages over sigmoid or tanh: ReLU is more similar to biological neurons' responses in their main operating regime. ReLU avoids vanishing gradients. ReLU is cheaper to compute. ReLU creates sparse representation naturally, because many hidden units output exactly zero for a given input. They also found empirically that deep networks trained with ReLU can achieve strong performance without unsupervised pre-training, especially on large, purely supervised tasks. In 2017, the rectified linear function became a central component of the transformer architecture introduced in the Vaswani et al paper "Attention Is All You Need". Within every transformer layer, ReLU is utilized in the position-wise feed-forward networks (FFN), defined by Equation 2 of their paper: FFN ⁡ ( x ) = max ( 0 , x W 1 + b 1 ) W 2 + b 2 {\displaystyle \operatorname {FFN} (x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2}} This equation is foundational to the model's capacity; while the attention mechanism determines the relationships between tokens, the ReLU-based FFN performs the majority of the numerical computation and houses the bulk of the model's parameters. The efficiency and scalability of this rectified framework triggered a global technological revolution, enabling the development of Large Language Models that have had a profound economic impact. The industrial response to this architecture—including the massive expansion of AI-specific hardware and the birth of the generative AI sector—has positioned the Transformer as a cornerstone of 21st-century infrastructure. During the post 2017 period of rapid AI advancement, the rectified linear unit function has been key to achieving increased model performance and scaling due to the fact that it zeros out responses that are immaterial for a given stimuli, preventing them from accumulating in massive scale models. It is the complete silencing of the parts of the model found to be stimuli-irrelevant during learning that allows for scaling. As the stimuli-irrelevant proportion of the model becomes more massive, these highly numerous connections within the model would inevitably accumulate during scaling no matter how small each individual response is. Therefore, the rectified linear unit function, with its absolute zeroing property, enabled the scaling to hundred billion parameter models and beyond. Early Transformer scaling giants like GPT-3 (2020) and Falcon-180B (2023) relied on the rectified linear unit function explicitly, while successors such as GPT-4 (2023) and Llama 3 (2024) utilized smoother variants like GELU or SwiGLU. These variants were used to improve training stability while fundamentally preserving the rectified principle of zeroing low responses. At the centre of modern artificial intelligence ReLU and its variants maintain absolute zero response across the bulk of the model at any one time, while maintaining approximately linear reponses for stimuli-relevant connections enabling high performance on each specific cognitive task. This feature of activation sparsity has been critical for massive scaling and performance gains of AI models right up to the present day. == Advantages == Advantages of ReLU include: Sparse activation: for example, in a randomly initialized network, only about 50% of hidden units are activated (i.e. have a non-zero output). Better gradient propagation: fewer vanishing gradient problems compared to sigmoidal activation functions that saturate in both directions. Efficiency: only requires comparison and addition. Scale-invariant (homogeneous, or "intensity equivariance"): max ( 0 , a x ) = a max ( 0 , x ) for a ≥ 0 {\displaystyle \max(0,ax)=a\max(0,x){\text{ for }}a\geq 0} . == Potential problems == Possible downsides can include: Non-differentiability at zero (however, it is differentiable anywhere else, and the value of the derivative at zero can be chosen to be 0 or 1 arbitrarily). Not zero-centered: ReLU outputs are always non-negative. This can make it harder for the network to learn during backpropagation, because gradient updates tend to push weights in one direction (positive or negative). Batch normalization can help address this. ReLU is unbounded. Redundancy of the parametrization: Because ReLU is scale-invariant, the network computes the exact same function by scaling the weights and biases in front of a ReLU activation by k {\displaystyle k} , and the weights after by 1 / k {\displaystyle 1/k} . Dying ReLU: ReLU neurons can sometimes be pushed into states