overcommit_memory

Linux:/proc/sys/vm/目录各文件详解 proc/sys/vm/ 目录是 Linux 系统中的一个特殊目录,它包含了与虚拟内存子系统相关的系统内核参数。这些参数可以用来配置系统的虚拟内存管理策略,包括内存分配、页面置换、内存压缩、NUMA 等方面的设置。用户和系统管理员可以通过修改这些内核参数来调整系统的内存管理行为,以优化系统性能或满足特定的应用需求。 阅读详情

目录

  • overcommit_memory
  • Overcommit Accounting
  • CommitLimit and Committed_AS
  • overcommit_kbytes or overcommit_ratio


overcommit_memory: Documentation for /proc/sys/vm/ — The Linux Kernel documentation

overcommit_memory这个vm参数用于控制malloc、mmap这些用于在进程虚拟地址空间申请虚拟内存的系统调用,例如一次性申请大量的虚拟地址是否允许成功,就取决于这个参数的配置。

参数位置:/proc/sys/vm/overcommit_memory(运行时配置内核参数,那你绝对要关注sysctl指令和/proc/sys目录)

overcommit_memory

This value contains a flag that enables memory overcommitment.

When this flag is 0, the kernel attempts to estimate the amount of free memory left when userspace requests more memory.

When this flag is 1, the kernel pretends there is always enough memory until it actually runs out.

When this flag is 2, the kernel uses a “never overcommit” policy that attempts to prevent any overcommit of memory. Note that user_reserve_kbytes affects this policy.

This feature can be very useful because there are a lot of programs that malloc() huge amounts of memory “just-in-case” and don’t use much of it.

The default value is 0.

See Overcommit Accounting and mm/util.c::__vm_enough_memory() for more information.

更多详情,需要去转看Overcommit Accounting。

补充:默认情况下,当malloc一次申请的虚拟内存大于128K则会转成mmap,此时不会在堆内存中通过brk指针申请虚拟内存,而是通过mmap在内存映射区划分内存。

1.先看malloc API文档中的MMAP_THRESHOLD:

Normally, malloc() allocates memory from the heap, and adjusts
       the size of the heap as required, using sbrk(2).  When allocating
       blocks of memory larger than MMAP_THRESHOLD bytes, the glibc
       malloc() implementation allocates the memory as a private
       anonymous mapping using mmap(2).  MMAP_THRESHOLD is 128 kB by
       default, but is adjustable using mallopt(3).  Prior to Linux 4.7
       allocations performed using mmap(2) were unaffected by the
       RLIMIT_DATA resource limit; since Linux 4.7, this limit is also
       enforced for allocations performed using mmap(2).

2.转到mallopt API文档中查看MALLOC_MMAP_THRESHOLD_:

MALLOC_MMAP_THRESHOLD_
              Controls the same parameter as mallopt() M_MMAP_THRESHOLD.

3.再转看mallopt API文档中的M_MMAP_THRESHOLD:

M_MMAP_THRESHOLD
              For allocations greater than or equal to the limit
              specified (in bytes) by M_MMAP_THRESHOLD that can't be
              satisfied from the free list, the memory-allocation
              functions employ mmap(2) instead of increasing the program
              break using sbrk(2).

              Allocating memory using mmap(2) has the significant
              advantage that the allocated memory blocks can always be
              independently released back to the system.  (By contrast,
              the heap can be trimmed only if memory is freed at the top
              end.)  On the other hand, there are some disadvantages to
              the use of mmap(2): deallocated space is not placed on the
              free list for reuse by later allocations; memory may be
              wasted because mmap(2) allocations must be page-aligned;
              and the kernel must perform the expensive task of zeroing
              out memory allocated via mmap(2).  Balancing these factors
              leads to a default setting of 128*1024 for the
              M_MMAP_THRESHOLD parameter.

              The lower limit for this parameter is 0.  The upper limit
              is DEFAULT_MMAP_THRESHOLD_MAX: 512*1024 on 32-bit systems
              or 4*1024*1024*sizeof(long) on 64-bit systems.

              Note: Nowadays, glibc uses a dynamic mmap threshold by
              default.  The initial value of the threshold is 128*1024,
              but when blocks larger than the current threshold and less
              than or equal to DEFAULT_MMAP_THRESHOLD_MAX are freed, the
              threshold is adjusted upward to the size of the freed
              block.  When dynamic mmap thresholding is in effect, the
              threshold for trimming the heap is also dynamically
              adjusted to be twice the dynamic mmap threshold.  Dynamic
              adjustment of the mmap threshold is disabled if any of the
              M_TRIM_THRESHOLD, M_TOP_PAD, M_MMAP_THRESHOLD, or
              M_MMAP_MAX parameters is set.


Overcommit Accounting

The Linux kernel supports the following overcommit handling modes

0

Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.

1

Always overcommit. Appropriate for some scientific applications. Classic example is code using sparse arrays and just relying on the virtual memory consisting almost entirely of zero pages.

2

Don’t overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.

Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.

The overcommit policy is set via the sysctl vm.overcommit_memory.

The overcommit amount can be set via vm.overcommit_ratio (percentage) or vm.overcommit_kbytes (absolute value).

The current overcommit limit and amount committed are viewable in /proc/meminfo as CommitLimit and Committed_AS respectively.

最后一句提到了/proc/meminfo中的CommitLimit and Committed_AS,转去看看。


CommitLimit and Committed_AS

地址:proc(5) - Linux manual page (man7.org)

CommitLimit %lu (since Linux 2.6.10)
                     This is the total amount of memory currently
                     available to be allocated on the system, expressed
                     in kilobytes.  This limit is adhered to only if
                     strict overcommit accounting is enabled (mode 2 in
                     /proc/sys/vm/overcommit_memory).  The limit is
                     calculated according to the formula described under
                     /proc/sys/vm/overcommit_memory.  For further
                     details, see the kernel source file
                     Documentation/vm/overcommit-accounting.rst.
Committed_AS %lu
                     The amount of memory presently allocated on the
                     system.  The committed memory is a sum of all of
                     the memory which has been allocated by processes,
                     even if it has not been "used" by them as of yet.
                     A process which allocates 1 GB of memory (using
                     malloc(3) or similar), but touches only 300 MB of
                     that memory will show up as using only 300 MB of
                     memory even if it has the address space allocated
                     for the entire 1 GB.

                     This 1 GB is memory which has been "committed" to
                     by the VM and can be used at any time by the
                     allocating application.  With strict overcommit
                     enabled on the system (mode 2 in
                     /proc/sys/vm/overcommit_memory), allocations which
                     would exceed the CommitLimit will not be permitted.
                     This is useful if one needs to guarantee that
                     processes will not fail due to lack of memory once
                     that memory has been successfully allocated.

mode 2才生效,即overcommit_memory=2时才生效,转回去详细看看mode 2。

mode 2即不允许over commit,在进程的虚拟地址空间允许commit的范围不能超过swap+物理内存的配置量(默认50%)。

swap这个好理解,free、/proc/swaps、swapon -s都能查看。

物理内存的的配置量,这个在哪里?overcommit_kbytes或者overcommit_ratio,都在/proc/sys/vm路径下。


overcommit_kbytes or overcommit_ratio

路径:/proc/sys/vm

overcommit_kbytes

地址:Documentation for /proc/sys/vm/ — The Linux Kernel documentation

When overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this amount of physical RAM. See below.

Note: overcommit_kbytes is the counterpart of overcommit_ratio. Only one of them may be specified at a time. Setting one disables the other (which then appears as 0 when read).

overcommit_ratio

地址:Documentation for /proc/sys/vm/ — The Linux Kernel documentation

When overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this percentage of physical RAM. See above.

当overcommit_memory=2(即不允许over commit)时,overcommit_kbytes或overcommit_ratio两个选项用于配置允许commit的虚拟地址大小:

  • 即允许commit的虚拟地址不能超过swap+overcommit_kbytes配置量
  • 或允许commit的虚拟地址不能超过swap+overcommit_ratio* physical RAM

看上文红色加粗部分,overcommit_kbytes和overcommit_ratio是二选一,你配置其中一个的时候另一个自动被禁用。当cat /proc/sys/vm/overcommit_kbytes或/proc/sys/vm/overcommit_ratio值为0的时候就表示该选项是被禁用的那个。

其实这个点是作为Linux OOM的延伸,追下来你会发现oom其实是allocate page中的一个子流程,也就是说oom是在分配物理内存的过程中才会触发。

至于malloc、mmap这些申请虚拟内存是不会导致oom的(要等到实际用到物理内存的时候才会通过缺页异常来分配物理内存,这时候才会导致oom),你能不能在进程的虚拟地址空间overcommit,取决于配置项overcommit_memory、swap、overcommit_kbytes、overcommit_ratio*physical RAM等配置项。

Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.

5-Linux 内存溢出oom Linux内核为了提高内存的使用效率采用过度分配内存(over-commit memory)的办法,造成物理内存过度紧张进而触发OOM机制来杀死一些进程回收内存。该机制会监控那些占用内存过大,尤其是瞬间很快消耗大量内存的进程,为了防止内存耗尽会把该进程杀掉。 阅读详情

相关推荐

Linux 内核参数:overcommit 相关

最终会调到overcommit 中,回头来简单看下该函数,大致分为: overcommit_memory 为1 (always) 直接pass,允许overcommit; overcommit_memory 为0 (guess,也是默认值),需要确认申请的pages 是否过大,不要太过分就行; 进入overcommit_memory 为2 (never)的流程; 进入vm_commit_limit,确认最终overcommit 允许的memory,最终结果存于allowed中; allowed 减去adm

私房菜 2156

内存分配策略:overcommit_memory

overcommit_memory overcommit_memory 是什么? overcommit_memory是内存分配策略,内核对内存分配的一种策略。 overcommit_memory 有什么作用? overcommit_memory取值又三种分别为0, 1, 2 overcommit_memory=0, 表示内核将检查是否有足够的可用内存供应用进程使用;如果有足够 的可用内存,内存申请允许;否则,内存申请失败,并把错误返回给应用进程。 overcommit_memory=1, 表示内核允

xsxb_yl的博客 1万+

PythonOCC基础使用:建模——倒角倒圆(二维+三维)

占坑

weixin_42755384的博客 4639

GNU/Linux - /proc/sys/vm/overcommit_memory

proc/sys/vm/overcommit_memory "是一个 Linux 内核参数,用于控制系统处理内存分配请求的方式。该参数对决定进程请求内存时内核的行为至关重要。vm.overcommit_kbytes: overcommit_ratio 的替代参数,指定以千字节为单位的超量分配限制。3. 模式 2:提供最高级别的内存分配保证,但可能会限制某些应用程序的性能。- 在这种模式下,无论当前内存使用情况如何,内核总是允许内存分配请求。- 内核通常允许轻微的超量分配,但拒绝明显的地址空间超量分配。

guoqx的专栏 1442

STM32 FOC电机控制:手把手教你用CubeMX配置TIM1中心对齐PWM(含代码避坑)

本文详细解析了STM32 FOC电机控制中TIM1中心对齐PWM的配置方法,涵盖CubeMX图形化设置、HAL库代码调用及常见问题排查。特别针对L6230Q驱动芯片的使能逻辑和死区时间配置提供实用指南,帮助开发者高效实现无刷电机的高性能控制。

weixin_27027459的博客 446

GOM Correlate拉伸试验后处理操作

本文描述了使用GOM Software 2022对拉伸试样进行后处理的操作,包括设置检测点、引伸计、输出应变数据到表格、输出报告为视频等

qq_45008924的博客 4669

Memory Overcommit

Memory Overcommit 1.什么是OverCommit Memory? Memory Overcommit:操作系统承诺给进程内存大小超过了实际可用的内存。一个保守的操作系统不会允许memory overcommit,有多少就分配多少,再申请就没有了,这其实有些浪费内存,因为进程实际使用到的内存往往比申请的内存要少,比如某个进程malloc()了200MB内存,但实际上只用到了100MB,按照UNIX/Linux的算法,物理内存页的分配发生在使用的瞬间,而不是在申请的瞬间,也就是说未用到的100

wll1228的博客 921

NVIDIA DGX Spark实战指南:从开箱到AI模型高效部署

本文详细介绍了NVIDIA DGX Spark的开箱体验、硬件架构及AI模型高效部署实战指南。从1 PetaFLOP算力的迷你设备到Grace Blackwell超级芯片的深度解析,再到五分钟快速上手指南和大模型部署技巧,帮助开发者充分利用这款全球最小AI超级计算机的性能优势。

weixin_29239279的博客 1366

有关docker下redis overcommit_memory的问题

背景: [13223] 17 Mar 13:18:02.207 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and t...

seowen的开发 小本子 5636

微店构造订单页面,自动化抢票

之前的文章写过微店抢票如何构造订单页面分析,这篇文章在之前的基础上,进行抢票。

猫敷雪 7208

Android系统去掉长按webview界面弹框中的分享功能

Android 系统去解决长按显示内容,去掉分享功能,不论那个界面,是所有界面都需要满足。如果仅仅去掉系统自带的,那么系统来处理如果客户自定义,部分App自定义的菜单项,不是调用系统的,那么还是需要应用去解决的。之前做过Editext,其实还需要去TextView,最难点的就是WebView中的HTML了 去分享功能了。WebView中涉及到分享功能菜单的界面,/***//***//***//***//***//***//***//***//**

十年饮冰,难凉热血 1021

LINUX的MEMORY OVERCOMMIT

MEMORY OVERCOMMIT

paopaohehe的专栏 468

Oracle EBS 中美国销售税的实现方式

Oracle EBS通过eBTax引擎实现美国销售税管理,需配置税务管辖区、税率及规则等要素,基于收货地址和商品分类计税。系统支持免税交易和使用税处理,但因美国税率复杂多变,建议集成Vertex等第三方服务自动更新税率。关键配置包括创建USSales Tax制度、维护地点层级及物料税务分类,同时需管理客户免税证书。对于跨国业务,系统也支持欧盟VAT等其他国家税制配置。

金牌架构师 297

MEMORY OVERCOMMIT

http://linuxperf.com/?p=102 Overcommit的意思是操作系统承诺给进程的内存大小超过了实际可用的内存。一个保守的操作系统不会允许memory overcommit,有多少就分配多少,再申请就没有了,这其实有些浪费内存,因为进程实际使用到的内存往往比申请的内存要少,比如某个进程malloc()了200MB内存,但实际上只用到了100MB,按照UNIX/Linux的算...

junlon2006的博客 624

ET2012新版免狗免安装绿色软件

ET2012新版免狗免安祖昂绿色软件。,新界面,新视觉,

计算机控制技术及应用

《计算机控制技术及应用》通过总结作者的科研成果和吸收国内外的先进理论、方法和技术,以计算机控制系统的网络化、开放化、智能化和集成化发展趋势为主线索,主要介绍计算机控制系统的基本概念和计算机控制系统的典型形式,计算机控制系统的数学描述与基本分析方法,以及数字PID控制器、最少拍控制系统、达林算法、状态变量设计法等设计方法及其各种改进算法。重点突出计算机控制系统输入输出通道接口的设计方法和典型控制系统的设计案例与实现方法等。

memory overcommit

Memory Overcommit的意思是操作系统承诺给进程的内存大小超过了实际可用的内存。一个保守的操作系统不会允许memory overcommit,有多少就分配多少,再申请就没有了,这其实有些浪费内存,因为进程实际使用到的内存往往比申请的内存要少,比如某个进程malloc()了200MB内存,但实际上只用到了100MB,按照UNIX/Linux的算法,物理内存页的分配发生在使用的瞬间,而不是在申请的瞬间,也就是说未用到的100MB内存根本就没有分配,这100MB内存就闲置了。

wmzjzwlzs的专栏 851

Probability and Statistics - 4th edition

Probability and Statistics - 4th edition

关于系统参数:vm.overcommit_memory 和 vm.overcommit_ratio;

overcomit_memory :     0 :用户申请内存的时候,系统会判断剩余的内存多少,如果不够的话那么就会失败。     1:  用户申请内存的时候,系统不进行任何检查任务内存足够用,直到使用内存超过可用内存。     2: 用户一次申请的内存大小不允许超过可用内存的大小。 overcommit_ratio:     当 overcomot_memory = 2 时,该参数

NI KI 9137

efm32 入门.pdf

本PDF共计22页,介绍的是操作驱动API,下载与否自己斟酌。

linux下overcommit_memory的问题

背景 公司的redis有时background save db不成功,通过log发现下面的告警,很可能由它引起的: [13223] 17 Mar 13:18:02.207 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue a...

轻飘飞扬 2892

linux vm.overcommit_memory 内存分配参数 简介

vm.overcommit_memory 表示内核在分配内存时候做检查的方式。这个变量可以取到0,1,2三个值。对取不同的值时的处理方式都定义在内核源码 mm/mmap.c 的 __vm_enough_memory 函数中。默认值为:0从内核文档里得知,该参数有三个值,分别是:0:当用户空间请求更多的的内存时,内核尝试估算出剩余可用的内存。1:当设这个参数值为1时,内核允许超量使用内存直到用完为止,主要用于科学计算。

whatday的专栏 1万+

什么是overcommit_memory?

1. 前言 在日常的工作中可能我们会经常遇到程序fork失败的问题。遇到fork失败往往有两种可能性: 进程数超标:可以通过cat /proc/sys/kernel/threads-max查看 内存不足 进程数预设值往往很大,几乎不太能超标,所以fork失败的原因大部分都是由于内存不足造成的。 我们知道,由于MMU实现了虚拟地址到物理地址的转换,所以我们在申请虚拟地址时往往可以申请一大块内存,这实际上是对资源的有效利用,毕竟只有内存真正被投入使用时(如memset)才会实际分配物理内存,这种允许内存超额

量子象限 2670

overcommit_memory 内核参数

1.   overcommit_memory是什么?        overcommit_memory是一个内核对内存分配的一种策略。 具体可见/proc/sys/vm/overcommit_memory下的值       2.  overcommit_memory有什么作用?       overcommit_memory取值又三种分别为0, 1, 2     overc

半月旋空 2万+

理解LINUX的MEMORY OVERCOMMIT

理解LINUX的MEMORY OVERCOMMIT 转自 http://linuxperf.com/?p=102 Memory Overcommit的意思是操作系统承诺给进程的内存大小超过了实际可用的内存。一个保守的操作系统不会允许memory overcommit,有多少就分配多少,再申请就没有了,这其实有些浪费内存,因为进程实际使用到的内存往往比申请的内存要少,比如某个进程malloc()了200MB内存,但实际上只用到了100MB,按照UNIX/Linux的算法,物理内存页的分配发生在使用的瞬间,而不

fly0512的博客 383
上一篇: Filesystem Management
下一篇: 【理论篇】计算机网络基础
程序员51
博客等级 码龄12年 215粉丝 140原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值