|
1 | | -# adaptivemm |
2 | | -## A userspace daemon for proactive memory management |
3 | | - |
| 1 | +# Adaptive Tools |
4 | 2 | ## Overview |
5 | | -adaptivemm offers montioring and tuning of various aspects of system |
6 | | -memory. adaptivemm consists of modules that each offer a different |
7 | | -set of functionality. |
8 | | - |
9 | | -Free memory management module in adaptivemm monitors current state |
10 | | -of free pages overall and free pages of each order. Based upon |
11 | | -current rate of free pages consumption and memory fragmentation, it |
12 | | -predicts if system is likely to run out of memory or if memory will |
13 | | -become severely fragmented in near future. If so, it adjusts |
14 | | -watermarks to force memory reclamation if system is about to run out |
15 | | -of memory. If memory is predicted to become severely fragmented, it |
16 | | -triggers compaction in the kernel. The goal is to avert memory |
17 | | -shortage and/or fragmentation by taking proactive measures. To |
18 | | -arrive at this prediction, adaptivemm samples free pages of each |
19 | | -order on each node periodically and fits a straight line using |
20 | | -method of least squares to these sample points. It also computes |
21 | | -current reclamation rate by monitoring `/proc/vmstat`. The equation |
22 | | -derived for best fit line is used to compute when free memory |
23 | | -exhaustion will occur taking into account current reclamation rate. |
24 | | -If this exhaustion is imminent in near future, watermarks are |
25 | | -adjusted to initiate reclamation. |
26 | | - |
27 | | -Negative dentry management module monitors and adjusts the the |
28 | | -negative dentry limit on the system. Negative dentry limit is |
29 | | -interpreted by kernel as a fraction of total system memory. When a |
30 | | -large number of hugepages are allocated on the system, a cap as a |
31 | | -fraction of total system memory can be too high and can result in |
32 | | -out-of-memory condition since hugepages are not swappable. |
33 | | -adaptivemm can make sure that negative dentry cap is always set to |
34 | | -accomplish what end user intends the cap to be as a fraction |
35 | | -irrespective of number of hugepages allocated on the system. |
36 | | - |
37 | | -Memory leak detection module looks for abnormal changes to free |
38 | | -memory. It uses data reported in /proc/meminfo to determine how much |
39 | | -memory is in use and based upon that number, does the amount of free |
40 | | -memory look reasonable. |
41 | | - |
42 | | -## How it works |
43 | | - |
44 | | -adaptivemm free memory management module samples `/proc/buddyinfo` |
45 | | -to monitor the total available free memory and the fraction thereof |
46 | | -that, whilst notionally free, requires compaction before it can be |
47 | | -used. At any given moment, the difference between these two |
48 | | -quantities is the amount of memory that is free for immediate use: |
49 | | -once this is exhausted, subsequent allocations will stall whilst a |
50 | | -portion of the fragmented memory is compacted. The program |
51 | | -calculates trends for both the total and fragmented free memory and, |
52 | | -by extrapolation, determines whether exhaustion is imminent. At the |
53 | | -last possible moment, i.e. only once it is absolutely necessary, the |
54 | | -program initiates compaction with a view to recovering the |
55 | | -fragmented memory before it is required by subsequent allocations. |
56 | | -It initiates compaction by writing 1 to |
57 | | -`/sys/devices/system/node/node%d/compact`. If number of free pages |
58 | | -is expected to be exhausted, it looks at the number of inactive |
59 | | -pages in cache buffer to determine if changing watermarks can result |
60 | | -in meaningful number of pages reclaimed. It adjusts watermark by |
61 | | -changing watermark scale factor in |
62 | | -`/proc/sys/vm/watermark_scale_factor`. |
63 | | - |
64 | | -## Prerequisites |
65 | | - |
66 | | -adaptivemm must be run as root and must have access to following files: |
67 | | - |
68 | | -`/proc/vmstat` |
69 | | -`/proc/buddyinfo` |
70 | | -`/proc/zoneinfo` |
71 | | -'/proc/kpagecount' |
72 | | -'/proc/kpageflags' |
73 | | -`/proc/sys/vm/watermark_scale_factor` |
74 | | -`/sys/devices/system/node/node%d/compact` |
75 | | -'/proc/sys/fs/negative-dentry-limit' |
76 | | - |
77 | | -adaptivemm daemon can be run standalone or can be started |
78 | | -automatically by systemd/init. |
79 | | - |
80 | | -## Usage |
81 | | - |
82 | | -adaptivemm supports three levels of aggressiveness - 1 = Low, 2 = |
83 | | -Normal (default), 3 = High. Aggressiveness level dictates how often |
84 | | -adaptivemm will sample system state and how aggressively it will |
85 | | -tune watermark scale factor. |
86 | | - |
87 | | -In simplest form, adaptivemmd can be started with: |
88 | | - |
89 | | - $ adaptivemmd |
90 | | - |
91 | | -Aggressiveness level can be changed with: |
92 | | - |
93 | | - $ adaptivemmd -a 3 |
94 | | - |
95 | | -If a maximum gap allowed between low and high watermarks as |
96 | | -adaptivemm tunes watermarks, is desired, it can be specified with |
97 | | -a -m flag where the argument is number of GB: |
98 | | - |
99 | | - $ adaptivemmd -m 10 |
100 | | - |
101 | | -## Developer Resources |
102 | | - |
103 | | -To develop for adaptivemm, obtain the source code from git repository at https://github.com/oracle/adaptivemm or from the source package. Source code is broken down into two primary files - adaptivemmd.c contains the main driving code for the daemon which does all the initialization and takes action in response to the results from prediction algorithm, predict.c contains the core of prediction algorithm which uses least square fit method to calculate a trend line for memory consumption. |
104 | | - |
105 | | -## Prerequisite to building |
106 | | - |
107 | | -adaptivemm requires only the basic build tools - gcc compiler and linker, and make. |
108 | | - |
109 | | -## Building |
110 | | - |
111 | | -Run `make all` to build the adaptivemmd daemon. |
112 | | - |
113 | | -Run `make clean` to remove binaries and intermediate files generated by build process. |
114 | | - |
115 | | -## Installation |
116 | | - |
117 | | -Copy adaptivemmd binary to a directory appropriate for your system, typically `/usr/sbin`. adaptivemm can use a configuration file as well which is `/etc/sysconfig/adaptivemmd` on rpm based system and `/etc/default/adaptivemmd` on deb based systems. Here is a sample configuration file: |
118 | | - |
119 | | - |
120 | | - # Configuration file for adaptivemmd |
121 | | - # |
122 | | - # Options in this file can be overridden with command line options |
123 | | - |
124 | | - # verbosity level (1-5) |
125 | | - VERBOSE=0 |
126 | | - |
127 | | - # Aggressiveness level for adaptivemmd (1-3) |
128 | | - AGGRESSIVENESS=2 |
129 | | - |
130 | | - # ============================ |
131 | | - # Free page management section |
132 | | - # ============================ |
133 | | - # Enable management of free pages through watermarks tuning |
134 | | - ENABLE_FREE_PAGE_MGMT=1 |
135 | | - |
136 | | - # Maximum gap between low and high watermarks (in GB) |
137 | | - # MAXGAP=5 |
138 | | - |
139 | | - # ================================== |
140 | | - # Negative dentry management section |
141 | | - # ================================== |
142 | | - # Enable management of negative dentry cap |
143 | | - ENABLE_NEG_DENTRY_MGMT=1 |
144 | | - |
145 | | - # Cap for memory consumed by negative dentries as a fraction of 1000. |
146 | | - # Range of values supported by kernel is 1-100. |
147 | | - # NOTE: for kernels with support for this functionality |
148 | | - # (Hint: look for /proc/sys/fs/negative-dentry-limit) |
149 | | - NEG_DENTRY_CAP=15 |
150 | | - |
151 | | - # ============================== |
152 | | - # Memory leack detection section |
153 | | - # ============================== |
154 | | - # Enable checks for possible memory leaks |
155 | | - ENABLE_MEMLEAK_CHECK=1 |
156 | | - |
157 | | - |
158 | | -## Documentation |
159 | | - |
160 | | -Source code includes documentation in form of a man page. This man page is contained in file `adaptivemmd.8`. If adaptivemm was installed as a package on your system, man page should be available as standard man page with `man adaptivemmd` command. If adaptivemm was not installed as a package, `adaptivemmd.8` file can be displayed as man page with `nroff -man adaptivemmd.8`. |
161 | | - |
162 | | -## Contributing |
163 | | - |
164 | | -This project welcomes contributions from the community. Before submitting a pull request, please [review our contribution guide](./CONTRIBUTING.md) |
165 | | - |
166 | | -## Security |
167 | | - |
168 | | -Please consult the [security guide](./SECURITY.md) for our responsible security vulnerability disclosure process |
| 3 | +This repo contains the source for the adaptivemm and adaptived tools. |
169 | 4 |
|
170 | | -## License |
| 5 | +[adaptivemm](./adaptivemm/README.md) monitors a machine's memory usage to track the rate of page consumption. It then uses this information to predict future memory usage and adjusts the watermark scale factor sysctl to kick off proactive background memory reclamation. This is done to avoid costly synchronous memory reclaim which can stall applications. |
171 | 6 |
|
172 | | -Copyright (c) 2019, 2023 Oracle and/or its affiliates. |
| 7 | +[adaptived](./adaptived/README.md) is a cause-and-effect daemon in which a user can specify their own causes which are used to decide if a certain action should be done in reponse to this cause. |
| 8 | + <br/> |
173 | 9 |
|
174 | | -Released under the GNU Public License version 2 |
| 10 | + Further information for both tools is found in their respective directories. |
0 commit comments