Recent searches
Search options
@lkundrak
expert? me?
@vbabka @ljs @sj are the experts.
I am just a dumb/curious undergraduate without Ph.D nor B.S. (yet) XD
The main benefit of NUMA architecture is to distribute memory bus traffics to several memory buses instead of a single global bus, because the global bus can be bottleneck as the number of CPUs and memory capacity grows.
A set of CPUs and memory near to those CPUs is called a NUMA node. If a CPU wants to access memory not in the local node, it reads data from a remote node via interconnect (instead of the local, faster bus)
Because local (to cpu) and remote NUMA node has different access latency and bandwidth, OS tries to utilize local node's memory first (ofc that depends on NUMA memory policy of the task/VMA)
But a laptop is too cheap and small system for a single bus to be a bottleneck, so I don't get why the hardware designer decided to adopt NUMA architecture.
And it's really strange that different ranges of physical memory from a single DIMM chip belongs to different NUMA nodes. Do they really have different performance characteristics?
expert? me?
@vbabka @ljs @sj are the experts.
I am just a dumb/curious undergraduate without Ph.D nor B.S. (yet) XD
The main benefit of NUMA architecture is to distribute memory bus traffics to several memory buses instead of a single global bus, because the global bus can be bottleneck as the number of CPUs and memory capacity grows.
A set of CPUs and memory near to those CPUs is called a NUMA node. If a CPU wants to access memory not in the local node, it reads data from a remote node via interconnect (instead of the local, faster bus)
Because local (to cpu) and remote NUMA node has different access latency and bandwidth, OS tries to utilize local node's memory first (ofc that depends on NUMA memory policy of the task/VMA)
But a laptop is too cheap and small system for a single bus to be a bottleneck, so I don't get why the hardware designer decided to adopt NUMA architecture.
And it's really strange that different ranges of physical memory from a single DIMM chip belongs to different NUMA nodes. Do they really have different performance characteristics?
@hyeyoo @lkundrak @sj @vbabka bro I'm not I'm an imposter, you're the real one. I don't even work in kernel mm (atm anyway)
Plus age/qualifications don't matter, you got talent which can't be taught. I have an undergrad in civ eng + taught myself :)
I'd say main benefit of NUMA isn't bottleneck, but rather accounting for different time taken for memory accesses thus allowing the kernel to stop you doing something stupid.
I always picture the literal physical setup of a 2 socket system where there's ram attached to each core and a slow interconnect between the two, you don't want to be using that interconnect!
I guess you could say you are trying to avoid the 'global bus' if this == the interconnect.
You find that by default most x86 just has NUMA turned on anyway even in laptop situations, I mean my desktop does too, just with a single node.
I sort of feel like we should have CONFIG_NUMA turned on by default, as it would simplify the code, and just say ok there's 1 node, and all the various mem policy stuff won't make any difference.
One thing that bugged me on my arm64 laptop is that put everything in ZONE_DMA because there zones don't matter. But still kind of... ugly
Plus age/qualifications don't matter, you got talent which can't be taught. I have an undergrad in civ eng + taught myself :)
I'd say main benefit of NUMA isn't bottleneck, but rather accounting for different time taken for memory accesses thus allowing the kernel to stop you doing something stupid.
I always picture the literal physical setup of a 2 socket system where there's ram attached to each core and a slow interconnect between the two, you don't want to be using that interconnect!
I guess you could say you are trying to avoid the 'global bus' if this == the interconnect.
You find that by default most x86 just has NUMA turned on anyway even in laptop situations, I mean my desktop does too, just with a single node.
I sort of feel like we should have CONFIG_NUMA turned on by default, as it would simplify the code, and just say ok there's 1 node, and all the various mem policy stuff won't make any difference.
One thing that bugged me on my arm64 laptop is that put everything in ZONE_DMA because there zones don't matter. But still kind of... ugly
@ljs @lkundrak @sj @vbabka
node 1 is cpuless, and
$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 0 size: 11937 MB
node 0 free: 6770 MB
node 1 cpus:
node 1 size: 23954 MB
node 1 free: 23854 MB
node distances:
node 0 1
0: 10 20
1: 20 10
I was like "wtf did I turn on fake numa?" but no.
$ cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt3)/vmlinuz-6.6.0-rc4+ root=UUID=96f9e501-caa5-4c39-bc11-5d104517f08d ro rootflags=subvol=root loglevel=8
node 1 is cpuless, and
$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
node 0 size: 11937 MB
node 0 free: 6770 MB
node 1 cpus:
node 1 size: 23954 MB
node 1 free: 23854 MB
node distances:
node 0 1
0: 10 20
1: 20 10
I was like "wtf did I turn on fake numa?" but no.
$ cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt3)/vmlinuz-6.6.0-rc4+ root=UUID=96f9e501-caa5-4c39-bc11-5d104517f08d ro rootflags=subvol=root loglevel=8
@Aissen @hyeyoo @lkundrak @sj @vbabka
^ this
The only cpuless structure I'd heard of before was some powerpc madness.
I guess it could be some 2-tier thing where node 1 is just slower for EVERYTHING.
You can have weirdness actually if there's a dimm size mismatch because node 0 might be dual channel (whole of 1 dimm, portion of another I guess?) and the rest is single channel?
Fact numactl --hardware suggests node 1 twice as slow hints at this
^ this
The only cpuless structure I'd heard of before was some powerpc madness.
I guess it could be some 2-tier thing where node 1 is just slower for EVERYTHING.
You can have weirdness actually if there's a dimm size mismatch because node 0 might be dual channel (whole of 1 dimm, portion of another I guess?) and the rest is single channel?
Fact numactl --hardware suggests node 1 twice as slow hints at this
@vbabka @lkundrak@octodon.social @hyeyoo @ljs @sj @Aissen For the record, I was talking about SGI Altix 3000, which was built of “bricks”. The C-brick contained four Itanium 2 CPUs and 32 GB RAM (that should be enough for everybody), but if it wasn't enough for you, you could buy an M-brick, which contained only memory, no CPU.