@ayo I like the mtime stuff! 😉 Thanks for the feedback on my patches.
@blunaxela Ooh I hadn't realized you were on Mastodon! Good work on the mtime patches, most people give up after my first round of feedback. 😅
@ayo Yeah, I saw your mastodon handle on your webpage and was tempted to send patches on here instead! 😝 BTW, I have a silly change set that makes dir_scan.c pthreaded. Still pretty slow... It's maybe 2x faster for NFS, but other cases are much slower. I'll let you know if I make any real progress.
@blunaxela So you weren't kidding about that threading!
Now I'm curious about your approach. The dir_output interface doesn't allow for much parallelism; Probably the only thing that can be "easily" done in parallel are the stat() calls for all files in a single directory, but I'd imagine quite a high synchronization overhead for that in many cases.
My approach has been to start a thread for each dir up to N threads. I had remove a number of global variables and pass the current dir context in order to make dir_walk and recur reentrant. It also fstats by full path since chdir isn't thread friendly. The most expensive part right now is mallocs and it's threading on every item, instead of just dirs. There's a lock on item() to protect the tree and addstatstoparents()
Oh, I just realized that there could be a lock on item() for each independent branch of the tree as well! That might be interesting to coordinate.
As for PATH_MAX, I want to switch to fstatat(). Then there's less checking for fated as well. Although I'm starting to wonder what's POSIX and what's Linux.
@blunaxela The point of export is that it doesn't need much memory...
I think the best approach is keeping the threading inside dir_scan.c and serialize/ coordinate calls to item() to ensure correct ordering. This complicates and limits the task distribution a bit, but it keeps the complexity from affecting other parts.
Fosstodon is a Mastodon instance that is open to anyone who is interested in technology; particularly free & open source software.