[LeapList]Directory Linking

[John Simpson]

--Apple-Mail-1--671233991
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=US-ASCII; format=flowed

On May 19, 2004, at 11:29, Ken Nordquist wrote:
>
> My initial thought is to soft link site directories to a main core
> directory.  Will I run into problems as the number of soft links grows?
> Can anyone think of any other issues?

if it's a case where there's one physical copy and many symlinks which 
all point to that one physical location (i.e. A is the physical 
locaion, B is a symlink to A, C is a symlink to A, D is a symlink to A, 
etc.) then there is no issue using symlinks instead of physical copies 
of the data- in fact this is how i did something similar for one of my 
clients.

if it's a case where you have a chain of symlinks (i.e. A is the 
physical location, B is a symlink to A, C is a symlink to B, D is a 
symlink to C, etc.) then each layer will slow things down.

===

there is something else to watch for, however... if you see this 
growing to the point of having hundreds or thousands of sites on the 
machine, you may want to break up the sites into separate directories. 
having a single directory with more than a certain number of entries 
(files, directories, whatever) can slow things down as well. that 
number depends on the machine's speed and memory usage- i've always 
tried to avoid having more than 1000 entries in any one directory, 
although with enough memory for caching i've seen directories with over 
100,000 entries still be able to find a given file fairly quickly.

here's an example that illustrates the problem: say you want to read 
the file "/home/ken/example/blah.txt". in order to open this file, the 
kernel has to do the following:

open the inode representing the root directory
read entries over and over until it finds "home"
open the corresponding inode
read entries over and over until it finds "ken"
open the corresponding inode
read entries over and over until it finds "example"
open the corresponding inode
read entries over and over until it finds "blah.txt"
open that inode and return a handle to the userland process

there's a lot of "read entries over and over" in there. if one of the 
directories happens to contain a lot of entries, this searching process 
can take a relatively long time. note that some filesystems are based 
on binary tree structures and this may not apply... but i know from 
experience that ext2, ext3, and all of the FAT filesystems work in this 
manner- each directory contains a list of its children (files, child 
directories, etc.) and the only way to find a particular child is to do 
a sequential search- read every entry until you find the one you want.

i mention this because it's something i've run into several times 
(having spent so many years building and running ISPs.) for example, a 
small ISP with 5000 users... if they keep the linux default of using 
"/home/{userid}" as each user's home directory, the directory "/home" 
will have 5000 entries, and the newer users (the ones whose directories 
are listed later within the directory) will experience longer seek 
times to read files in their home directories.

you might think with caching it wouldn't be a big issue because the 
blocks containing those directory entries would already be in memory- 
but this machine was doing pop3, imap, and personal web pages for all 
of these users, and it was constantly serving a lot of different files- 
the average lifetime of any cached disk block was about two seconds. 
(yes, i know more RAM would have helped, but the budget wasn't there 
for it.)

the solution, at least for the number of users we had, was to use 
"/home/{letter}/{userid}" as the home directory, i.e. "/home/j/jms1" or 
"/home/b/bubba". this way "/home" contains 28 entries (the 26 letters 
plus "." and "..") and the child directories within it will each have a 
lower number of entries (in this case the biggest one was "/home/n" 
with a little over 200 entries.)

for a larger system i would probably do something like "/home/j/m/jms1" 
and "/home/b/u/bubba"... and the point where you go to a second layer 
of indirection like this would depend on the system itself, my guess 
would be somewhere in the range of 15K-20K users.

qmail uses a similar idea for managing its queue- each message has a 
fixed number assigned, which is the same as the inode number of the 
file containing the message text (thereby guaranteeing a unique 
number.) qmail takes this number, divides it by 23, and uses the 
remainder to tell which "bucket" to place the message in... this keeps 
the buckets all roughly the same size, and it works well until the 
queue reaches about 20,000 messages. if the machine will routinely be 
handling so much mail where you need your queue to be able to hold this 
many messages, you can recompile qmail with a larger number of 
"buckets"- any prime number works well.

there's another issue you may run into as a result of having many files 
in any one directory... if you type "/bin/echo *", the shell will 
expand the "*" to a list of the items in the current directory (files, 
directories, etc.) before trying to call exec() to run /bin/echo. a 
while back there was a thread about using a ridiculously long path and 
not being able to run commands, and i walked the kernel source code to 
discover that there's a limit of 128K for the environment plus the 
command line on any new process.

http://www.matrixlist.com/pipermail/leaplist/2002-August/023528.html

if you're in a directory with thousands of files and use a shell 
expansion which results in more than 128K combined environment and 
command line, you will get "Argument list too long" instead of running 
the command. (the program "xargs" is your friend if you're in this 
situation.)

oops... i didn't mean to walk that far on a tangent, sorry if i've 
bored anybody to tears...

-----------------------------------------------
| John Simpson - KG4ZOW - Programmer At Large |
| http://www.jms1.net/        <jms1@jms1.net> |
-----------------------------------------------

--Apple-Mail-1--671233991
content-type: application/pgp-signature; x-mac-type=70674453;
	name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iD8DBQFAq/DuEB9RczMG/PsRAtAfAKCwVKKul20l6TWTiUNsFxwYwJixZgCfXJ/N
rIEpIoxc/WEp4a8nxMQFuFE=
=WFIY
-----END PGP SIGNATURE-----

--Apple-Mail-1--671233991--