40. Management of lage files (> 200GB )

Storage architecture.

The /global/work file system (and /global/home too) is served by a number of storage arrays that each contain smaller pieces of the file system, the size of the chunks are 2TB (2000GB) each. In the default setup each file is contained within one storage array so the default filesize limit is thus 2TB. In practice the file limit is considerably smaller as each array contains a lot of files.

Increasing the file size limitation by striping.

Each user can change the default placement of the files it creates by striping files over several storage arrays. This is done with the following command:

lfs setstripe . 0 -1 4

after this has been done all new files created in the current directory will be spread over 4 storage arrays each having 1/4th of the file. The file can be accessed as normal no special action need to be taken. When the striping is set this way it will be defined on a per directory basis so different dirs can have different stripe setups in the same file system, new subdirs will inherit the striping from its parent at the time of creation.

Stripe count recommendation.

We recommend users to set the stripe count so that each chunk will be approx. 200-300GB each, for example

Changing stripe count for files.

Once a file is created the stripe count cannot be changed. This is because the physical bits of the data already are written to a certain subset of the storage arrays. However the following trick can used after one has changed the striping as described above:

# mv file file.bu
# cp -a file.bu file
# rm file.bu

The use of -a flag ensures that all permissions etc are preserved.