wiki:BoincFiles

Version 3 (modified by davea, 17 years ago) (diff)

--

Storage

Files and data servers

The BOINC storage model is based on files. Examples of files:

  • The inputs and/or outputs of computation;
  • Components of application: executables, libraries, etc.
  • Data for its own sake, e.g. to implement a distributed storage system.

The BOINC core client transfers files to and from project-operated data servers using HTTP.

Once a file is created (on a data server or a participant host) it is immutable. This means that all replicas of that file are assumed (and required) to be identical. If a file is changed, even by a single byte, it becomes a new file, and must be given a different name than the original.

File properties

Files have various properties, including:

  • Name: unique identifier for the file. Must be nonempty, at most 255 chars, and may not contain '..' or '%'.
  • Sticky: don't delete file on client (see below).
  • Report on RPC: include a description of this file in scheduler requests.
  • Maximum size: if an output file exceeds its maximum size, the computation is aborted.

File properties are specified in XML elements that appear, for example, in workunit templates.

File references

In addition to the properties of a file itself, there are several properties of the way a particular application uses a file; this is called a file reference.

  • open_name (or 'logical name'): the name by which the program will refer to the file (BOINC provides a mechanism for mapping logical names to physical names, so that your applications don't have to know the physical names of their input and output files).
  • copy file: use this if your application doesn't use BOINC's filename-mapping mechanism.
  • optional: Use this for output files, to indicate that the application doesn't always generate the file, and that its absence should not be treated as an error.

File reference properties are specified in XML elements.

File management

BOINC's default behavior is to delete files when they aren't needed any more. Specifically:

  • On the client, input files are deleted when no workunit refers to them, and output files are deleted when no result refers to them. Application-version files are deleted when they are referenced only from superseded application versions.
  • On the client, the 'sticky' flag overrides the above mechanisms and suppresses the deletion of the file. The file may deleted by an explicit server request. The file may also be deleted at any time by the core client in order to honor limits on disk-space usage.
  • On the server, the file deleter daemon deletes input and output files that are no longer needed. This can be suppressed using the 'no_delete' flag, or using command-line options to the file deleter.