Writing Modules: GnomeVFS - Filesystem Abstraction library

Writing Modules

Writing Modules — basic gnome-vfs module concepts

Introduction

This section will introduce the basic concepts that are needed for writing GNOME Virtual File System modules.

GNOME VFS URIs (Uniform Resource Identifiers)

The GNOME Virtual file system uses URIs similiar to the standard WWW URIs. The basic difference between a VFS URI and WWW URI is that, while with WWW URIs you can only use a single protocol for accessing a certain file, with GNOME VFS URIs you can combine different access methods in sequence.

For example, suppose you want to access file hello.c in a tar.gz file which is in turn accessible through FTP from a remote machine. In order to access this file, you would need to:

Connect to the FTP site.
Fetch the tar.gz file.
Decompress the tar.gz file using GZIP.
Extract hello.c from the resulting uncompressed tar file.

The GNOME Virtual File System lets you express this by combining the three access methods (i.e. tar, GZIP and FTP) into a single URI. Access methods are combined in the URI by using the `#' character, followed by the name for the access method and the subpath for that specific access method. The subpath can be omitted for those storage methods that do not need a path to retrieve the file. (For example, a GZIP file always contains a single uncompressed file, so no path is needed to locate the uncompressed file within the GZIP file. But on the other hand, the TAR method requires a path to locate a specific file or directory.)

For example, in the case we outlined above, the URI would look something like:


        ftp://username:password@host.net/path/to/file.tar.gz#gzip:#tar:/path/to/hello.c

Each method/subpath couple is called a URI element. When URI elements are combined like this, each URI element uses the previous one to access a base resource into which it will look up a file, using the subpath information. For this reason, we will say that each element is the parent element for the following one.

The first URI element, the one which has no parent, is called the toplevel element. It does not use the `#' character; instead, it uses the standard syntax of WWW URIs:


        method://user:password@host/path/to/file

This way, normal WWW URIs can be used with the GNOME Virtual File System.

Toplevel elements are also special because they let users specify user names, passwords and host names, while non-toplevel elements don't.

The GnomeVFSURI type

Within the GNOME Virtual File System library, URI elements are represented by a special type, GnomeVFSURI, which is meant to represent user-provided URIs in a machine-optimized way.

Every GnomeVFSURI contains the following information:

A reference counter
A pointer to the parent GnomeVFSURI URI element.
The subpath.
The name of the access method.
A pointer to a GnomeVFSMethod object, describing the access method (see below).

GNOME Virtual File System access method implementation

In the GNOME Virtual File System, the implementations for all the access methods are loaded at runtime, as shared library modules. The modules are loaded during parsing of the string URI. If the parser encounters an access method for which no implementation is currently loaded, it retrieves the corresponding library file, dynamically links it into the executable, and initializes it.

After initialization, the module returns a special GnomeVFSMethod object that contains pointers to the various implementation functions for that specific method. By storing a pointer to this object into the GnomeVFSURI type, the VFS library is then able to use these functions for file access.

How file access is performed

When the VFS library needs to perform some file operation, it performs the following steps:

If the URI is given in textual form (i.e. as a string), it parses it and activates the necessary access method modules.
It retrieves a pointer to the lowmost level URI element.
It retrieves a pointer to the GnomeVFSMethod object that corresponds to the access method for that URI element.
It retrieves a pointer to the implementation function for that operation from the GnomeVFSMethodobject.
It invokes that implementation function passing the pointer to the lowmost level URI element.

Combining the access methods is always done within the method implementation. If the method implementation needs to do some file operation on the the parent URI element, it can do so by simply invoking the corresponding VFS function, by using the parent pointer in the GnomeVFSURI object.

For example, suppose you have to read a simple URI like the following:


        file:/home/ettore/file.gz#gzip:

In this case, the GZIP method will be invoked with a pointer to the GnomeVFSURI describing the `gzip' part. The GZIP method will be able to read file.gz by just invoking the corresponding GNOME VFS library function on its parent and decompressing it on the fly.

Implementing an access method in practice

Implementing a new access method is really not difficult at all. This section explains how this is done.

Using shared libraries

Every module must be compiled as a shared library (i.e. a .so file).

The current way for accessing the right module for the right method is very simple, and is based on file names. In practice, a module implementing an access method named foo must be named libfoo.so. For example, the module implementing the ftp: access method is called libftp.so, the module implementing #gzip: access is called libgzip.so and so on.

This might change in the future.

The initialization/shutdown functions

Every shared library module must provide two functions:


GnomeVFSMethod *vfs_module_init (const char *method_name, const char *args);
void vfs_module_shutdown (GnomeVFSMethod *method);

These are the only functions that the VFS library will access directly. All the other symbols (i.e. functions and variables) in the module should be made static.

vfs_module_init() is called as soon as the module is loaded in memory. It will have to return a pointer to a GnomeVFSMethod object that will contain the pointers to the method's implementation functions. We will describe this later.

vfs_module_shutdown, instead, is called before the module is unloaded or the program that uses it dies. This functions should:

Deallocate all the memory allocated by the module.
Close all the file descriptors associated with the module.
Kill any external process spawned by the module.
In general, make sure that any operation that was going on before this function was called will be interrupted correctly, as soon as possible and without any leaks.

The GnomeVFSMethod object

This object contains pointers to the module implementation functions.

A method can choose itself which functions to implement. However, it must at least provide a GnomeVFSMethodOpenFunc and GnomeVFSMethodIsLocalFunc implementation.

struct GnomeVFSMethod {
	gsize method_table_size;			/* Used for versioning */
	GnomeVFSMethodOpenFunc open;
	GnomeVFSMethodCreateFunc create;
	GnomeVFSMethodCloseFunc close;
	GnomeVFSMethodReadFunc read;
	GnomeVFSMethodWriteFunc write;
	GnomeVFSMethodSeekFunc seek;
	GnomeVFSMethodTellFunc tell;
	GnomeVFSMethodTruncateHandleFunc truncate_handle;
	GnomeVFSMethodOpenDirectoryFunc open_directory;
	GnomeVFSMethodCloseDirectoryFunc close_directory;
	GnomeVFSMethodReadDirectoryFunc read_directory;
	GnomeVFSMethodGetFileInfoFunc get_file_info;
	GnomeVFSMethodGetFileInfoFromHandleFunc get_file_info_from_handle;
	GnomeVFSMethodIsLocalFunc is_local;
	GnomeVFSMethodMakeDirectoryFunc make_directory;
	GnomeVFSMethodRemoveDirectoryFunc remove_directory;
	GnomeVFSMethodMoveFunc move;
	GnomeVFSMethodUnlinkFunc unlink;
	GnomeVFSMethodCheckSameFSFunc check_same_fs;
	GnomeVFSMethodSetFileInfo set_file_info;
	GnomeVFSMethodTruncateFunc truncate;
	GnomeVFSMethodFindDirectoryFunc find_directory;
	GnomeVFSMethodCreateSymbolicLinkFunc create_symbolic_link;
	GnomeVFSMethodMonitorAddFunc monitor_add;
	GnomeVFSMethodMonitorCancelFunc monitor_cancel;
	GnomeVFSMethodFileControlFunc file_control;
	GnomeVFSMethodForgetCacheFunc forget_cache;
	GnomeVFSMethodGetVolumeFreeSpaceFunc get_volume_free_space;
};

typedef GnomeVFSResult (* GnomeVFSMethodOpenFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle
			       	 	**method_handle_return,
					 GnomeVFSURI *uri,
					 GnomeVFSOpenMode mode,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodCreateFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle
			       	 	**method_handle_return,
					 GnomeVFSURI *uri,
					 GnomeVFSOpenMode mode,
					 gboolean exclusive,
					 guint perm,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodCloseFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle *method_handle,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodReadFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle *method_handle,
					 gpointer buffer,
					 GnomeVFSFileSize num_bytes,
					 GnomeVFSFileSize *bytes_read_return,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodWriteFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle *method_handle,
					 gconstpointer buffer,
					 GnomeVFSFileSize num_bytes,
					 GnomeVFSFileSize *bytes_written_return,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodSeekFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle *method_handle,
					 GnomeVFSSeekPosition  whence,
					 GnomeVFSFileOffset    offset,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodTellFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle *method_handle,
					 GnomeVFSFileSize *offset_return);

typedef GnomeVFSResult (* GnomeVFSMethodOpenDirectoryFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle **method_handle,
					 GnomeVFSURI *uri,
					 GnomeVFSFileInfoOptions options,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodCloseDirectoryFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle *method_handle,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodReadDirectoryFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle *method_handle,
					 GnomeVFSFileInfo *file_info,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodGetFileInfoFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSURI *uri,
					 GnomeVFSFileInfo *file_info,
					 GnomeVFSFileInfoOptions options,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodGetFileInfoFromHandleFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSMethodHandle *method_handle,
					 GnomeVFSFileInfo *file_info,
					 GnomeVFSFileInfoOptions options,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodTruncateFunc) (GnomeVFSMethod *method,
						       GnomeVFSURI *uri,
						       GnomeVFSFileSize length,
						       GnomeVFSContext *context);
typedef GnomeVFSResult (* GnomeVFSMethodTruncateHandleFunc) (GnomeVFSMethod *method,
							     GnomeVFSMethodHandle *handle,
							     GnomeVFSFileSize length,
							     GnomeVFSContext *context);

typedef gboolean       (* GnomeVFSMethodIsLocalFunc)
					(GnomeVFSMethod *method,
					 const GnomeVFSURI *uri);

typedef GnomeVFSResult (* GnomeVFSMethodMakeDirectoryFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSURI *uri,
					 guint perm,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodFindDirectoryFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSURI *find_near_uri,
					 GnomeVFSFindDirectoryKind kind,
					 GnomeVFSURI **result_uri,
					 gboolean create_if_needed,
					 gboolean find_if_needed,
					 guint perm,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodRemoveDirectoryFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSURI *uri,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodMoveFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSURI *old_uri,
					 GnomeVFSURI *new_uri,
					 gboolean force_replace,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodUnlinkFunc)
                                        (GnomeVFSMethod *method,
					 GnomeVFSURI *uri,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodCheckSameFSFunc)
					(GnomeVFSMethod *method,
					 GnomeVFSURI *a,
					 GnomeVFSURI *b,
					 gboolean *same_fs_return,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodSetFileInfo)
					(GnomeVFSMethod *method,
					 GnomeVFSURI *a,
					 const GnomeVFSFileInfo *info,
					 GnomeVFSSetFileInfoMask mask,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodCreateSymbolicLinkFunc)
                                        (GnomeVFSMethod *method,
                                         GnomeVFSURI *uri,
                                         const gchar *target_reference,
                                         GnomeVFSContext *context);
typedef GnomeVFSResult (* GnomeVFSMethodMonitorAddFunc)
     					(GnomeVFSMethod *method,
      					 GnomeVFSMethodHandle **method_handle_return,
      					 GnomeVFSURI *uri,
      					 GnomeVFSMonitorType monitor_type);

typedef GnomeVFSResult (* GnomeVFSMethodMonitorCancelFunc)
     					(GnomeVFSMethod *method,
      					 GnomeVFSMethodHandle *handle);

typedef GnomeVFSResult (* GnomeVFSMethodFileControlFunc)
     					(GnomeVFSMethod *method,
      					 GnomeVFSMethodHandle *method_handle,
					 const char *operation,
					 gpointer operation_data,
					 GnomeVFSContext *context);

typedef GnomeVFSResult (* GnomeVFSMethodForgetCacheFunc)
     					(GnomeVFSMethod *method,
      					 GnomeVFSMethodHandle *method_handle,
					 GnomeVFSFileOffset offset,
					 GnomeVFSFileSize size);

typedef GnomeVFSResult (* GnomeVFSMethodGetVolumeFreeSpaceFunc)
     					(GnomeVFSMethod *method,
					 const GnomeVFSURI *uri,
				 	 GnomeVFSFileSize *free_space);

Handling cancellation

As VFS operations might take very long to complete, especially in the case of transient errors (such as a network server that has gone down), the GNOME Virtual File System Library provides a standard way to handle the cancellation of VFS operations.

The GnomeVFSCancellation object

The object that encapsulates this functionality is GnomeVFSCancellation. Most implementation functions get a pointer to such an object, and are expected to use this object to recognize when an operation should be interrupted.

The most simple way to check for a cancellation request is to poll the object with gnome_vfs_cancellation_check():

  
gboolean gnome_vfs_cancellation_check (GnomeVFSCancellation *cancellation);

This function will return a nonzero value if the current operation should be cancelled.

Notice that cancellation is an asynchronous operation that might happen outside your function, in parallel with the code that you are writing. For example, in the case of threads, the request will be set in the master thread; in the case of slave CORBA-driven processes, the request will be activated by a Unix signal. So you can expect a cancellation request to happen (and consequently be signalled in GnomeVFSCancellation) at any time.

For this reason, you should be calling this function periodically, whenever you are going to perform several iterations of the same task, or execute a single expensive task. When the function returns a nonzero value, the correct way to react is:

Clean things up so that the result of the operations that have been performed are all cancelled.
Return the GNOME_VFS_ERROR_CANCELLED error code.

Note, there are some other situations in which you want to be able to interrupt an I/O operation when a cancellation request is performed. In such cases, polling is not a viable option.

For this reason, GnomeVFSCancellation provides an alternative way of sending notifications, using a file descriptor. To use this feature, you should use the following function:


gint gnome_vfs_cancellation_get_fd (GnomeVFSCancellation *cancellation);

When this function is called, it will return an open file descriptor, which is the read-side of a pipe. The pipe will be given a character from the write side as soon as a cancellation request is sent. You can check for a cancellation by using the select() system call with this file descriptor. As soon as select reports that some data is available on this file descriptor, you know that a cancellation is being requested.

For example, if you are reading from a file descriptor and you want to check for a pending cancellation at the same time, you can set up selectfor checking if data is available on both the cancellation file descriptor and the file descriptor you are reading from.

Dealing with EINTR

In order to maximize the chance of cancelling an operation immediately, the GNOME Virtual File System can sends a signal to the asynchronous thread or process. This does not happen on all the systems and setups, though.

The result of this is that, if a process is in the middle of a Unix system call while receiving this signal, the system call might be interrupted and return a EINTR error.

For this reason, when you receive EINTR you should check if a cancellation request is pending, using gnome_vfs_cancellation_check() on the GnomeVFSCancellation object that the implementation function received:

If a cancellation is indeed pending (gnome_vfs_cancellation_check() returns a nonzero value), you should cancel the operation, cleaning up all the effects, and return GNOME_VFS_ERROR_INTERRUPTED or GNOME_VFS_ERROR_CANCELLED
Otherwise, retry the system call as you would normally do.

Basic guidelines for writing a module

Writing GNOME VFS modules is easy, but there are a few things that you must keep in mind when hacking them:

All of the code must be completely thread safe. The reason for this is that the asynchronous GNOME VFS engine will use threads when available; if you don't make sure that the code is thread-safe, every kind of weird and unexpected errors will happen. As debugging these problems can be very hard, it's important to write the code with threads in mind right from the start.
Use the special gnome_vfs_*_cancellable() VFS functions instead of the standard non-cancellable ones, passing them the same GnomeVFSCancellation object you are given, so that the operation can always be interrrupted at any time.
The code should respect the basic GNOME guidelines for source code indentation and style.

How to make the code thread safe

Although it might sound scary at first, making the code for the modules thread safe is not complicated at all.

First of all, make sure the amount of global variables is kept to the bare minimum. If possible, you should avoid them at all cost.

For those cases where globals are inevitable (such as caches, connection pools or things like that), you have to make sure every variable is properly associated with a mutex, and that the mutex is locked before every access to this variable and released afterwards. You can also use G_LOCK_DEFINE_STATIC, G_LOCK and G_UNLOCK for this.

Generally speaking, if you are going to dynamically allocate structures that are shared by more than one operation/file, you should provide all of them with their nice mutex locks.

Finally, make sure mutexes are used only if they are available. One way to do so is to use macros like the following:


#ifdef G_THREADS_ENABLED
#define MUTEX_NEW()     g_mutex_new ()
#define MUTEX_FREE(a)   g_mutex_free (a)
#define MUTEX_LOCK(a)   if ((a) != NULL) g_mutex_lock (a)
#define MUTEX_UNLOCK(a) if ((a) != NULL) g_mutex_unlock (a)
#else
#define MUTEX_NEW()     NULL
#define MUTEX_FREE(a)
#define MUTEX_LOCK(a)
#define MUTEX_UNLOCK(a)
#endif

G_LOCK_DEFINE_STATIC, G_LOCK and G_UNLOCK in GLib are always safe to use, as they are already defined to be nothing when thread support is not available.

(Probably it would be a good idea to have something in the private GNOME VFS API that does this stuff for all the modules.)