tardis.apps.hsm package

Subpackages

• tardis.apps.hsm.migrations package
  • Submodules
  • tardis.apps.hsm.migrations.0001_initial_data module
    • Migration
      • Migration.dependencies
      • Migration.operations
    • forward_func()
    • reverse_func()
  • Module contents
• tardis.apps.hsm.tests package
  • Submodules
  • tardis.apps.hsm.tests.test_api module
    • HsmAppApiTestCase
      • HsmAppApiTestCase.setUp()
      • HsmAppApiTestCase.tearDown()
      • HsmAppApiTestCase.test_ds_check()
      • HsmAppApiTestCase.test_online_check_unsupported_storage_class()
      • HsmAppApiTestCase.test_online_check_unverified_file()
      • HsmAppApiTestCase.test_online_check_valid_storage_class()
      • HsmAppApiTestCase.test_online_check_with_bad_password()
      • HsmAppApiTestCase.test_online_check_without_acl_access_to_dfo()
      • HsmAppApiTestCase.test_online_count()
      • HsmAppApiTestCase.test_recall()
      • HsmAppApiTestCase.test_stat_subprocess()
  • tardis.apps.hsm.tests.test_app_config module
    • HsmAppConfigTestCase
      • HsmAppConfigTestCase.test_app_config()
  • tardis.apps.hsm.tests.test_email_templates module
    • HsmAppEmailTemplateTestCase
      • HsmAppEmailTemplateTestCase.setUp()
      • HsmAppEmailTemplateTestCase.test_recall_complete_email()
      • HsmAppEmailTemplateTestCase.test_recall_failed_email()
  • tardis.apps.hsm.tests.test_migrations module
    • HsmAppMigrationTestCase
      • HsmAppMigrationTestCase.connection
      • HsmAppMigrationTestCase.test_migration()
  • Module contents

Submodules

tardis.apps.hsm.api module

Additions to MyTardis’s REST API

class tardis.apps.hsm.api.DatasetAppResource(api_name=None)

Bases: DatasetResource

Extends MyTardis’s API for Datasets, adding in a method to count online files in a Hierarchical Storage Management (HSM) system

class Meta

Bases: Meta

authorization

Authorisation class for Tastypie.

queryset

resource_name = 'dataset'

base_fields = {'created_time': <tastypie.fields.DateTimeField object>, 'description': <tastypie.fields.CharField object>, 'directory': <tastypie.fields.CharField object>, 'experiments': <tastypie.fields.ToManyField object>, 'id': <tastypie.fields.IntegerField object>, 'immutable': <tastypie.fields.BooleanField object>, 'instrument': <tastypie.fields.ForeignKey object>, 'modified_time': <tastypie.fields.DateTimeField object>, 'parameter_sets': <tastypie.fields.ToManyField object>, 'resource_uri': <tastypie.fields.CharField object>}

dataset_recall(request, **kwargs)

Send and email to Site admin to recall Dataset from HSM system

declared_fields = {}

online_count(request, **kwargs)

Return the number of online files and the total number of files in a dataset stored in a Hierarchical Storage Management (HSM) system

prepend_urls()

A hook for adding your own URLs or matching before the default URLs.

class tardis.apps.hsm.api.ReplicaAppResource(api_name=None)

Bases: ReplicaResource

Extends MyTardis’s API for DFOs, adding in a recall method and an online check method for files in a Hierarchical Storage Management (HSM) system

class Meta

Bases: Meta

authorization

Authorisation class for Tastypie.

queryset

resource_name = 'replica'

base_fields = {'created_time': <tastypie.fields.DateTimeField object>, 'datafile': <tastypie.fields.ForeignKey object>, 'id': <tastypie.fields.IntegerField object>, 'last_verified_time': <tastypie.fields.DateTimeField object>, 'resource_uri': <tastypie.fields.CharField object>, 'uri': <tastypie.fields.CharField object>, 'verified': <tastypie.fields.BooleanField object>}

declared_fields = {}

dfo_is_online(request, **kwargs)

Return the online status of a DataFileObject stored in a Hierarchical Storage Management (HSM) system

prepend_urls()

A hook for adding your own URLs or matching before the default URLs.

recall_dfo(request, **kwargs)

Recall archived DataFileObject from HSM system

tardis.apps.hsm.apps module

class tardis.apps.hsm.apps.HsmConfig(app_name, app_module)

Bases: AppConfig

name = 'tardis.apps.hsm'

tardis.apps.hsm.check module

HSM check module. Method for detecting whether a MyTardis DataFileObject in Hierarchical Storage Management is online or offline (on tape).

tardis.apps.hsm.check.dataset_online_count(dataset)

Checks how many of a dataset’s files are online

Parameters

datasetDataset

The Dataset for which to check the status

Returns

int

The number of online files in this dataset

tardis.apps.hsm.check.dfo_online(dfo)

Checks whether the underlying file of a DataFileObject is online

Parameters

dfoDataFileObject

The DataFileObject for which to check the status

Returns

bool

Status for whether dfo is online.

Raises

DataFileObjectNotVerified

If dfo is unverified

StorageClassNotSupportedError

If the django_storage_class for the StorageBox of the input DataFileObject is not supported

tardis.apps.hsm.default_settings module

tardis.apps.hsm.default_settings.HSM_MAX_INODE_FILE_SIZE = 384

The maximum size of files that can be stored within the inode (with stat reporting 0 blocks).

To determine the right value for your filesystem, you can create some small test files and check how small they have to be to have stat report 0 blocks.

https://en.wikipedia.org/wiki/Inode#Inlining

tardis.apps.hsm.email_text module

tardis.apps.hsm.email_text.email_dataset_recall_requested(dataset, user)

tardis.apps.hsm.email_text.email_dfo_recall_complete(dfo, user)

tardis.apps.hsm.email_text.email_dfo_recall_failed(dfo, user)

tardis.apps.hsm.email_text.email_dfo_recall_requested(dfo_id, user)

tardis.apps.hsm.email_text.interpolate_template(template_name, **kwargs)

tardis.apps.hsm.exceptions module

Exceptions related to Hierarchical Storage Management (HSM)

exception tardis.apps.hsm.exceptions.DataFileObjectNotVerified

Bases: HsmException

Exception raied when an operation is attempted on an unverified DataFile

exception tardis.apps.hsm.exceptions.HsmException

Bases: Exception

Base class for other exceptions to inherit from

exception tardis.apps.hsm.exceptions.StorageClassNotSupportedError

Bases: HsmException

Exception raised when a storage class is not supported

tardis.apps.hsm.storage module

class tardis.apps.hsm.storage.HsmFileSystemStorage(*args, **kwargs)

Bases: FileSystemStorage

This storage class is used to describe Hierarchical Storage Management filesystems where files may be offline (only or tape), but can be recalled to disk on demand

tardis.apps.hsm.tasks module

Tasks for recalling data from tape in Hierarchical Storage Management (HSM) systems

tardis.apps.hsm.urls module

Minimal urls.py, so we can do a reverse lookup for the ‘hsm_api_download_dfo’ URL pattern.

‘hsm_api_download_dfo’ is defined in the prepend_urls method of the ReplicaAppResource class in api.py

The API endpoint defined in this app is mapped to a URL in tardis/urls/api.py (along with API endpoints defined by other MyTardis apps).

tardis.apps.hsm.utils module

HSM utils module. Utilities for detecting whether files in Hierarchical Storage Management are online or offline (on tape).

tardis.apps.hsm.utils.file_is_online(path)

Detects whether a file is online or offline (on tape).

Parameters

pathstr

Path to the file for which we want to determine online/offline status.

Returns

bool

specifies whether the file in online i.e., not on tape.

Module contents