OSN Administrator¶
OSN Administrators are responsible for processing Coldfront Allocation Requests and for monitoring automated AMIE processing.
OSN Resources¶
OSN has extended the Coldfront application via the Coldfront plugin architecture and has added two OSN-specific resources to the application: the Open Storage Network resource and the OSN Bucket resource. These resources and their associated attributes are described in detail in the section Coldfront Projects, Allocations and Resources of the user documentation. Briefly, a network allocation gives a project the right to store a certain amount of data across all OSN pods and a bucket allocation corresponds to a single bucket created on a specific pod. Users/projects acquire a network allocation once then acquire one or more bucket allocations during the life of the project; each OSN Bucket resource allocation corresponds to a bucket on a pod in the network.
Processing Allocation Requests¶
Principal Investigator (PI) users are allowed to request resources for their projects using the Coldfront application. When a PI user creates a request in the application, an email is sent to a list of OSN administrators. To process the request, an OSN admin navigates to the “Allocation Requests” page using the Admin -> Allocation Requests navigation bar menu choice (the notification email also contains a link to this page as a convenience). This page displays a list of all outstanding resource requests. Each line in the list includes the name of a PI and the resource requested; using this information, an OSN admin can locate specific PI requests
The last column on each line contains two buttons “Approve” and “Details”. To begin approving an allocation request, an admin clicks the “Details” button which will show the Allocation Details page for the request.
Warning
Do not click the “Approve” button. Approving an OSN allocation without setting any attributes would create a “broken” allocation because the provisioning process needs allocation attributes to create resources (e.g. bucket provisioning requires a bucket name) If you attempt to approve an allocation without any resources, Coldfront will warn you before allowing you to proceed.
Edit Allocation Information¶
On the Allocation Details page, there are two sections: Allocation Information and Allocation Attributes. When processing a request it’s recommended to edit and update the information in the Allocation Information section first. In that section the OSN admin follows these steps:
Enter a start date
Enter an end date
Enter a description of the form <size> some brief info where <size> is the allocation size in TiB (e.g. 11TiB for Folding Images)
Click the “Update” button
DO NOT CHANGE STATUS YET, LEAVE AS “New”
DO NOT CLICK “APPROVE”
Adding Allocation Attributes¶
Each resource type has allocation attributes that are specific to the resource. Attributes added by OSN administrators are used by provisioning software when allocating storage. To add an attribute, the admin visits the “Allocation Attributes” section of the “Allocation Details” page and clicks the “Add Allocation Attribute” button. The resulting page displays a dropdown that allows the admin to select an attribute type and a text box to enter the value for the attribute. The admin adds as many attributes as are required for the specific allocation type being processed.
Warning
The application does not provide any “context help”; it will gladly let you assign “cpu hours” to a storage allocation and “Quota (TiB)” to a microscope. Unfortunately, many allocation attributes have similar names and can be confusing. Be careful when adding attributes that you select the correct types.
Activating the Allocation¶
Finally, once all the allocation attributes have been added and carefully reviewed, return to the Allocation Information section, change the status from “New” to “Active” and click “Update”. Once an new allocation request has been activated the provisioning process will read the allocation attributes and allocate the corresponding storage on the OSN network.
Processing an OSN Bucket Allocation Request¶
A project must have an Open Storage Network resource allocation before its PI can create buckets. This resource allocation is created automatically for AMIE-provisioned projects but not for non-AMIE projects.
Creating an Open Storage Network allocation for non-AMIE Users¶
For non-AMIE projects, an OSN administrator creates a network allocation on behalf of the project the first time the project makes an OSN Bucket resource request (provided that the project has been approved to access the network).
Note
The mechanism a non-AMIE project uses to request approval for network storage is not defined here and is expected to happen “out-of-band” via an OSN Help ticket with more formalized process TBD.
If a bucket allocation request is received for an approved project and no network allocation exists, the admin creates the request as follows:
Navigate to the project
Click on the “Request Resource Allocation” button in the “Allocations” section
Fill out the form, choosing the “Open Storage Network (Cloud)” resource option
Specify the number of TiB the allocation should receive.
This generates a network allocation request which is then processed like any other resource request.
Processing the Open Storage Network resource allocation request¶
Processing the Open Storage Network Resource allocation request requires setting one allocation attribute: the network quota. To do this the admin does the following:
Note/write-down the value shown on the “Quantity” line of “Allocation Information” section of the allocation request
Click the “Add Allocation Attribute” on the “Allocation Details” page
Select the “OSN Network Quota (TB)” option in the “Allocation Attribute type” drop-down
Enter the quantity value from the first step
Click the “Add” Button.
Bucket Allocation Request Processing¶
Processing a bucket allocation request requires setting three allocation attributes, they are:
- OSN Bucket Quota (TB)
This is the size of the bucket to be created. This information can be found on the “Quantity” line of “Allocation Information” section.
- OSN Bucket Name
This is the name of the bucket to be created; the requestor must supply this information in the justification textbox when filling out the request form. When processing the request, an admin can find this information on the “Justification” line in the “Allocation Information” section.
Warning
OSN requires that PIs follow the convention of providing bucket names in the justification text however, this is not programmatically enforced. If a PI omits required information in this or other cases, the admin will need to contact the PI outside of the application.
- OSN Bucket Pod
The OSN admin must add this attribute to tell the provisioning software where to create the bucket. The admin sets the value of this attribute to the name of a pod in the OSN network.
Note
The OSN plugin injects some javascript on this form to provide some assistance and validation when entering OSN pod names.
Text hints: Clicking (double clicking on some browsers) in the value text box will show a list of valid pod names.
Ugly but helpful bullet list: A bulleted list of valid osn pod names is also displayed below the text box; useful for copy/paste.
Client side validation: Attempting to submit the form with an invalid pod name will be blocked
Note that, unfortunately, YMMV with this assistance feature depending on your browser and the current mood of the javascript gods.
There is one optional, attribute that may be assigned to a bucket allocation:
- OSN Bucket Anonymous
Add this attribute with a value of “Yes” to allow anonymous read access to the bucket. Set this attribute to “No” to remove anonymous access. A PI can indicate they want anonymous access via the justification text.
Warning
Removing the anonymous attribute from an allocation where it was present and set to “Yes” will not change the anonymous access policy back to “No”. The provisioner runs when it detects an attribute that has changed since is last run. It will not detect that an attribute that was previously present is no longer present.
Bucket Provisioning¶
The bucket provisioner is implemented as a Django command (provision_buckets). The command looks for all active allocations and checks to see if any attributes have changed since the provisioner last processed the allocation. The provisioner timestamps the allocation with a (private) attribute named “OSN Bucket Processing Timestamp” whenever it processes a bucket allocation.
Forced Bucket Processing¶
An admin can force the provisioner to process an allocation by setting a dummy attribute named “OSN Pending” on the allocation. Adding this attribute and setting it to “Yes” signals to the provisioner that the allocation requires attention. The next time the provisioner runs, it will provision the allocation regardless of the value of the last processing timestamp.
AMIE Processing¶
The OSN Coldfront application integrates with the AMIE resource management platform. AMIE portal users can request OSN resources there and, if approved, commands are sent from AMIE to the OSN Coldfront application to provision access.
When an allocation is created in the AMIE portal, AMIE sends one “Request Project Create” (RPC) command to the OSN Coldfront application which creates a project and a network allocation. The RPC command is frequently followed by several AMIE “Request Account Create”(RAC) commands which create and add users to the project and allocation.
When OSN processes a RPC command, the OSN Coldfront application:
Creates a new project
Creates a user (or locates an existing user matching the provided ACCESS ID) to be the PI
Assigns the PI role to the user
Creates an Open Storage Network allocation for the project
Adds the PI user to the allocation
Once OSN notifies AMIE that the RPC command has been processed, AMIE typically sends one or more RAC commands which add users to the project and the allocation. When an RAC is received the OSN Coldfront application:
Creates a new user (or locates an existing user matching the provided ACCESS ID)
Adds the user to the project
Adds the user to allocation
The AMIE system is responsible for all changes to the entities created via the RPC and RAC commands, specifically the AMIE portal manages:
The Project
The Open Storage Network Allocation
The users that were created when processing AMIE RPC and RAC commands
Warning
It is possible to manage the project, allocation and users created through AMIE using the OSN Coldfront application. This should be avoided as these changes are not sent back to AMIE and therefore would create an inconsistent state between the AMIE portal and the OSN System.
AMIE Processing and Buckets¶
The AMIE system has no concept of buckets so, bucket management must be managed via the OSN Coldfront application. When RPC and RAC commands are processed and the associated users are created, those users are setup so that the are able to logon to the OSN Coldfront application using their ACCESS credentials (which are the same credentials they use to access the AMIE provisioning portal). A PI user visiting the OSN Coldfront application after AMIE provisioning would see that their project had been set up using the information that they provided to AMIE (e.g. project name, description, etc.). They would also see that an Open Storage Network allocation had been created with a quota attribute equal to the number of “Service Units” (an AMIE concept) of storage that they had requested. At that point, the PI would request additional OSN Bucket allocations just like a “non-AMIE” user. From an OSN admin standpoint, an OSN Bucket allocation request from an AMIE created user/project looks identical to one that came from a non-AMIE user; no special handling is required.
AMIE Processing State¶
AMIE processing is carried out via a series of JSON messages exchanged between AMIE and OSN. Transaction processing is implemented as a “pull” model where OSN periodically contacts the AMIE server and downloads packets for processing and sends back results as each packet is handled. Each time a new transaction packet is retrieved the OSN Coldfront application creates a transaction object in the Django database. The transaction object records the JSON content of the original request and tracks the state of the processing as the OSN Coldfront application processes the request. A typical transaction goes through the following states:
Received - This is the initial state of a transaction after it has been retrieved from AMIE
Processing - The packet has been dispatched to a processing code specific to the transaction type
Success - The packet has been processed and provisioning has succeeded in the OSN Coldfront application
Notified - A notification message has been successfully sent to the AMIE system
Complete - The transaction is complete
Exception States¶
There are two exception states. Transactions in these states will be ignored by the processing software:
Failed - A step in a transaction may fail permanently (e.g. bad or missing data in the AMIE packet), in which case the transaction will be put in the failed state. An operator must intervene to retry a failed transaction.
Ignore - An admin may use the OSN Coldfront application to set the status of a transaction to the ignore state. This is useful to temporarily prevent failing transactions from being retried when there is some transient issue present like a storage node being down for maintenance.
An operator can resume a failed or ignored transaction by setting its state to Continue in the OSN Coldfront application UI.
An admin can edit a transaction by selecting the Admin -> Coldfront Administration navbar menu choice then selecting the “Amie Transactions” link in the COLDFRONT_PLUGIN_OSN section of the admin screen. The resulting screen will show all transactions that are not in the Completed state. To see all transactions, including completed transactions, click the “Also show completed” filter link at the top right on the page.
AMIE Packet Contents¶
When debugging issues, it’s often helpful to see exactly what AMIE sent the OSN system. One can view the JSON content of the original AMIE packet that initiated an OSN Coldfront transaction by visiting the transaction in the administration UI (described above) and inspecting the “Packet” field; this contains the JSON of the original AMIE request.
Command Processing¶
There are two Django admin commands that are used to interact with AMIE and one that takes care of bucket provisioning. They are as follows:
amie_fetch_packets - This command contacts the AMIE system, retrieves available packets and creates corresponding transactions for those packets in the OSN Coldfront application database.
amie_process_packets - This command looks at all transactions, decides which ones can be processed then dispatches the packet to a command processor specific to the transaction type.
provision_buckets - This command looks at all active OSN Bucket allocations and creates new buckets if the allocation’s bucket does not exist or updates exiting ones.
These commands are run as kubernetes cronjobs every minute (incomplete jobs block new jobs from starting).
One can infer the state of processing from three sources:
Transaction state - Using the administrator interface, OSN admins can look at the outstanding transactions and determine if any of the transactions are in a failed state or have been in a processing state for too long; most transactions only require several seconds to execute so a transaction that has been in a processing state for many minutes is usually in trouble.
Kubernetes job state - One can inspect the state of the kubernetes jobs and associated pod logs to see whether they completed successfully.
OSN Command Log - All OSN commands log to a file on a permanent cluster volume that can be tailed using the following command (with the appropriate context set):
kubectl -n osncoldfront exec -it deploy/osncoldfront -- tail -f /var/log/osn/osncommands.log
Changing the Command Logging Level¶
Since commands are started in new pods on each cronjob run, changes to the configmap which controls logging level takes effect on the next run. The command logging level can be changed via the following (in the appropriate context):
kubectl -n osncoldfront edit configmap osncoldfront-configmap
and then editing the OSNCMD_LOG_LEVEL value (INFO is the default value DEBUG is the most verbose)
PI Status Upgrade Requests¶
Users who want to become PIs can request via the help desk or in the coldfront application. In the latter case, the user visits their profile and clicks the “Upgrade Account” button. This will send an email to help@osn.mghpcc.org. To service this request, an admin visits the Admin -> Coldfront Administration menu pick and from that navigates to “Coldfront Administration”. There, in the “User” section (very bottom of the page) click the “Profile” link. On the resulting page one can search for the user and edit their profile object. There is a single check box on the profile object page labeled “Is PI”; check this and save to upgrade the user’s status.