Using Globus with Princeton Data Commons (PDC)

Contents

  1. How do I use Globus with PDC?
  2. Depositing Data Over 100mb
  3. How to Transfer Files from Research Computing Servers to PRDS Endpoint
  4. How to get a Globus Connect Personal Account to PRDS Endpoint
  5. How Transfer Files using your Globus Account 

 

 

Thank you for choosing to publish your data in Princeton Data Commons (PDC). All data sets over 100MB must be submitted to PDC by using a Globus account. 

To facilitate your data transfer, we will open a shared endpoint on our workserver. Using Globus, you will be able to transfer your files to us via this PRDS endpoint. Don’t forget to include your README! 

NOTE: Before transferring any data to our system, you will need to ensure that the files themselves you plan to transfer have given us read/write permissions.

How do I use Globus with PDC?

Globus provides helpful introductory information about how to log into Globus and transfer files on their website. 

Depositing Data over 100mb

For submissions over 100mb, the data curators will work with you to submit your data. To begin, indicate in the field provided on the submission webform where your files are located (e.g., on your local machine, on Research Computing clusters, on a Globus Personal Endpoint, etc.). 

Once you have completed your submission through the web-portal, a data curator will reach out to you with instructions on how to transfer your data to our PRDS Globus endpoint. The curator will create a share folder in the endpoint that will be accessible only to the depositing researcher. 

 

How to Transfer files from Research Computing Servers to PRDS Endpoint

  1. If you have never used Globus before, create a Globus account using your Princeton University credentials and sign in. (If you have used Globus before, proceed to step 2).
  2. Email prds@princeton.edu to notify curators that you are ready to transfer your data. 
  3. Wait to receive an email from Globus (no-reply@globus.org) with an invite from our department to a shared folder for you to deposit your data into in our PRDS endpoint.

Example text from email:

Globus user [someone from our department] shared the folder "[folder name]" on the endpoint "[endpoint name]" (endpoint id: [identification number]) with [your name and email] Princeton University identity.

Use this URL to access the folder: [link]

  1. Click on the URL in the email you received. This will open the Globus browser client.
  2. Make sure your Globus File Manager is set to two panels (an option in the upper right corner of your browser window). At this point, your screen should look similar to:
Image of Globus File Manager Page with the panel buttons highlighted in the upper right corner

The left panel in your window should show your deposit folder. This folder will be empty. The right panel will be blank, save for the text “Search for a collection to begin. Get started by taking a short tour.”

  1. In the search bar associated with the right panel, search for where your data is located. 
    1. If on a Princeton cluster, you can search for “Princeton TIGRESS,” “Princeton DELLA /scratch/gpfs,” “Princeton TIGRESS /projects,” and more. You will then find the correct folder (which may be based on your netID, your PI, or your lab/group name). 
    2. If your data is located elsewhere or on your personal machine, set up Globus Connect Personal.

NOTE: If you are looking for “Princeton TIGER,” you will navigate to the “Princeton TIGRESS” collection, move up one folder, and select the “tiger” folder. 

NOTE: Research Computing has instructions for how to begin your data transfer between Globus and Princeton-specific resources.

  1. Once you have navigated to the folder where your data is located, in the top banner on the right pane, click the “select all.” All contents should then be highlighted, with a checkmark next to each file.
Image of Globus File Manager Page with all files selected using "Select all" or "select none" radial button
  1. In the right pane, look for the button with an arrow that says “Start.” This will begin your data transfer into your deposit folder.
Image of Start Button
  1. Once your data transfer has been completed (you will receive an email from Globus), please notify our department (by emailing prds@princeton.edu) so we can begin the curation process.

 

How to get a Globus Connect Personal Account (Transfer files from your computer) to PRDS Endpoint

  1. If you have never used Globus before, create a Globus account using your Princeton University credentials and sign in.
  2. Then, install Globus Connect Personal onto your computer. This will be different depending on your operating system. 
  3. To set up your endpoint, you can follow the information provided in this video or follow the instructions provided for each type of operating system:
    1. Instructions for Mac OS X 
    2. Instructions for Windows
    3. Instructions for Linux 

Once the installation is complete, you may begin transferring files to or from your personal computer. 

  1. Reach out to PRDS (prds@princeton.edu) to let us know that your Globus Connect Personal account has been set up.
  2. Wait to receive an email from Globus (no-reply@globus.org) with an invite from our department to a shared folder for you to deposit your data into in our PRDS endpoint.

Example text from email:

Globus user [someone from our department] shared the folder "[folder name]" on the endpoint "[endpoint name]" (endpoint id: [identification number]) with [your name and email] Princeton University identity.

Use this URL to access the folder: [link]

  1. Click on the URL in the email you received. This will open the Globus browser client.

 

How to Transfer Files using your Globus Account 

  1. Login to your account in Globus (using your Princeton NetID). 
  2. Navigate to the File Transfer Tab (left sidebar) 
Image of Globus File Manager Page
  1. Open a second panel in the file transfer window (by clicking the middle panel icon)
Image of Globus File Manager Page with the panel buttons highlighted in the upper right corner
  1. Select the files you want to transfer on one side and the location you would like to deposit those files on the opposite side. Your transfer to your personal share folder in the PRDS endpoint will start when you click the “start” button below the path where the files are currently stored.
Image of Start Button

Downloading from PDC Using Globus

In order to download files to your computer, you will need to first set up Globus Connect Personal.  A free account is required for anyone wishing to access data via Globus, but it is often managed with SSO through the institution (as in Princeton's CAS).

 

If you are only interested in downloading a single file, you may do so through the web browser file manager. 

Download button in middle of File Manager Screen of Globus

Keep in mind that this will not work for downloading a folder or multiple files at the same time. If you need more than one file, you must set up a Globus Personal Endpoint, using Globus Connect Personal.