SSH Project

There are four ways of setting up SSH-based copick projects:

Option 1: overlay-only

Using this method, there will be a single, remote project directory, all entities in the copick project will be writable. This is a good choice for when one wants to visualize and annotate data from a personal project that is stored on a remote server.

Get config template
Option 2: static/overlay

Using this method there will be two project directories, static and overlay, respectively. Files in the "static"-project directory will be read-only within copick, while files in the "overlay"-directory will be writeable. This is a good choice if you want to maintain the input data.

Get config template
Option 3: overlay/other static

Using this method, any user-created copick entities will be stored in a remote overlay, while static data (e.g. tomograms and baseline annotations) are stored in a different storage backend. This will be useful when one wants to curate a dataset and have the curated data directly accessible on a remote server.

Get config templates
Option 4: other overlay/static

Using this method, the static part of the project will be stored remotely, while new copick entities are written to another storage backend. This could be useful together with a group of people curating a large set of data.

Get config templates

Option 1: Setting up an overlay-only copick project

Set up your project root directory

In order to create an overlay-only-project we first need to set up one project directory on a remote server.

SSH authentication

Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.

In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands below.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

Create the config file

All information necessary to use the copick API is stored in a config file in json-Format. This has to be created before accessing the data is possible.

Create a json file with your favorite editor and paste the below template. Fill in the abolute path of the copick root directory on the remote server as indicated below.

vi copick_config.json

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating an overlay-only ssh-backend project.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "filesystem",
    "overlay_root": "ssh:///hpc/storage/copick_project/",
    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the overlay_fs_args-field will be passed to sshfs.SSHFileSystem.

More about overlay_fs_args ...

The username, host and port-fields are necessary to set up the SSH connection. Refer to the SSHFS documentation for detailed information.

An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands above.

{
  "overlay_fs_args": {
      "username": "user.name",
      "host": "localhost",
      "port": 2222
  }
}

The project is now set up, you can now add objects, add tomograms or store annotations.

Option 2: Setting up a static/overlay copick project

Set up your project root directory

In order to create an static/overlay-project we first need to set up two project directories, one to store the static data, and another to store any new or curated annotations.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project"
ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project_static"
# Replace port, user name and path to the project overlay with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project"
ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project_static"
# Replace port, user name and path to the project overlay with the correct values

Create the config file

All information necessary to use the copick API is stored in a config file in json-Format. This has to be created before accessing the data is possible.

Create a json file with your favorite editor and paste the below template. Fill in the abolute path of the copick root directory as indicated below.

vi copick_config.json

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating overlaying a ssh-backend ona project in a ssh-backend.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "filesystem",
    "overlay_root": "ssh:///hpc/storage/copick_project/",
    "static_root": "ssh:///hpc/storage/copick_project_static/",
    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    },
    "static_fs_args": {
        "username": "user.name",
        "password": "1234",
        "host": "hpc.example.com",
        "port": 22
    }
}

Option 3: SSH overlay, other static backend

Choose your static backend:

CZ cryoET Data PortalSMB ShareAWS S3Local

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating overlaying a ssh-backend on a CZ cryoET Data Portal dataset.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "cryoet_data_portal",
    "overlay_root": "ssh:///hpc/storage/copick_project/",
    "dataset_ids": [
        10301,
        10302
    ],
    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

Set up your overlay project

This directory will contain all newly created data for your project.

SSH authentication

Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.

In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands below.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the overlay_fs_args-field will be passed to sshfs.SSHFileSystem.

{
  "overlay_root": "ssh:///path/to/copick_project/",

    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

More about overlay_fs_args ...

The username, host and port-fields are necessary to set up the SSH connection. Refer to the SSHFS documentation for detailed information.

An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands above.

{
  "overlay_fs_args": {
      "username": "user.name",
      "host": "localhost",
      "port": 2222
  }
}

Set up your static project

In the case of CZ cryoET data portal datasets, setting up the static project is as easy as specifying one or multiple dataset IDs. The below example selects runs from datasets 10301 and 10302.

{
  "dataset_ids": [
    10301,
    10302
  ]
}

Configuration Type

When using the CZ cryoET data portal, the config_type-field should be set to cryoet-data-portal.

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating overlaying a ssh-backend ona project in a smb-backend.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "filesystem",
    "overlay_root": "ssh:///hpc/storage/copick_project/",
    "static_root": "smb:///shared_drive/copick_project_static/",
    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    },
    "static_fs_args": {
        "host": "192.158.1.38",
        "username": "user.name",
        "password": "1234",
        "temppath": "/shared_drive",
        "auto_mkdir": true
    }
}

Set up your overlay project

This directory will contain all newly created data for your project.

SSH authentication

Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.

In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands below.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the overlay_fs_args-field will be passed to sshfs.SSHFileSystem.

{
  "overlay_root": "ssh:///path/to/copick_project/",

    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

More about overlay_fs_args ...

The username, host and port-fields are necessary to set up the SSH connection. Refer to the SSHFS documentation for detailed information.

An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands above.

{
  "overlay_fs_args": {
      "username": "user.name",
      "host": "localhost",
      "port": 2222
  }
}

Set up your static project

This SMB-share will contain all static data for your project.

In the config file, the location should be passed to the static_root-field. Any arguments specified to the static_fs_args-field will be passed to SMBFileSystem.

{
    "static_root": "smb:///shared_drive/copick_project/",
    "static_fs_args": {
        "host": "192.158.1.38",
        "username": "user.name",
        "password": "1234",
        "temppath": "/shared_drive",
        "auto_mkdir": true,
    }
}

More about static_fs_args ...

The auto_mkdir-flag is necessary to create copick-directories if they do not yet exist. The tmpath-flag is not strictly necessary, this depends on your SMB setup (e.g. if only a specific directory is shared).

{
    "static_root": "smb:///shared_drive/copick_project/",
    "static_fs_args": {
        "host": "192.158.1.38",
        "username": "user.name",
        "password": "1234",
        "auto_mkdir": true,
    }
}

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating overlaying a ssh-backend ona project in a s3-backend.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "filesystem",
    "overlay_root": "ssh:///hpc/storage/copick_project/",
    "static_root": "s3://bucket/copick_project_static/",
    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    },
    "static_fs_args": {
        "profile": "your_profile"
    }
}

Set up your overlay project

This directory will contain all newly created data for your project.

SSH authentication

Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.

In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands below.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the overlay_fs_args-field will be passed to sshfs.SSHFileSystem.

{
  "overlay_root": "ssh:///path/to/copick_project/",

    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

More about overlay_fs_args ...

The username, host and port-fields are necessary to set up the SSH connection. Refer to the SSHFS documentation for detailed information.

An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands above.

{
  "overlay_fs_args": {
      "username": "user.name",
      "host": "localhost",
      "port": 2222
  }
}

Set up your static project

This S3 URI will contain all static data for your project.

Make sure the intended S3 bucket is writable:

echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project_static/test.txt
aws s3 ls s3://your-bucket-name/copick_project_static/
aws s3 rm s3://your-bucket-name/copick_project_static/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI

AWS authentication

Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.

In the config file, the location should be passed to the static_root-field. Any arguments specified to the static_fs_args-field will be passed to S3FileSystem. profile should be one of the profiles set up in your ~/.aws/credentials file.

{
  "static_root": "s3://bucket-name/copick_project_static/",
  "static_fs_args": {
        "profile": "example_profile"
    }
}

More about static_fs_args ...

Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation for detailed information.

For local MinIO buckets, the following config may be appropriate:

{
    "static_fs_args": {
        "key":"bucketkey",
        "secret":"bucketsecret",
        "endpoint_url":"http://10.30.121.49:7070",
        "client_kwargs":{
            "region_name":"us-east-1"
        }
}

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating overlaying a ssh-backend ona project in a local-backend.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "filesystem",
    "overlay_root": "ssh:///hpc/storage/copick_project/",
    "static_root": "local:///path/to/copick_project_static/",
    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    },
    "static_fs_args": {
        "auto_mkdir": true
    }
}

Set up your overlay project

This directory will contain all newly created data for your project.

SSH authentication

Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.

In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands below.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values

In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the overlay_fs_args-field will be passed to sshfs.SSHFileSystem.

{
  "overlay_root": "ssh:///path/to/copick_project/",

    "overlay_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

More about overlay_fs_args ...

The username, host and port-fields are necessary to set up the SSH connection. Refer to the SSHFS documentation for detailed information.

An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands above.

{
  "overlay_fs_args": {
      "username": "user.name",
      "host": "localhost",
      "port": 2222
  }
}

Set up your static project

This directory will contain all static data for your project.

Make sure it exists on the filesystem and is writable:

touch /path/to/copick_project_static
# Replace /path/to/copick_project with the path to your static project

If it does not yet exist, create it with the following command:

mkdir /path/to/copick_project_static
# Replace /path/to/copick_project with the path to your static project

In the config file, the location should be passed to the static_root-field. Any arguments specified to the static_fs_args-field will be passed to LocalFileSystem.

{
  "static_root": "local:///path/to/copick_project",
  "static_fs_args": {
    "auto_mkdir": true
  }
}

More about static_fs_args ...

The auto_mkdir-flag is necessary to create copick-directories if they do not yet exist.

Option 4: Other overlay, SSH static backend

Choose your overlay backend:

SMB ShareAWS S3Local

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating overlaying a smb-backend ona project in a ssh-backend.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "filesystem",
    "overlay_root": "smb:///shared_drive/copick_project/",
    "static_root": "ssh:///hpc/storage/copick_project_static/",
    "overlay_fs_args": {
        "host": "192.158.1.38",
        "username": "user.name",
        "password": "1234",
        "temppath": "/shared_drive",
        "auto_mkdir": true
    },
    "static_fs_args": {
        "username": "user.name",
        "password": "1234",
        "host": "hpc.example.com",
        "port": 22
    }
}

Set up your overlay project

This SMB-share will contain all newly created data for your project.

In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the overlay_fs_args-field will be passed to SMBFileSystem.

{
    "overlay_root": "smb:///shared_drive/copick_project/",
    "overlay_fs_args": {
        "host": "192.158.1.38",
        "username": "user.name",
        "password": "1234",
        "temppath": "/shared_drive",
        "auto_mkdir": true,
    }
}

More about overlay_fs_args ...

The auto_mkdir-flag is necessary to create copick-directories if they do not yet exist. The tmpath-flag is not strictly necessary, this depends on your SMB setup (e.g. if only a specific directory is shared).

{
    "overlay_root": "smb:///shared_drive/copick_project/",
    "overlay_fs_args": {
        "host": "192.158.1.38",
        "username": "user.name",
        "password": "1234",
        "auto_mkdir": true,
    }
}

Set up your static project

This directory will contain all static data for your project.

SSH authentication

Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.

In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands below.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project_static"
# Replace port, user name and path to the static project with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project_static"
# Replace port, user name and path to the static project with the correct values

In the config file, the location should be passed to the static_root-field. Any arguments specified to the static_fs_args-field will be passed to sshfs.SSHFileSystem.

{
  "static_root": "ssh:///path/to/copick_project_static/",

    "static_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

More about static_fs_args ...

The username, host and port-fields are necessary to set up the SSH connection. Refer to the SSHFS documentation for detailed information.

An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands above.

{
  "static_fs_args": {
      "username": "user.name",
      "host": "localhost",
      "port": 2222
  }
}

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating overlaying a s3-backend ona project in a ssh-backend.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "filesystem",
    "overlay_root": "s3://bucket/copick_project/",
    "static_root": "ssh:///hpc/storage/copick_project_static/",
    "overlay_fs_args": {
        "profile": "your_profile"
    },
    "static_fs_args": {
        "username": "user.name",
        "password": "1234",
        "host": "hpc.example.com",
        "port": 22
    }
}

Set up your overlay project

This S3 URI will contain all newly created data for your project.

Make sure the intended S3 bucket is writable:

echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project/test.txt
aws s3 ls s3://your-bucket-name/copick_project/
aws s3 rm s3://your-bucket-name/copick_project/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI

AWS authentication

Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.

In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the overlay_fs_args-field will be passed to S3FileSystem. profile should be one of the profiles set up in your ~/.aws/credentials file.

{
  "overlay_root": "s3://bucket-name/copick_project/",
  "overlay_fs_args": {
        "profile": "example_profile"
    }
}

More about overlay_fs_args ...

Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation for detailed information.

For local MinIO buckets, the following config may be appropriate:

{
    "overlay_fs_args": {
        "key":"bucketkey",
        "secret":"bucketsecret",
        "endpoint_url":"http://10.30.121.49:7070",
        "client_kwargs":{
            "region_name":"us-east-1"
        }
}

Set up your static project

This directory will contain all static data for your project.

SSH authentication

Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.

In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands below.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project_static"
# Replace port, user name and path to the static project with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project_static"
# Replace port, user name and path to the static project with the correct values

In the config file, the location should be passed to the static_root-field. Any arguments specified to the static_fs_args-field will be passed to sshfs.SSHFileSystem.

{
  "static_root": "ssh:///path/to/copick_project_static/",

    "static_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

More about static_fs_args ...

The username, host and port-fields are necessary to set up the SSH connection. Refer to the SSHFS documentation for detailed information.

An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands above.

{
  "static_fs_args": {
      "username": "user.name",
      "host": "localhost",
      "port": 2222
  }
}

Cofiguration Template

{
    "name": "Example Project",
    "description": "This is an example project, demonstrating overlaying a local-backend ona project in a ssh-backend.",
    "version": "0.5.0",
    "pickable_objects": [
        {
            "name": "ribosome",
            "is_particle": true,
            "label": 1,
            "color": [
                0,
                117,
                220,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "atpase",
            "is_particle": true,
            "label": 2,
            "color": [
                251,
                192,
                147,
                255
            ],
            "radius": 150.0
        },
        {
            "name": "membrane",
            "is_particle": false,
            "label": 3,
            "color": [
                200,
                200,
                200,
                255
            ],
            "radius": 10.0
        }
    ],
    "user_id": "example.user",
    "config_type": "filesystem",
    "overlay_root": "local:///path/to/copick_project/",
    "static_root": "ssh:///hpc/storage/copick_project_static/",
    "overlay_fs_args": {
        "auto_mkdir": true
    },
    "static_fs_args": {
        "username": "user.name",
        "password": "1234",
        "host": "hpc.example.com",
        "port": 22
    }
}

Set up your overlay project

This directory will contain all newly created data for your project.

Make sure it exists on the filesystem and is writable:

touch /path/to/copick_project
# Replace /path/to/copick_project with the path to your project overlay

If it does not yet exist, create it with the following command:

mkdir /path/to/copick_project
# Replace /path/to/copick_project with the path to your project overlay

In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the overlay_fs_args-field will be passed to LocalFileSystem.

{
  "overlay_root": "local:///path/to/copick_project",
  "overlay_fs_args": {
    "auto_mkdir": true
  }
}

More about overlay_fs_args ...

The auto_mkdir-flag is necessary to create copick-directories if they do not yet exist.

Set up your static project

This directory will contain all static data for your project.

SSH authentication

Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.

In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands below.

Make sure it exists on the remote filesystem and is writable:

ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project_static"
# Replace port, user name and path to the static project with the correct values

If it does not yet exist, create it with the following command:

ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project_static"
# Replace port, user name and path to the static project with the correct values

In the config file, the location should be passed to the static_root-field. Any arguments specified to the static_fs_args-field will be passed to sshfs.SSHFileSystem.

{
  "static_root": "ssh:///path/to/copick_project_static/",

    "static_fs_args": {
        "username": "user.name",
        "host": "hpc.example.com",
        "port": 22
    }
}

More about static_fs_args ...

The username, host and port-fields are necessary to set up the SSH connection. Refer to the SSHFS documentation for detailed information.

An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.

ssh -L 2222:localhost:22 user.name@hpc.example.com

and then use localhost:2222 as the host in the config and commands above.

{
  "static_fs_args": {
      "username": "user.name",
      "host": "localhost",
      "port": 2222
  }
}