AWS S3 Project
There are four ways of setting up local copick projects via AWS or local S3 buckets (e.g. MinIO):
-
Option 1: overlay-only
Using this method, there will be a single, shared project S3-prefix, all entities in the copick project will be writable.
-
Option 2: static/overlay
Using this method there will be two S3-prefixes, static and overlay, respectively. Files with the "static"-project prefixes will be read-only within copick, while files with the "overlay"-prefixes will be writeable. This is a good choice if you want to maintain the input data.
-
Option 3: overlay/other static
Using this method, any user-created copick entities will be stored in a shared overlay, while static data (e.g. tomograms and baseline annotations) are stored in a different storage backend. This will be useful when multiple users are curating a dataset and have access to local copies of the static data.
-
Option 4: other overlay/static
Using this method, the static part of the project will be stored in a shared directory, while new copick entities are written to another storage backend. This could be useful when curating a large set of data with a group.
Option 1: Setting up an overlay-only copick project
Set up your project root prefix
The following example assumes that you have a shared bucket on AWS S3 or a local S3 instance that you can access. To test access:
```bash
echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project/test.txt
aws s3 ls s3://your-bucket-name/copick_project/
aws s3 rm s3://your-bucket-name/copick_project/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI
AWS authentication
Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.
Create the config file
All information necessary to use the copick API is stored in a config file in json-Format. This has to be created
before accessing the data is possible.
Create a json file with your favorite editor and paste the below template. Fill in the S3 URI of the copick root prefix as indicated below.
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating an overlay-only s3-backend project.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "filesystem",
"overlay_root": "s3://bucket/copick_project/",
"overlay_fs_args": {
"profile": "your_profile"
}
}
The project is now set up, you can now add objects, add tomograms or store annotations.
More about overlay_fs_args ...
Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation
for detailed information.
For local MinIO buckets, the following config may be appropriate:
Option 2: Setting up a static/overlay copick project
Set up your project root prefix
The following example assumes that you have a shared bucket on AWS S3 or a local S3 instance that you can access. Test access as above.
Create the config file
All information necessary to use the copick API is stored in a config file in json-Format. This has to be created
before accessing the data is possible.
Create a json file with your favorite editor and paste the below template. Fill in the S3 URI of the copick root as indicated below.
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating overlaying a s3-backend ona project in a s3-backend.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "filesystem",
"overlay_root": "s3://bucket/copick_project/",
"static_root": "s3://bucket/copick_project_static/",
"overlay_fs_args": {
"profile": "your_profile"
},
"static_fs_args": {
"profile": "your_profile"
}
}
Option 3: S3 overlay, other static backend
Choose your static backend:
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating overlaying a s3-backend on a CZ cryoET Data Portal dataset.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "cryoet_data_portal",
"overlay_root": "s3://bucket/copick_project/",
"dataset_ids": [
10301,
10302
],
"overlay_fs_args": {
"profile": "your_profile"
}
}
Set up your overlay project
This S3 URI will contain all newly created data for your project.
Make sure the intended S3 bucket is writable:
echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project/test.txt
aws s3 ls s3://your-bucket-name/copick_project/
aws s3 rm s3://your-bucket-name/copick_project/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI
AWS authentication
Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.
In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the
overlay_fs_args-field will be passed to S3FileSystem.
profile should be one of the profiles set up in your ~/.aws/credentials file.
{
"overlay_root": "s3://bucket-name/copick_project/",
"overlay_fs_args": {
"profile": "example_profile"
}
}
More about overlay_fs_args ...
Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation
for detailed information.
For local MinIO buckets, the following config may be appropriate:
Set up your static project
In the case of CZ cryoET data portal datasets, setting up the static project is as easy as specifying one or multiple dataset IDs. The below example selects runs from datasets 10301 and 10302.
Configuration Type
When using the CZ cryoET data portal, the config_type-field should be set to cryoet-data-portal.
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating overlaying a s3-backend ona project in a smb-backend.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "filesystem",
"overlay_root": "s3://bucket/copick_project/",
"static_root": "smb:///shared_drive/copick_project_static/",
"overlay_fs_args": {
"profile": "your_profile"
},
"static_fs_args": {
"host": "192.158.1.38",
"username": "user.name",
"password": "1234",
"temppath": "/shared_drive",
"auto_mkdir": true
}
}
Set up your overlay project
This S3 URI will contain all newly created data for your project.
Make sure the intended S3 bucket is writable:
echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project/test.txt
aws s3 ls s3://your-bucket-name/copick_project/
aws s3 rm s3://your-bucket-name/copick_project/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI
AWS authentication
Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.
In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the
overlay_fs_args-field will be passed to S3FileSystem.
profile should be one of the profiles set up in your ~/.aws/credentials file.
{
"overlay_root": "s3://bucket-name/copick_project/",
"overlay_fs_args": {
"profile": "example_profile"
}
}
More about overlay_fs_args ...
Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation
for detailed information.
For local MinIO buckets, the following config may be appropriate:
Set up your static project
This SMB-share will contain all static data for your project.
In the config file, the location should be passed to the static_root-field. Any arguments specified to the
static_fs_args-field will be passed to SMBFileSystem.
{
"static_root": "smb:///shared_drive/copick_project/",
"static_fs_args": {
"host": "192.158.1.38",
"username": "user.name",
"password": "1234",
"temppath": "/shared_drive",
"auto_mkdir": true,
}
}
More about static_fs_args ...
The auto_mkdir-flag is necessary to create copick-directories if they do not yet exist. The tmpath-flag is not
strictly necessary, this depends on your SMB setup (e.g. if only a specific directory is shared).
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating overlaying a s3-backend ona project in a local-backend.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "filesystem",
"overlay_root": "s3://bucket/copick_project/",
"static_root": "local:///path/to/copick_project_static/",
"overlay_fs_args": {
"profile": "your_profile"
},
"static_fs_args": {
"auto_mkdir": true
}
}
Set up your overlay project
This S3 URI will contain all newly created data for your project.
Make sure the intended S3 bucket is writable:
echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project/test.txt
aws s3 ls s3://your-bucket-name/copick_project/
aws s3 rm s3://your-bucket-name/copick_project/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI
AWS authentication
Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.
In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the
overlay_fs_args-field will be passed to S3FileSystem.
profile should be one of the profiles set up in your ~/.aws/credentials file.
{
"overlay_root": "s3://bucket-name/copick_project/",
"overlay_fs_args": {
"profile": "example_profile"
}
}
More about overlay_fs_args ...
Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation
for detailed information.
For local MinIO buckets, the following config may be appropriate:
Set up your static project
This directory will contain all static data for your project.
Make sure it exists on the filesystem and is writable:
touch /path/to/copick_project_static
# Replace /path/to/copick_project with the path to your static project
If it does not yet exist, create it with the following command:
mkdir /path/to/copick_project_static
# Replace /path/to/copick_project with the path to your static project
In the config file, the location should be passed to the static_root-field. Any arguments specified to the
static_fs_args-field will be passed to LocalFileSystem.
More about static_fs_args ...
The auto_mkdir-flag is necessary to create copick-directories if they do not yet exist.
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating overlaying a s3-backend ona project in a ssh-backend.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "filesystem",
"overlay_root": "s3://bucket/copick_project/",
"static_root": "ssh:///hpc/storage/copick_project_static/",
"overlay_fs_args": {
"profile": "your_profile"
},
"static_fs_args": {
"username": "user.name",
"password": "1234",
"host": "hpc.example.com",
"port": 22
}
}
Set up your overlay project
This S3 URI will contain all newly created data for your project.
Make sure the intended S3 bucket is writable:
echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project/test.txt
aws s3 ls s3://your-bucket-name/copick_project/
aws s3 rm s3://your-bucket-name/copick_project/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI
AWS authentication
Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.
In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the
overlay_fs_args-field will be passed to S3FileSystem.
profile should be one of the profiles set up in your ~/.aws/credentials file.
{
"overlay_root": "s3://bucket-name/copick_project/",
"overlay_fs_args": {
"profile": "example_profile"
}
}
More about overlay_fs_args ...
Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation
for detailed information.
For local MinIO buckets, the following config may be appropriate:
Set up your static project
This directory will contain all static data for your project.
SSH authentication
Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.
In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.
and then uselocalhost:2222 as the host in the config and commands below.
Make sure it exists on the remote filesystem and is writable:
ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project_static"
# Replace port, user name and path to the static project with the correct values
If it does not yet exist, create it with the following command:
ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project_static"
# Replace port, user name and path to the static project with the correct values
In the config file, the location should be passed to the static_root-field. Any arguments specified to the
static_fs_args-field will be passed to sshfs.SSHFileSystem.
{
"static_root": "ssh:///path/to/copick_project_static/",
"static_fs_args": {
"username": "user.name",
"host": "hpc.example.com",
"port": 22
}
}
More about static_fs_args ...
The username, host and port-fields are necessary to set up the SSH connection. Refer to the
SSHFS documentation for detailed information.
An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.
and then use localhost:2222 as the host in the config and commands above.
Option 4: Other overlay, S3 static backend
Choose your overlay backend:
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating overlaying a local-backend ona project in a s3-backend.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "filesystem",
"overlay_root": "local:///path/to/copick_project/",
"static_root": "s3://bucket/copick_project_static/",
"overlay_fs_args": {
"auto_mkdir": true
},
"static_fs_args": {
"profile": "your_profile"
}
}
Set up your overlay project
This directory will contain all newly created data for your project.
Make sure it exists on the filesystem and is writable:
touch /path/to/copick_project
# Replace /path/to/copick_project with the path to your project overlay
If it does not yet exist, create it with the following command:
mkdir /path/to/copick_project
# Replace /path/to/copick_project with the path to your project overlay
In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the
overlay_fs_args-field will be passed to LocalFileSystem.
More about overlay_fs_args ...
The auto_mkdir-flag is necessary to create copick-directories if they do not yet exist.
Set up your static project
This S3 URI will contain all static data for your project.
Make sure the intended S3 bucket is writable:
echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project_static/test.txt
aws s3 ls s3://your-bucket-name/copick_project_static/
aws s3 rm s3://your-bucket-name/copick_project_static/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI
AWS authentication
Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.
In the config file, the location should be passed to the static_root-field. Any arguments specified to the
static_fs_args-field will be passed to S3FileSystem.
profile should be one of the profiles set up in your ~/.aws/credentials file.
{
"static_root": "s3://bucket-name/copick_project_static/",
"static_fs_args": {
"profile": "example_profile"
}
}
More about static_fs_args ...
Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation
for detailed information.
For local MinIO buckets, the following config may be appropriate:
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating overlaying a smb-backend ona project in a s3-backend.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "filesystem",
"overlay_root": "smb:///shared_drive/copick_project/",
"static_root": "s3://bucket/copick_project_static/",
"overlay_fs_args": {
"host": "192.158.1.38",
"username": "user.name",
"password": "1234",
"temppath": "/shared_drive",
"auto_mkdir": true
},
"static_fs_args": {
"profile": "your_profile"
}
}
Set up your overlay project
This SMB-share will contain all newly created data for your project.
In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the
overlay_fs_args-field will be passed to SMBFileSystem.
{
"overlay_root": "smb:///shared_drive/copick_project/",
"overlay_fs_args": {
"host": "192.158.1.38",
"username": "user.name",
"password": "1234",
"temppath": "/shared_drive",
"auto_mkdir": true,
}
}
More about overlay_fs_args ...
The auto_mkdir-flag is necessary to create copick-directories if they do not yet exist. The tmpath-flag is not
strictly necessary, this depends on your SMB setup (e.g. if only a specific directory is shared).
Set up your static project
This S3 URI will contain all static data for your project.
Make sure the intended S3 bucket is writable:
echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project_static/test.txt
aws s3 ls s3://your-bucket-name/copick_project_static/
aws s3 rm s3://your-bucket-name/copick_project_static/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI
AWS authentication
Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.
In the config file, the location should be passed to the static_root-field. Any arguments specified to the
static_fs_args-field will be passed to S3FileSystem.
profile should be one of the profiles set up in your ~/.aws/credentials file.
{
"static_root": "s3://bucket-name/copick_project_static/",
"static_fs_args": {
"profile": "example_profile"
}
}
More about static_fs_args ...
Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation
for detailed information.
For local MinIO buckets, the following config may be appropriate:
Cofiguration Template
{
"name": "Example Project",
"description": "This is an example project, demonstrating overlaying a ssh-backend ona project in a s3-backend.",
"version": "0.5.0",
"pickable_objects": [
{
"name": "ribosome",
"is_particle": true,
"label": 1,
"color": [
0,
117,
220,
255
],
"radius": 150.0
},
{
"name": "atpase",
"is_particle": true,
"label": 2,
"color": [
251,
192,
147,
255
],
"radius": 150.0
},
{
"name": "membrane",
"is_particle": false,
"label": 3,
"color": [
200,
200,
200,
255
],
"radius": 10.0
}
],
"user_id": "example.user",
"config_type": "filesystem",
"overlay_root": "ssh:///hpc/storage/copick_project/",
"static_root": "s3://bucket/copick_project_static/",
"overlay_fs_args": {
"username": "user.name",
"host": "hpc.example.com",
"port": 22
},
"static_fs_args": {
"profile": "your_profile"
}
}
Set up your overlay project
This directory will contain all newly created data for your project.
SSH authentication
Copick will work best via SSH if you have set up passwordless SSH authentication. Refer to the SSH documentation for more information. In general, adding plain text passwords into copick configuration files is strongly discouraged.
In cases of mandatory 2-FA authentication, you may need to set up an SSH tunnel to the remote filesystem, e.g.
and then uselocalhost:2222 as the host in the config and commands below.
Make sure it exists on the remote filesystem and is writable:
ssh -p 22 user.name@hpc.example.com "touch /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values
If it does not yet exist, create it with the following command:
ssh -p 22 user.name@hpc.example.com "mkdir /path/to/copick_project"
# Replace port, user name and path to the project overlay with the correct values
In the config file, the location should be passed to the overlay_root-field. Any arguments specified to the
overlay_fs_args-field will be passed to sshfs.SSHFileSystem.
{
"overlay_root": "ssh:///path/to/copick_project/",
"overlay_fs_args": {
"username": "user.name",
"host": "hpc.example.com",
"port": 22
}
}
More about overlay_fs_args ...
The username, host and port-fields are necessary to set up the SSH connection. Refer to the
SSHFS documentation for detailed information.
An easy way to use the SSH filesystem is to tunnel to the remote filesystem via SSH, e.g.
and then use localhost:2222 as the host in the config and commands above.
Set up your static project
This S3 URI will contain all static data for your project.
Make sure the intended S3 bucket is writable:
echo "Hello, World!" > test.txt
aws s3 cp test.txt s3://your-bucket-name/copick_project_static/test.txt
aws s3 ls s3://your-bucket-name/copick_project_static/
aws s3 rm s3://your-bucket-name/copick_project_static/test.txt
# Replace s3://your-bucket-name/copick_project/ with your S3 URI
AWS authentication
Make sure you have the necessary AWS credentials set up and available in the shell you're running the above commands in. Refer to the AWS CLI documentation for more information.
In the config file, the location should be passed to the static_root-field. Any arguments specified to the
static_fs_args-field will be passed to S3FileSystem.
profile should be one of the profiles set up in your ~/.aws/credentials file.
{
"static_root": "s3://bucket-name/copick_project_static/",
"static_fs_args": {
"profile": "example_profile"
}
}
More about static_fs_args ...
Specifying profile is one possible way of setting up AWS credentials. Refer to the S3FS documentation
for detailed information.
For local MinIO buckets, the following config may be appropriate: