Data Portal

The filesystem implementation is a concrete implementation of the abstract copick API that reads data from the cryoET Data Portal and reads/writes data to/from any storage supported by fsspec. The filesystem implementation is defined in the copick.impl.cryoet_data_portal module.

Tilt series, alignments and CTF

A portal project serves the portal's tilt series and alignments as read-only copick entities, next to any written to its overlay:

Tilt series have the type raw (or aligned, for stacks of aligned images); several of one type are told apart with run.get_tilt_series_list("raw", portal_meta_query={"id": ...}). Images are read from the portal's OME-Zarr on S3. The metadata is built from the portal's records: the stage angle of each image (raw_angle), its acquisition order (from 0), dose and pre-exposure (Frame), and its CTF fit (PerSectionParameters; the phase shift is converted from radians to degrees). The portal records no amplitude contrast; 0.07 is assumed and marked as a default (amplitude_contrast_source). metadata.ctf_status tells whether all, some or none of the images have a CTF fit, or whether a fit has a non-positive defocus.
Alignments are named {method}-{type}-{portal id}, e.g. projectionMatching-local-18924. Their per-section parameters are converted from the portal's convention (pixel shifts, centres at N/2) to copick's (see Geometry); the reference volume is the portal's alignment volume. Of a local alignment only the rigid part is available. The status of an alignment:
- author: per-section parameters that fit the tilt series;
- portal_default: no per-section parameters, all fields at the portal's defaults. The portal creates such a placeholder when a deposition holds no alignment;
- no_per_section: no per-section parameters, but not a placeholder;
- inconsistent: parameters that do not fit the tilt series.
tilt_series.valid_alignments lists only author alignments, the portal's standard ones first. - Tomograms report the alignment the portal links them to (tomogram.alignment, tomogram.tilt_series). Linking a portal tomogram to an alignment of the overlay (tomogram.set_alignment(...)) takes precedence; removing that link restores the portal's.

These records are queried once for all configured datasets, on first access to a run's tilt series or alignments (not when the project is opened), and kept in the portal disk cache next to the annotation data (.copick_cache/portal_acq_*.json; see copick cache info and copick cache clear).

Metadata Models

copick.impl.cryoet_data_portal.CopickConfigCDP

Bases: CopickConfig

Data Entities

copick.impl.cryoet_data_portal.CopickRootCDP

Bases: CopickRoot

reconnect

reconnect() -> None

Force reconnection of the overlay filesystem and invalidate caches.

copick.impl.cryoet_data_portal.CopickObjectCDP

Bases: CopickObjectOverlay

copick.impl.cryoet_data_portal.CopickRunCDP

Bases: CopickRunOverlay

get_tilt_series_list

get_tilt_series_list(ts_type: Union[str, List[str]] = None, portal_meta_query: Dict[str, Any] = None, **kwargs) -> List[CopickTiltSeriesCDP]

Get tilt series by type. Portal tilt series are of type raw (or aligned for stacks of aligned images); several of one type are told apart by portal_meta_query.

Parameters:

ts_type (Union[str, List[str]], default: None ) –

Type(s) of the tilt series. Default: all.
portal_meta_query (Dict[str, Any], default: None ) –

Values to compare for equality against the fields of the portal's cryoet_data_portal.TiltSeries record (e.g. {"id": 17489}). Overlay tilt series never match a query.
**kwargs –

Additional parameters passed to the parent class.

Returns:

List[CopickTiltSeriesCDP] –

List[CopickTiltSeries]: The matching tilt series.

get_picks

get_picks(object_name: str = None, user_id: str = None, session_id: str = None, portal_meta_query: Dict[str, Any] = None, portal_author_query: List[str] = None, **kwargs) -> List[CopickPicksCDP]

Get picks by name, user_id or session_id (or combinations). Portal metadata are compared for equality. Portal authors are checked for inclusion in the full author list.

Parameters:

object_name (str, default: None ) –

Name of the object to search for.
user_id (str, default: None ) –

User ID to search for.
session_id (str, default: None ) –

Session ID to search for.
portal_meta_query (Dict[str, Any], default: None ) –

Dictionary of values to compare against portal metadata of this annotation. Allowed keys are the scalar fields of cryoet_data_portal.Annotation
portal_author_query (List[str], default: None ) –

List of author names. Segmentations are included if this author is in the portal annotation's author list.
**kwargs –

Additional parameters passed to parent class.

Returns:

List[CopickPicksCDP] –

List[CopickPicks]: List of picks that match the search criteria.

get_segmentations

get_segmentations(user_id: str = None, session_id: str = None, is_multilabel: bool = None, name: str = None, voxel_size: float = None, portal_meta_query: Dict[str, Any] = None, portal_author_query: List[str] = None, *, is_instance: Optional[bool] = False, is_panoptic: Optional[bool] = False, **kwargs) -> List[CopickSegmentationCDP]

Get segmentations by user_id, session_id, name, type or voxel_size (or combinations) and portal metadata and authors. Portal metadata are compared for equality. Portal authors are checked for inclusion in the full author list.

Parameters:

user_id (str, default: None ) –

User ID to search for.
session_id (str, default: None ) –

Session ID to search for.
is_multilabel (bool, default: None ) –

Whether the segmentation is multilabel or not.
name (str, default: None ) –

Name of the segmentation to search for.
voxel_size (float, default: None ) –

Voxel size to search for.
portal_meta_query (Dict[str, Any], default: None ) –

Dictionary of values to compare against portal metadata of this annotation. Allowed keys are the scalar fields of cryoet_data_portal.Annotation
portal_author_query (List[str], default: None ) –

List of author names. Segmentations are included if this author is in the portal annotation's author list.
is_instance (Optional[bool], default: False ) –

Whether to select instance segmentations (True) or not (False, the default); None selects any type.
is_panoptic (Optional[bool], default: False ) –

Whether to select panoptic segmentations (True) or not (False, the default); None selects any type.
**kwargs –

Additional parameters passed to parent class.

Returns:

List[CopickSegmentationCDP] –

List[CopickSegmentation]: List of segmentations that match the search criteria.

ensure

ensure(create: bool = False) -> bool

Checks if the run record exists in the static or overlay directory, optionally creating it in the overlay filesystem if it does not.

Parameters:

create (bool, default: False ) –

Whether to create the run record if it does not exist.

Returns:

bool ( bool ) –

True if the run record exists, False otherwise.

copick.impl.cryoet_data_portal.CopickPicksCDP

Bases: CopickPicksOverlay

copick.impl.cryoet_data_portal.CopickMeshCDP

Bases: CopickMeshOverlay

copick.impl.cryoet_data_portal.CopickSegmentationCDP

Bases: CopickSegmentationOverlay

copick.impl.cryoet_data_portal.CopickVoxelSpacingCDP

Bases: CopickVoxelSpacingOverlay

ensure

ensure(create: bool = False) -> bool

Checks if the voxel spacing record exists in the static or overlay directory, optionally creating it in the overlay filesystem if it does not.

Parameters:

create (bool, default: False ) –

Whether to create the voxel spacing record if it does not exist.

Returns:

bool ( bool ) –

True if the voxel spacing record exists, False otherwise.

get_tomograms

get_tomograms(tomo_type: str = None, portal_meta_query: Dict[str, Any] = None, portal_author_query: List[str] = None, **kwargs) -> List[CopickTomogramCDP]

Get a tomogram by type. Portal metadata are compared for equality. Authors are compared for inclusion.

Parameters:

tomo_type (str, default: None ) –

The type of tomogram to get. For portal tomograms, this is cryoet_data_portal.Tomogram.reconstruction_method.
portal_meta_query (Dict[str, Any], default: None ) –

Dictionary of values to compare against portal metadata of this tomogram. Allowed keys are the scalar fields of cryoet_data_portal.Tomogram
portal_author_query (List[str], default: None ) –

List of author names. Tomograms are included if this author is in the portal annotation's author list.
**kwargs –

Additional parameters passed to parent class.

Returns:

List[CopickTomogramCDP] –

List[CopickTomogram]: The list of tomograms that match the query.

copick.impl.cryoet_data_portal.CopickTiltSeriesCDP

Bases: CopickTiltSeriesOverlay

A tilt series of the CryoET Data Portal (read-only, raw or aligned) or of the project's overlay.

portal_tilt_series

portal_tilt_series: bool

Whether this tilt series is from the portal.

shape

shape: Optional[Tuple[int, int, int]]

Shape (section, y, x) of the full-size images; of a portal tilt series from its record, without opening the images.

copick.impl.cryoet_data_portal.CopickAlignmentCDP

Bases: CopickAlignmentOverlay

An alignment of the CryoET Data Portal (read-only), converted to copick's conventions.

copick.impl.cryoet_data_portal.CopickTomogramCDP

Bases: CopickTomogramOverlay

tomo_type

tomo_type: str

The type of tomogram. For data portal tomograms, this is derived as cryoet_data_portal.Tomogram.reconstruction_method + "-" + cryoet_data_portal.Tomogram.processing + ["-" + cryoet_data_portal.Tomogram.processing_software + "-" + cryoet_data_portal.Tomogram.ctf_corrected], where cryoet_data_portal.Tomogram.processing_software and cryoet_data_portal.Tomogram.ctf_corrected are discarded if null in the database.

portal_tomo

portal_tomo: bool

Whether this tomogram is from the portal or not.

portal_alignment_id

portal_alignment_id: Optional[int]

CryoET Data Portal identifier of the alignment this portal tomogram was reconstructed with.

copick.impl.cryoet_data_portal.CopickFeaturesCDP

Bases: CopickFeaturesOverlay