elektronn2.data.cnndata module

class elektronn2.data.cnndata.AgentData(input_node, side_target_node, path_prefix=None, raw_files=None, skel_files=None, vec_files=None, valid_skels=None, target_vec_ix=None, target_discrete_ix=None, abs_offset=None, aniso_factor=2)[source]

Bases: elektronn2.data.cnndata.BatchCreatorImage

Load raw_cube, vec_prob_obj_cube and skelfiles + rel.offset

get_newslice(position_l, direction_il, batch_size=1, source='train', aniso=True, z_shift=0, gamma=0, grey_augment_channels=None, r_max_scale=0.9, tracing_dir_prior_c=0.5, force_dense=False, flatfield_p=0.001, scale=1.0, last_ch_max_interp=False)[source]
getbatch(batch_size=1, source='train', aniso=True, z_shift=0, gamma=0, grey_augment_channels=None, r_max_scale=0.9, tracing_dir_prior_c=0.5, force_dense=False, flatfield_p=0.001)[source]

Prepares a batch by randomly sampling, shifting and augmenting patches from the data

Parameters:
  • batch_size (int) – Number of examples in batch (for CNNs often just 1)
  • source (str) – Data set to draw data from: ‘train’/’valid’
  • grey_augment_channels (list) – List of channel indices to apply grey-value augmentation to
  • warp (bool or float) – Whether warping/distortion augmentations are applied to examples (slow –> use multiprocessing). If this is a float number, warping is applied to this fraction of examples e.g. 0.5 –> every other example.
  • warp_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.transformations.get_warped_slice()
  • ignore_thresh (float) – If the fraction of negative targets in an example patch exceeds this threshold, this example is discarded (Negative targets are ignored for training [but could be used for unsupervised target propagation]).
  • force_dense (bool) – If True the targets are not sub-sampled according to the CNN output strides. Dense targets requires MFP in the CNN!
  • affinities –
  • nhood_targets –
  • ret_ll_mask (bool) – If True additional information for reach batch example is returned. Currently implemented are two ll_mask arrays to indicate the targeting mode. The first dimension of those arrays is the batch_size!
  • nga_blur_noise_probability (bool or float) – Probability of applying a Gaussian filter and noise to the input data. The value must be a bool or be within the range [0.0, 1.0] Default: False (disabled)
  • blurry_blobs_probability (bool or float) – Probability of augmenting the input data with blurry “blobs”. “Blobs” mean random cubes across the input data. The region within a cube is blurred with Gaussian smoothing. The level of smoothing and the position of each cube are random. The value must be a bool or be within the range [0.0, 1.0] Default: False (disabled)
  • blurry_blobs_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.blurry_blobs(). Effective if setting blurry_blobs_probability > 0.
  • noisy_random_erasing_probability (float) – Probability of augmenting the input data with noise blobs. The blob region is filled with random noise values between 0 and 255.
  • noisy_random_erasing_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.noisy_random_erasing(). Effective if setting noisy_random_erasing_probability > 0.
  • uniform_random_erasing_probability (float) – Probability of augmenting the input data with uniform blobs. The blob region is filled with one value.
  • uniform_random_erasing_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.uniform_random_erasing(). Effective if uniform_random_erasing_probability > 0.
Returns:

  • data (np.ndarray) – [bs, ch, x, y] or [bs, ch, z, y, x] for 2D and 3D CNNS
  • target (np.ndarray) – [bs, ch, x, y] or [bs, ch, z, y, x]
  • ll_mask1 (np.ndarray) – (optional) [bs, n_target]
  • ll_mask2 (np.ndarray) – (optional) [bs, n_target]
getskel(source)[source]

Draw an example skeleton according to sampling weight on training data, or randomly on valid data

load_data()[source]
Parameters:
  • d_path/l_path (string) – Directories to load data from
  • d_files/l_files (list) – List of data/target files in <path> directory (must be in the same order!). Each list element is a tuple in the form (<Name of h5-file>, <Key of h5-dataset>)
  • cube_prios (list) – (not normalised) list of sampling weights to draw examples from the respective cubes. If None the cube sizes are taken as priorities.
  • valid_cubes (list) – List of indices for cubes (from the file-lists) to use as validation data and exclude from training, may be empty list to skip performance estimation on validation data.
read_files()[source]

Image files on disk are expected to be in order (ch,x,y,z) or (x,y,z) But image stacks are returned as (z,ch,x,y) and target as (z,x,y,) irrespective of the order in the file. If the image files have no channel this dimension is extended to a singleton dimension.

class elektronn2.data.cnndata.BatchCreatorImage(input_node, target_node=None, d_path=None, l_path=None, d_files=None, l_files=None, cube_prios=None, valid_cubes=None, border_mode='crop', aniso_factor=2, target_vec_ix=None, target_discrete_ix=None, h5stream=False, zxy=True)[source]

Bases: object

check_files()[source]

Check if file paths in the network config are available.

getbatch(batch_size=1, source='train', grey_augment_channels=None, warp=False, warp_args=None, ignore_thresh=False, force_dense=False, affinities=False, nhood_targets=False, ret_ll_mask=False, nga_blur_noise_probability=False, blurry_blobs_probability=False, blurry_blobs_args=None, noisy_random_erasing_probability=0, noisy_random_erasing_args=None, uniform_random_erasing_probability=0, uniform_random_erasing_args=None)[source]

Prepares a batch by randomly sampling, shifting and augmenting patches from the data

Parameters:
  • batch_size (int) – Number of examples in batch (for CNNs often just 1)
  • source (str) – Data set to draw data from: ‘train’/’valid’
  • grey_augment_channels (list) – List of channel indices to apply grey-value augmentation to
  • warp (bool or float) – Whether warping/distortion augmentations are applied to examples (slow –> use multiprocessing). If this is a float number, warping is applied to this fraction of examples e.g. 0.5 –> every other example.
  • warp_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.transformations.get_warped_slice()
  • ignore_thresh (float) – If the fraction of negative targets in an example patch exceeds this threshold, this example is discarded (Negative targets are ignored for training [but could be used for unsupervised target propagation]).
  • force_dense (bool) – If True the targets are not sub-sampled according to the CNN output strides. Dense targets requires MFP in the CNN!
  • affinities –
  • nhood_targets –
  • ret_ll_mask (bool) – If True additional information for reach batch example is returned. Currently implemented are two ll_mask arrays to indicate the targeting mode. The first dimension of those arrays is the batch_size!
  • nga_blur_noise_probability (bool or float) – Probability of applying a Gaussian filter and noise to the input data. The value must be a bool or be within the range [0.0, 1.0] Default: False (disabled)
  • blurry_blobs_probability (bool or float) – Probability of augmenting the input data with blurry “blobs”. “Blobs” mean random cubes across the input data. The region within a cube is blurred with Gaussian smoothing. The level of smoothing and the position of each cube are random. The value must be a bool or be within the range [0.0, 1.0] Default: False (disabled)
  • blurry_blobs_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.blurry_blobs(). Effective if setting blurry_blobs_probability > 0.
  • noisy_random_erasing_probability (float) – Probability of augmenting the input data with noise blobs. The blob region is filled with random noise values between 0 and 255.
  • noisy_random_erasing_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.noisy_random_erasing(). Effective if setting noisy_random_erasing_probability > 0.
  • uniform_random_erasing_probability (float) – Probability of augmenting the input data with uniform blobs. The blob region is filled with one value.
  • uniform_random_erasing_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.uniform_random_erasing(). Effective if uniform_random_erasing_probability > 0.
Returns:

  • data (np.ndarray) – [bs, ch, x, y] or [bs, ch, z, y, x] for 2D and 3D CNNS
  • target (np.ndarray) – [bs, ch, x, y] or [bs, ch, z, y, x]
  • ll_mask1 (np.ndarray) – (optional) [bs, n_target]
  • ll_mask2 (np.ndarray) – (optional) [bs, n_target]
load_data()[source]
Parameters:
  • d_path/l_path (string) – Directories to load data from
  • d_files/l_files (list) – List of data/target files in <path> directory (must be in the same order!). Each list element is a tuple in the form (<Name of h5-file>, <Key of h5-dataset>)
  • cube_prios (list) – (not normalised) list of sampling weights to draw examples from the respective cubes. If None the cube sizes are taken as priorities.
  • valid_cubes (list) – List of indices for cubes (from the file-lists) to use as validation data and exclude from training, may be empty list to skip performance estimation on validation data.
read_files()[source]

Image files on disk are expected to be in order (ch,x,y,z) or (x,y,z) But image stacks are returned as (z,ch,x,y) and target as (z,x,y,) irrespective of the order in the file. If the image files have no channel this dimension is extended to a singleton dimension.

warp_cut(img, target, warp, warp_params)[source]

(Wraps elektronn2.data.transformations.get_warped_slice())

Cuts a warped slice out of the input and target arrays. The same random warping transformation is each applied to both input and target.

Warping is randomly applied with the probability defined by the warp parameter (see below).

Parameters:
  • img (np.ndarray) – Input image
  • target (np.ndarray) – Target image
  • warp (float or bool) – False/True disable/enable warping completely. If warp is a float, it is used as the ratio of inputs that should be warped. E.g. 0.5 means approx. every second call to this function actually applies warping to the image-target pair.
  • warp_params (dict) – kwargs that are passed through to elektronn2.data.transformations.get_warped_slice(). Can be empty.
Returns:

  • d (np.ndarray) – (Warped) input image slice
  • t (np.ndarray) – (Warped) target slice
warp_stats
class elektronn2.data.cnndata.GridData(*args, **kwargs)[source]

Bases: elektronn2.data.cnndata.AgentData

getbatch(**get_batch_kwargs)[source]

Prepares a batch by randomly sampling, shifting and augmenting patches from the data

Parameters:
  • batch_size (int) – Number of examples in batch (for CNNs often just 1)
  • source (str) – Data set to draw data from: ‘train’/’valid’
  • grey_augment_channels (list) – List of channel indices to apply grey-value augmentation to
  • warp (bool or float) – Whether warping/distortion augmentations are applied to examples (slow –> use multiprocessing). If this is a float number, warping is applied to this fraction of examples e.g. 0.5 –> every other example.
  • warp_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.transformations.get_warped_slice()
  • ignore_thresh (float) – If the fraction of negative targets in an example patch exceeds this threshold, this example is discarded (Negative targets are ignored for training [but could be used for unsupervised target propagation]).
  • force_dense (bool) – If True the targets are not sub-sampled according to the CNN output strides. Dense targets requires MFP in the CNN!
  • affinities –
  • nhood_targets –
  • ret_ll_mask (bool) – If True additional information for reach batch example is returned. Currently implemented are two ll_mask arrays to indicate the targeting mode. The first dimension of those arrays is the batch_size!
  • nga_blur_noise_probability (bool or float) – Probability of applying a Gaussian filter and noise to the input data. The value must be a bool or be within the range [0.0, 1.0] Default: False (disabled)
  • blurry_blobs_probability (bool or float) – Probability of augmenting the input data with blurry “blobs”. “Blobs” mean random cubes across the input data. The region within a cube is blurred with Gaussian smoothing. The level of smoothing and the position of each cube are random. The value must be a bool or be within the range [0.0, 1.0] Default: False (disabled)
  • blurry_blobs_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.blurry_blobs(). Effective if setting blurry_blobs_probability > 0.
  • noisy_random_erasing_probability (float) – Probability of augmenting the input data with noise blobs. The blob region is filled with random noise values between 0 and 255.
  • noisy_random_erasing_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.noisy_random_erasing(). Effective if setting noisy_random_erasing_probability > 0.
  • uniform_random_erasing_probability (float) – Probability of augmenting the input data with uniform blobs. The blob region is filled with one value.
  • uniform_random_erasing_args (dict) – Additional keyword arguments that get passed through to elektronn2.data.non_geometric_augmentation.uniform_random_erasing(). Effective if uniform_random_erasing_probability > 0.
Returns:

  • data (np.ndarray) – [bs, ch, x, y] or [bs, ch, z, y, x] for 2D and 3D CNNS
  • target (np.ndarray) – [bs, ch, x, y] or [bs, ch, z, y, x]
  • ll_mask1 (np.ndarray) – (optional) [bs, n_target]
  • ll_mask2 (np.ndarray) – (optional) [bs, n_target]