Reference

cp(src::CSeis, dst[, extents=:]; batch_size=32, workers=Distributed.workers)

Copy a CloudSeis data-set to dst and where dst is either of type Container or of type Vector{Container}. The latter is used for sharding data across multiple storage accounts.

The option extents can be used to copy a sub-set of the CloudSeis data extents (e.g. 1:10). The batch_size option allows extents to be copied in batches and where the number of extents associated with each batch is set via batch_size. Within a batch, each extent is copied via an asynchronous task.

The cp method will be executed on the set of machines defined by workers. Note that the work will be sent to the workers one batch at a time.

mv(src::CSeis, dst::Container; batch_size=32, workers=Distributed.workers)

move a CloudSeis data-set to dst. See cp for a description of batch_size and workers.

rm(io::CSeis)

Delete a CloudSeis data-set.

close(io::CSeis)

Close a handle to a CloudSeis data-set. This may also flush contents of the CloudSeis buffer to storage.

copy!(ioout::CSeis, hdrsout::AbstractMatrix{UInt8}, ioin::CSeis, hdrsin::AbstractMatrix{UInt8})

copy headers from hdrsin to hdrsout, and where hdrsin are headers from a frame of ioin, and where hdrsout are headers from a frame of ioout.

Example

ioin = csopen(AzContainer("mydataset-in-cs";storageaccount="mystorageaccount"))
ioout = csopen(AzContainer("mydataset-out-cs";storageaccount="mystorageaccount"))

hdrsin = readframehdrs(ioin, 1)
hdrsout = allocframehdrs(ioout)
copy!(ioout, hdrsout, ioin, hdrsin)

empty!(io::CSeis)

Empty (i.e. delete the extents) from a CloudSeis data-set.

get(prop::TraceProperty, hdrs::Matrix, i)

Returns the vale of a trace header property for the ith column in the trace headers hdrs.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
hdrs = readframehdrs(io, 1)
get(prop(io, "TRACE"), hdrs, 1)
close(io)

get(prop::TraceProperty, hdr::Vector)

Returns the vale of a trace header property for the trace header hdr.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
hdrs = readframehdrs(io,1)
hdr = @view hdrs[:,1]
get(prop(io, "TRACE"), hdr)
close(io)

in(propdef::TracePropertyDef, io::CSeis)

return true if propdef exists in the CloudSeis data-set, io.

length(io::CSeis)

Return the total number of frames in a CloudSeis data-set.

ndims(io::CSeis)

Return the number of dimensions in a CloudSeis data-set.

reduce(io[; optional keyword arguments...])

Consolodate data-set extents into fewer extents.

Optional keyword arguments

• mbytes_per_extent=1000 set the size of each output extent via either the size of each extent
• frames_per_extent=0 set the size of each output extent via the number of frames allowed in each extent.

size(io::CSeis[,i])

Return the size of a CloudSeis data-set. If i is specified, then return the size of dimension i.

write(io::CSeis, trcs, rng...)

Write traces to io, and where the size and dimension of trcs must match the size and dimensions of rng....

Example 1

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container, "w"; axis_length=[100,101,102])
write(io, rand(Float32,100,101,102), :, :, :)
close(io)

Example 2

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container, "w"; axis_lengths=[100,101,102])
write(io, rand(Float32,100,50,1), :, 1:50, 1)
close(io)

cscreate(containers[; optional keyword arguments...])

Create a new CloudSeis dataset without creating a corresponding handle and without opening the data-set. Please see help for csopen for the optional keyword arguments.

containers is either of type Container or Vector{<:Container}. In the former case, all extents are stored in a single container, and in the later case, extents are sharded accross multiple containers.

In "r" and "r+" mode, and when the existing data-set is sharded over multiple containers, it is only required to supply the primary container. The "description.json" object is in the "primary" container.

io = csopen(containers[, mode="r"]; axis_lengths::Vector{Int}[, optional keyword arguments])

returns a handle to a CloudSeis data-set, and where container::Container corresponds to a dataset where all extents are in a single container, and container::Vector{<:Container} corresponds to a dataset where extents are sharded across mulitple containers. The container(s) can be either POSIX folders (container::Folder), or cloud storage container (e.g. container::AzContainer).

mode is one of "r" - read, "w" - write, "r+" - open existing data-set for reading and writing.

If mode="w", then axis_lenghts is a required key-word argument. axis_lengths specifies the size of the container. The axis_lengths vector is of at-least length 3.

Optional keyword arguments

• similarto::String An existing CloudSeis dataset. If set, then all other named arguments can be used to modify the data context that belongs to the existing CloudSeis dataset.
• history history dictionary (see notes, below, for how it should be formatted/constructed).
• datatype::String Examples are CMP, SHOT, etc. If not set, then UNKNOWN is used.
• traceformat::DataType=Float32 Not supported. We have only tested against traceformat=Float32.
• byteorder::String
• extents::Vector{UnitRange} List of integer ranges (frame range) for each extent. Each extent maps to an object in block storage (set only one of extents, frames_per_extent and mbytes_per_extent).
• frames_per_extent::Int Nominal number of frames per extent (set only one of extents, frames_per_extent and mbytes_per_extent).
• mbytes_per_extent::Int Nominal size of each extent in units of mega-bytes (set only one of extents, frames_per_extent and mbytes_per_extent).
• geometry::Geometry An optional three point geometry can be embedded in the CloudSeis file.
• tracepropertydefs::Vector{TracePropertyDef} An array of trace property definitions. These are in addition to the axes property definitions.
• dataproperties::Vector{DataProperty} An array of data properties. One property per dataset rather than one property per trace (as is true for tracepropertydefs).
• axis_propdefs::Vector{TracePropertyDef} Trace properties corresponding to the CloudSeis axes. If not set, then SAMPLE, TRACE, FRAME, VOLUME and HYPRCUBE are used.
• axis_units::Vector{String} Units corresponding to the CloudSeis axes. e.g. SECONDS, METERS, etc. If not set, then UNKNOWN is used.
• axis_domains::Vector{String} Domains corresponding to the CloudSeis axes. e.g. SPACE, TIME, etc. If not set, then UNKNOWN is used.
• axis_pstarts::Vector{Float64} Physical origins for each axis. If not set, then 0.0 is used for the physical origin of each axis.
• axis_pincs::Vector{Float64} Physical deltas for each axis. If not set, then 1.0 is used for the physical delta for each axis.
• axis_lstarts::Vector{Int} Logical starts for each axis. If not set, then 1 is used for the logical start for each axis.
• axis_lincs::Vector{Int} Logical increments for each axis. If not set, then 1 is used for the logical increment for each axis.
• compressor="leftjustify" Compress the cache before writing to disk. This is particularly useful for data with variable fold. chooose from: ("none", "blosc", "leftjustify", "zfp", "cvx")
• compressor_options=() Pass options as key-word arguments to the compression algorithm. Currently, only "zfp" and "cvx" have options[1]

Example

Azure storage

using AzStorage, CloudSeis
container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container, "w"; axis_lengths=[10,11,12], axis_pincs=[0.004,10.0,20.0])
close(io)

POSIX storage

using AzStorage, FolderStorage
container = Folder("mydataset-cs")
io = csopen(container, "w"; axis_lengths=[10,11,12], axis_pincs=[0.004,10.0,20.0])

Notes

• When using the similarto option, one can change the number of dimensions of the data-set via axis_lengths. If one shrinks the number of dimensions,

then various data-set properties (e.g. axis_units) will be truncated. The truncation can be customized by using appropriate key-word arguments.

• The zfp compression options are tol, precision and rate. So, for example:

using AzStorage, CloudSeis
container = AzContainer("mydataset-cs"; storageaccount="mystorageacccount")
io = csopen(container, "w"; axis_lengths=[10,11,12], compressor="zfp", compressor_options=(tol=1e-4,))

Please refer to ZFPCompressor.jl for more information. If compressor_options is not supplied, then the defaults are (precision=16,). Also, note that compressor_options=() results in ZFP lossless compression. ZFP lossless compression will be used for the headers and foldmap regardless of the choice of compressor_options.

• The cvx compression options are b1, b2, b3 and scale. So, for example:

Please refer to CvxCompress.jl for more information. If compressor_options is no supplied, then the defaults are (b1=16,b2=16,b3=16,scale=1e-2).

• The history dictionary must follow a specific structure. One can get the history from an existing data-set via history(io::CSeis), and can

construct and/or agument history via history!. In general the structure is,

Dict(
    "inputs" => [
        Dict(
            "container" => container, # dictionary describing where the input data-set is.
            "history" => history dictionary that, recursively, embeds the history of the input data-set.
        )
    ],
    "flow" => Dict(
        "flow" => Dict(
            "parameters" => flowparameters, # dictionary describing parameters that apply to all processes
            "processes" => [
                "process" => process, # dictionary or string that uniquely locates the process run
                "parameters" => parameters # dictionary describing parameters passed to the process
            ]
        )
    )
)

dataproperty(io::CSeis, name[, default_value])

Return the CloudSeis data property with name that is in the CloudSeis data-set io. If default_value is provided and a property with name does not exist, then return default_value.

description!(io::CSeis, kwargs...)

Limited mutation of the properties of an existing CloudSeis dataset.

Optional key-word arguments

• axis_lengths = nothing Grow the size of the axis lengths (3rd dimension and higher)[1].

Notes

• [1] axis_lengths must be the same length as size(io). In addition, prod(axis_lengths[3:end])

must be greater than prod(size(io)[3:end]) and axis_lengths[i] must equal size(io,i) for i∈(1..ndims(io)-1).

hasdataproperty(io::CSeis, name)

Return true if the data property called name exists in io.

headerlength(io::CSeis)

Returns the length (number of bytes) of a trace header for a CloudSeis data-set.

h = history!([io::CSeis|history::Dict]; kwargs...)

Mutate the history of an existing CloudSeis dataset or dictionary, or create a new history dictionary.

optional key-word arguments

• process="" name of a process that was run in a flow that produced the data-set.
• process_parameters=Dict() process parameters.
• flow_parameters=Dit() flow parameters.

Notes

• If more than one process was run (e.g. a flow of processes) to produce the data-set, then call history! in the

order that the processes were run.

• The history can be passed to the csopen and cscreate methods via the history key-word argument.

trcs,hdrs = allocframe(io::CSeis)

Allocate and return memory to store traces and trace headers for a single frame.

hdrs = allocframehdrs(io::CSeis)

Allocate and return memory to store trace headers for a single frame.

trcs = allocframetrcs(io::CSeis)

Allocate and return memory to store traces for a single frame.

domains(io::CSeis[,i])

Return the domain for the data-context axes. If i is specified, then return the domain for the ith data-context axis.

fold(io::CSeis, i)

Return the fold of a CloudSeis frame corresponding, and where the frame is described by its index i. The frame index can be either an Int, a VarArg{Int}, or a CartesianIndex.

Example

io = csopen("foo.cs", "w"; axis_lengths=[10,11,12])
fold(io, 5, 6) # fold corresponding to 5th frame and 6th volume

geometry(io::CSeis)

Return the geometry (if any) associated with the CloudSeis data-set.

leftjustify!(io::CSeis, trcs, hdrs)

This is primarily used for compression, but one can also use this as a convenience method to move all live traces to the beginning of its frame.

leftjustify!(io::CSeis, hdrs)

Convenience method to move all live traces to the beginning of its frame.

lincs(io::CSeis[,i])

Return the logical increment for the data-context axes. If i is specfied, then return the logical increment for the ith data-context axis

lstarts(io::CSeis[,i])

Return the logical start for the data-context axes. If i is specfied, then return the logical start for the ith data-context axis

pincs(io::CSeis[,i])

Return the physical increment for the data-context axes. If i is specified, then return the physical increment for the ith data-context axis.

prop(io::CSeis, propertydef)

Given a property definition propertydef, return a trace property corresponding to the CloudSeis data-set. propertydef can either be a string or a TracePropertyDef.

propdefs(io::CSeis[,i])

Return the property definitions corresponding to the data-context axes. If i is specified, then return the property definition for the ith data-context axis.

props(io::CSeis[,i])

Return the trace property corresponding to the data-context axes. If i is specified, then return the property for the ith data-context axis.

pstarts(io::CSeis[,i])

Return the physical start for the data-context axes. If i is specified, then return the physical start for the ith data-context axis.

readframe!(io::CSeis, trcs, hdrs, idx...; regularize=true)

Read traces and headers from io into trcs::Matrix, and hdrs::Matrix for the frame idx.... idx... can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

trcs,hdrs = readframe(io::CSeis, idx...; regularize=true)

Read traces and headers from io for the frame idx.... idx... can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

readframehdrs!(io::CSeis, trcs, idx...; regularize=true)

Read headers from io into hdrs::Matrix for the frame idx.... idx... can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

hdrs = readframehdrs(io::CSeis, idx...; regularize=true)

Read headers from io for the frame idx.... idx.. can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

readframetrcs!(io::CSeis, trcs, idx...[; regularize=true])

Read traces from io into trcs::Matrix for the frame idx.... idx... can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

trcs = readframetrcs(io::CSeis, idx...; regularize=true)

Read traces from io for the frame idx.... idx.. can either be integer(s) or a CartesianIndex.

The regularize named argument is only applicable when the compression method is LeftJustifyCompressor. If set to true, then traces are regularized to their correct context locations. Otherwise, they remain left justified. Note that one can subsequently use the regularize! method.

readhdrs!(io::CSeis, hdrs, rng...)

Read trace headers from io into hdrs::AbstractArray, and where the size and dimension of hdrs must correspond to the range specification in rng.... Unlike readtrcs!, we always read the entirity of each trace header. Therefore, the first range specifier must be :.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
size(io) # (100,101,102)
hdrs = readhdrs!(io, Array{Float32,3}(undef,headerlength(io),101,1), :, :, 1)
close(io)

readhdrs(io::CSeis, rng...)

Read trace headers from io. Unlike readtrcs!, we always read the entirity of each trace header. Therefore, the first range specifier must be :.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
size(io) # (100,101,102)
hdrs = readhdrs(io, :, :, 1)
close(io)

readtrcs!(io::CSeis, trcs, rng...)

Read traces from io into trcs::AbstractArray, and where the size and dimension of trcs must correspond to the range specification in rng....

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
size(io) # (100,101,102)
trcs = readtrcs!(io, Array{Float32,3}(undef,50,101,1), 1:50, :, 1)
close(io)

readtrcs(io::CSeis, trcs, rng...)

Read traces from io for the specified rng....

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
size(io) # (100,101,102)
trcs = readtrcs(io, 1:50, :, 1)
close(io)

regularize!(io::CSeis[, fld], trcs, hdrs)

This is primarily used for compression, but one can also use this as a convenience method undo the effect of leftjustify!, moving traces from the beginning of the frame to their recorded trace location.

set!(prop::TraceProperty, hdrs::Matrix, i, value)

Set the vale of the header property for the ith column of hdrs.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
hdrs = readframehdrs(io, 1)
set!(prop(io, "TRACE"), hdr, 1, 2)
close(io)

set!(prop::TraceProperty, hdr::Vector, value)

Set the vale of a trace header property for the hdr.

Example

container = AzContainer("mydataset-cs"; storageaccount="mystorageaccount")
io = csopen(container)
hdrs = readframehdrs(io, 1)
hdr = @view hdrs[:,1]
set!(prop(io, "TRACE"), hdr, 1)
close(io)

units(io::CSeis[,i])

Return the physical unit for the data-context axes. If i is specfied, then return the physical unit for the ith data-context axis.

writeframe(io::CSeis, trcs, hdrs)

Write a frame to io. The location of the frame is determined by the axis headers set in hdrs.

writeframe(io::CSeis, trcs, idx...)

Write a frame to io. The location of the frame is determined from idx... which can either be integer(s) or a CartesianIndex. A minimal set of headers are created from idx... and are also written to io.

g = Geometry([;kwargs...])

Returns a geometry object associated with the CloudSeis data-set. This includes an origin (o) and three orthogonal vectors (u,v,w). In addition, one can specify how azimuth is defined for TTI earth models by giving a specific direction for positive azimuth, and the axis from which it is measured. This can be useful when describing the orientation of a model in three dimensional space.

key-word arguments

• ox=0,oy=0,oz=0 grid origin
• ux=0,uy=0,uz=1 u vector such that the end of the u axis is at the point (ox+ux,oy+uy,oz+uz)
• vx=0,vy=0,vz=1 v vector such that the end of the u axis is at the point (ox+vx,oy+vy,oz+vz)
• wx=0,wy=0,wz=1 w vector such that the end of the u axis is at the point (ox+wx,oy+wy,oz+wz)
• u1=1,un=2 integer end-points (u axis) that can be used to describe a grid (e.g. for finite difference)
• v1=1,vn=2 integer end-points (v axis) that can be used to describe a grid (e.g. for finite difference)
• w1=1,wn=2 integer end-points (w axis) that can be used to describe a grid (e.g. for finite difference)
• x_direction="east" compass direction that 'x' is parallel to
• y_direction="north" compass direction that 'y' is parallel to
• z_direction="depth" compass direction that 'z' is parallel to
• tti_angle_units="degrees" for TTI models specify if the tilt and azimuth angles are stored in ("degrees", "radians", or "unknown")
• tti_azimuth_positive_direction="counter clockwise" for TTI models, define the orientation of the azimuth ("clockwise", "counter clockwise" or "unknown")
• tti_azimuth_origin_axis="v" for TTI models, define the axis from which azimuth is measured and at which azimuth is 0 (choose from: "u", "v", "w", "x", "y", "-u", "-v", "-w", "-x", "-y" or "unknown")
• tti_symmetry_axis_z_direction="elevation" for TTI models, define if the projection of the normal to the symmetry axis onto z = (0,0,1) point up (elevation) or down (depth)

notes

• this method does not check to see if the u,v,w vectors are orthogonal
• for models that do not required azimuthal anisotropy (e.g. isotropic, VTI), it is convenient to set tti_azimumth_positive_direction and tti_azimuth_origin_axis to "unknown"

I = LogicalIndices(io::CSeis)

Returns a construct similar to CartesianIndices that allows conversion from linear indices to cartesian indices that are offest by the logical starts and deltas of the CloudSeis data context. In addition, LogicalIndices implements iteration for looping over all frames in a data-set. For example,

io = csopen(AzContainer("foo"; storageaccount="bar))
idx = LogicalIndices(io)[2] # get the index corresponding to the second frame in the data-set.
for idx in LogicalIndices(io)
    @show idx
    trcs, hdrs = getframe(io, idx)
end