Skip to content

spec.md

Latest commit

 

History

History
648 lines (462 loc) · 19.4 KB

spec.md

File metadata and controls

648 lines (462 loc) · 19.4 KB
Raw

SecretFlow Open Specification

This page is auto-generated with protoc-gen-doc.

Table of Contents

DATA

COMPONENT

EVALUATION

REPORT

(DATA)=

DATA

Proto file: secretflow/spec/v1/data.proto

Messages

(distdata)=

DistData

A public record for a general distributed data.

The type of this distributed data, should be meaningful to components.

The concrete data format (include public and private parts) is defined by other protos.

Suggested internal types, i.e.

  • sf.table.vertical_table represent a secretflow vertical table
  • sf.table.individual_table represent a secretflow individual table
Field Type Description
name string The name of this distributed data.
type string Type.
system_info SystemInfo Describe the system information that used to generate this distributed data.
meta google.protobuf.Any Public information, known to all parties. i.e. VerticalTable.
data_refs repeated DistData.DataRef Remote data references.

(distdatadataref)=

DistData.DataRef

A reference to a data that is stored in the remote path.

Field Type Description
uri string The path information relative to StorageConfig of the party.
party string The owner party.
format string The storage format, i.e. csv.

(individualtable)=

IndividualTable

IndividualTable describes a table owned by a single party.

Field Type Description
schema TableSchema Schema.
line_count int64 If -1, the number is unknown.

(storageconfig)=

StorageConfig

A StorageConfig specifies the root for all data for one party.

  • At this moment, only local_fs is supported
  • We would support OSS, databases in future.
Field Type Description
type string Supported: local_fs.
local_fs StorageConfig.LocalFSConfig local_fs config.

(storageconfiglocalfsconfig)=

StorageConfig.LocalFSConfig

For local_fs.

Field Type Description
wd string Working directory.

(systeminfo)=

SystemInfo

Describe the application related to data.

Field Type Description
app string The application name. Supported: secretflow
app_meta google.protobuf.Any Meta for application.

(tableschema)=

TableSchema

The schema of a table.

  • A col must be one of id | feature | label. By default, it should be a feature.
  • All names must match the regexp [A-Za-z0-9.][A-Za-z0-9_>./]*.
  • All data type must be one of
  • int8
  • int16
  • int32
  • int64
  • uint8
  • uint16
  • uint32
  • uint64
  • float16
  • float32
  • float64
  • bool
  • int
  • float
  • str
Field Type Description
ids repeated string Id column name(s). Optional, can be empty.
features repeated string Feature column name(s).
labels repeated string Label column name(s). Optional, can be empty.
id_types repeated string Id column data type(s). Len(id) should match len(id_types).
feature_types repeated string Feature column data type(s). Len(features) should match len(feature_types).
label_types repeated string Label column data type(s). Len(labels) should match len(label_types).

(verticaltable)=

VerticalTable

VerticalTable describes a virtual vertical partitioning table from multiple parties.

Field Type Description
schemas repeated TableSchema The vertical partitioned slices' schema. Must match data_refs in the parent DistData message.
line_count int64 If -1, the number is unknown.

Enums

(COMPONENT)=

COMPONENT

Proto file: secretflow/spec/v1/component.proto

Messages

(attribute)=

Attribute

The value of an attribute

Field Type Description
f float FLOAT
i64 int64 INT NOTE(junfeng): "is" is preserved by Python. Replaced with "i64".
s string STRING
b bool BOOL
fs repeated float FLOATS
i64s repeated int64 INTS
ss repeated string STRINGS
bs repeated bool BOOLS
is_na bool Indicates the value is missing explicitly.

(attributedef)=

AttributeDef

Describe an attribute.

Field Type Description
prefixes repeated string Indicates the ancestors of a node, e.g. [name_a, name_b, name_c] means the path prefixes of current Attribute is name_a/name_b/name_c/. Only ^[a-zA-Z0-9_.-]*$ is allowed. input and output are reserved.
name string Must be unique in the same level just like Linux file systems. Only ^[a-zA-Z0-9_.-]*$ is allowed. input and output are reserved.
desc string none
type AttrType none
atomic AttributeDef.AtomicAttrDesc none
union AttributeDef.UnionAttrGroupDesc none
custom_protobuf_cls string Extras for custom protobuf attribute

(attributedefatomicattrdesc)=

AttributeDef.AtomicAttrDesc

Extras for an atomic attribute. Including: AT_FLOAT | AT_INT | AT_STRING | AT_BOOL | AT_FLOATS | AT_INTS | AT_STRINGS | AT_BOOLS.

Field Type Description
list_min_length_inclusive int64 Only valid when type is AT_FLOATS | AT_INTS | AT_STRINGS | AT_BOOLS.
list_max_length_inclusive int64 Only valid when type is AT_FLOATS | AT_INTS | AT_STRINGS | AT_BOOLS.
is_optional bool If True, when Atmoic Attr is not provided or is_na, default_value would be used. Else, Atmoic Attr must be provided.
default_value Attribute A reasonable default for this attribute if the user does not supply a value.
allowed_values Attribute Only valid when type is AT_FLOAT | AT_INT | AT_STRING | AT_FLOATS | AT_INTS | AT_STRINGS. Please use list fields of AtomicParameter, i.e. ss, i64s, fs. If the attribute is a list, allowed_values is applied to each element.
lower_bound_enabled bool Only valid when type is AT_FLOAT | AT_INT | AT_FLOATS | AT_INTS . If the attribute is a list, lower_bound is applied to each element.
lower_bound Attribute none
lower_bound_inclusive bool none
upper_bound_enabled bool Only valid when type is AT_FLOAT | AT_INT | AT_FLOATS | AT_INTS . If the attribute is a list, upper_bound is applied to each element.
upper_bound Attribute none
upper_bound_inclusive bool none

(attributedefunionattrgroupdesc)=

AttributeDef.UnionAttrGroupDesc

Extras for a union attribute group.

Field Type Description
default_selection string The default selected child.

(complistdef)=

CompListDef

A list of components

Field Type Description
name string none
desc string none
version string none
comps repeated ComponentDef none

(componentdef)=

ComponentDef

The definition of a comp.

Field Type Description
domain string Namespace of the comp.
name string Should be unique among all comps of the same domain.
desc string none
version string Version of the comp.
attrs repeated AttributeDef none
inputs repeated IoDef none
outputs repeated IoDef none

(iodef)=

IoDef

Define an input/output for component.

Field Type Description
name string should be unique among all IOs of the component.
desc string none
types repeated string Must be one of DistData.type in data.proto
attrs repeated IoDef.TableAttrDef Only valid for tables. The attribute path for a TableAttrDef is {input|output}/{IoDef name}/{TableAttrDef name}.

(iodeftableattrdef)=

IoDef.TableAttrDef

An extra attribute for a table.

If provided in a IoDef, e.g.

{
  "name": "feature",
  "types": [
      "int",
      "float"
  ],
  "col_min_cnt_inclusive": 1,
  "col_max_cnt": 3,
  "attrs": [
      {
          "name": "bucket_size",
          "type": "AT_INT"
      }
  ]
}

means after a user provide a table as IO, they should also specify cols as "feature":

  • col_min_cnt_inclusive is 1: At least 1 col to be selected.
  • col_max_cnt_inclusive is 3: At most 3 cols to be selected. And afterwards, user have to fill an int attribute called bucket_size for each selected cols.
Field Type Description
name string Must be unique among all attributes for the table.
desc string none
types repeated string Accepted col data types. Please check comments of TableSchema in data.proto.
col_min_cnt_inclusive int64 inclusive
col_max_cnt_inclusive int64 none
extra_attrs repeated AttributeDef extra attribute for specified col.

Enums

(attrtype)=

AttrType

Supported attribute types.

Name Number Description
ATTR_TYPE_UNSPECIFIED 0 none
AT_FLOAT 1 FLOAT
AT_INT 2 INT
AT_STRING 3 STRING
AT_BOOL 4 BOOL
AT_FLOATS 5 FLOATS
AT_INTS 6 INTS
AT_STRINGS 7 STRINGS
AT_BOOLS 8 BOOLS
AT_STRUCT_GROUP 9 none
AT_UNION_GROUP 10 none
AT_SF_TABLE_COL 11 none
AT_CUSTOM_PROTOBUF 12 none

(EVALUATION)=

EVALUATION

Proto file: secretflow/spec/v1/evaluation.proto

Messages

(nodeevalparam)=

NodeEvalParam

Evaluate a node.

  • CompListDef + StorageConfig + NodeEvalParam + other extra configs -> NodeEvalResult

NodeEvalParam contains all the information to evaluate a component.

Field Type Description
domain string Domain of the component.
name string Name of the component.
version string Version of the component.
attr_paths repeated string The path of attributes. The attribute path for a TableAttrDef is (input|output)/(IoDef name)/(TableAttrDef name)(/(column name)(/(extra attributes))?)?.
attrs repeated Attribute The value of the attribute. Must match attr_paths.
inputs repeated DistData The input data, the order of inputs must match inputs in ComponentDef. NOTE: Names of DistData doesn't need to match those of inputs in ComponentDef definition.
output_uris repeated string The output data uris, the order of output_uris must match outputs in ComponentDef.

(nodeevalresult)=

NodeEvalResult

NodeEvalResult contains outputs of a component evaluation.

Field Type Description
outputs repeated DistData Output data.

Enums

(REPORT)=

REPORT

Proto file: secretflow/spec/v1/report.proto

Messages

(descriptions)=

Descriptions

Displays multiple read-only fields in groups.

Field Type Description
name string Name of the Descriptions.
desc string none
items repeated Descriptions.Item none

(descriptionsitem)=

Descriptions.Item

Field Type Description
name string Name of the field.
desc string none
type string Must be one of bool/int/float/str
value Attribute none

(div)=

Div

A division or a section of a page.

Field Type Description
name string Name of the Div.
desc string none
children repeated Div.Child none

(divchild)=

Div.Child

Field Type Description
type string Supported: descriptions, table, div.
descriptions Descriptions none
table Table none
div Div none

(report)=

Report

Field Type Description
name string Name of the Report.
desc string none
tabs repeated Tab none
err_code int32 none
err_detail string Structed error detail (JSON encoded message).

(tab)=

Tab

A page of a report.

Field Type Description
name string Name of the Tab.
desc string none
divs repeated Div none

(table)=

Table

Displays rows of data.

Field Type Description
name string Name of the Table.
desc string none
headers repeated Table.HeaderItem none
rows repeated Table.Row none

(tableheaderitem)=

Table.HeaderItem

Field Type Description
name string none
desc string none
type string Must be one of bool/int/float/str

(tablerow)=

Table.Row

Field Type Description
name string none
desc string none
items repeated Attribute none

Enums

Scalar Value Types

Type Notes C++ Type Java Type Python Type

double		 double double float

float		 float float float

int32		 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. int32 int int

int64		 Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. int64 long int/long

uint32		 Uses variable-length encoding. uint32 int int/long

uint64		 Uses variable-length encoding. uint64 long int/long

sint32		 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. int32 int int

sint64		 Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. int64 long int/long

fixed32		 Always four bytes. More efficient than uint32 if values are often greater than 2^28. uint32 int int

fixed64		 Always eight bytes. More efficient than uint64 if values are often greater than 2^56. uint64 long int/long

sfixed32		 Always four bytes. int32 int int

sfixed64		 Always eight bytes. int64 long int/long

bool		 bool boolean boolean

string		 A string must always contain UTF-8 encoded or 7-bit ASCII text. string String str/unicode

bytes		 May contain any arbitrary sequence of bytes. string ByteString str