ImportOptions

Import options for event stream

Since R2022b

This object requires Streaming Data Framework for MATLAB^® Production Server™.

Description

An ImportOptions object specifies how MATLAB imports tabular data from event streams. The object contains properties that control the data import process, including handling of errors and missing data.

Creation

You can create an ImportOptions object by using either the detectImportOptions function or the eventStreamImportOptions function. The preferred way is to use detectImportOptions.

• Use detectImportOptions to detect and populate the import properties based on the contents of the event stream specified by stream.

  opts = detectImportOptions(stream)

• Use eventStreamImportOptions to create import properties by specifying import options as name-value arguments.

  opts = eventStreamImportOptions(Name1=Value1,...,NameN=ValueN)

Properties

SelectedVariableNames — Subset of variables to import

character vector | string scalar | cell array of character vectors | string array

Subset of variables to import, specified as a character vector, string scalar, cell array of character vectors, or string array.

SelectedVariableNames must be a subset of names contained in the VariableNames property. By default, SelectedVariableNames contains all the variable names from the VariableNames property, which means that all variables are imported.

Use the SelectedVariableNames property to import only the variables of interest. Specify a subset of variables using the SelectedVariableNames property and use the readtimetable function to import only that subset.

Example: opts.SelectedVariableNames = "x" imports only the variable x from the event stream when you use readtimetable to import event stream data into a timetable.

Data Types: char | string

VariableNames — Variable names

cell array of character vectors | string array

Variable names, specified as a cell array of character vectors or string array. The VariableNames property contains the names to use when importing variables from the event stream into a timetable.

These variable names must exist in the stream. If the modified variable name is not in the stream, the import operation fails.

Example: io.VariableNames returns the current variable names in the event stream.

Example: io.VariableNames(3) = {'Mass'} changes the name of the third variable to Mass.

Data Types: cell | string

VariableTypes — Data type of variables

cell array of character vectors | string array

Data type of variables, specified as a cell array of character vectors or string array containing a set of valid data type names. The VariableTypes property designates the data types to use when importing variables from the event stream into a timetable.

The import operation attempts to convert the values in the stream to these types. This operation succeeds only when the conversion between primitive types is known and unambiguous, such as:

• Integer to string conversions

• Conversions where the constructor of the target type can accept a variable of that type in the stream

To update the VariableTypes property, use the setvartype function.

Example: io.VariableTypes returns the data types that stream variables have after they are imported into MATLAB timetables, which are the types of the corresponding timetable columns.

Example: io = setvartype(io,"vMotor","int32") changes the data type of the vMotor variable to int32.

Data Types: cell | string

KeyVariable — Key variable name

key (default) | string scalar | character vector

Name of the key variable in the event stream, specified as a string scalar or character vector. The default value is key.

Data Types: string | char

Object Functions

setvartype

Set data types used to import and export variables to stream

Examples

Create Import Options from Event Stream Data

Assume that you have a Kafka^® server running at the network address kafka.host.com:9092 that has a topic Triangles with JSON-encoded event value '{"triangle":{"x":"3","y":"4","z":"5"}}'. Each event value contains a variable "triangle", which is a structure of side lengths "x", "y", and "z". The side lengths are integers but are encoded as strings. readtimetable creates a triangle table column for this variable and sets the column value to this structure:

triangle = 

  struct with fields:

    x: "3"
    y: "4"
    z: "5"

ImportOptions enables you to change the types of the values in the imported structure and select which fields to import.

Create a KafkaStream object connected to the Triangles topic.

ks = kafkaStream("kafka.host.com",9092,"Triangles");

Create an ImportOptions object from the Kafka stream object. The data type of the length of each side is string.

opts = detectImportOptions(ks)

opts = 

  ImportOptions with properties:

            VariableNames: ["triangle/x"    "triangle/y"    "triangle/z"]
            VariableTypes: ["string"    "string"    "string"]
              KeyVariable: "key"
    SelectedVariableNames: ["triangle/x"    "triangle/y"    "triangle/z"]

To perform mathematical operations on the imported data, update the data type of variables to double. Because the side length variables are nested within "triangle", use a forward slash ("/") to specify the path to these variables.

opts = setvartype(opts, ["triangle/x", "triangle/y", "triangle/z"], "double")

opts = 

  ImportOptions with properties:

            VariableNames: ["triangle/x"    "triangle/y"    "triangle/z"]
            VariableTypes: ["double"    "double"    "double"]
              KeyVariable: "key"
    SelectedVariableNames: ["triangle/x"    "triangle/y"    "triangle/z"]

Update the ImportOptions property of the KafkaStream object.

ks.ImportOptions = opts

ks = 

  KafkaStream with properties:

                  Topic: "Triangles"
                  Group: "85c42e39-695d-467a-86f0-f0095792e7de"
                  Order: EventTime
                   Host: "kafka.host.com"
                   Port: 9092
      ConnectionTimeout: 30
         RequestTimeout: 61
          ImportOptions: "Import to MATLAB types"
          ExportOptions: "Source: function eventSchema"
          PublishSchema: "true"
             WindowSize: 50
            KeyVariable: "key"
            KeyEncoding: "utf16"
                KeyType: "text"
           KeyByteOrder: "BigEndian"
           BodyEncoding: "utf8"
             BodyFormat: "JSON"
              ReadLimit: "Size"
    TimestampResolution: "Milliseconds"

When importing the triangles, readtimetable converts the side lengths to double values.

tt = readtimetable(ks);
tt(1,:).triangle

ans = 

  struct with fields:

    x: 3
    y: 4
    z: 5

Create Import Options by Specifying Import Variable Names and Types

Create a schema for importing data from an event stream into MATLAB by specifying variable names and their data types to use during the import.

names = ["x","symbol"];
types = ["double","string"];

Construct an ImportOptions object using this data import schema.

opts = eventStreamImportOptions(VariableNames=names,VariableTypes=types)

opts = 

  ImportOptions with properties:

            VariableNames: ["x"    "symbol"]
            VariableTypes: ["double"    "string"]
              KeyVariable: [0×0 string]
    SelectedVariableNames: ["x"    "symbol"]

Apply the import options when creating a KafkaStream object.

ks = kafkaStream("kafka.host.com",9092,"Your_Kafka_Topic",ImportOptions=opts);

Import the data. The "Your_Kafka_Topic" topic must have events with exactly two variables, x and symbol. In addition, the types of these variables must be convertible to double and string, respectively. Otherwise, readtimetable throws an error.

tt = readtimetable(ks);

Version History

Introduced in R2022b