ValueStore

Data storage shared by MATLAB clients and workers

Since R2022a

Description

ValueStore is an object that stores data owned by a specific job. Each entry of the object consists of a value and its corresponding key. When the owning job is deleted, the ValueStore object is deleted as well. Use ValueStore to store data from MATLAB^® workers that can be retrieved by MATLAB clients during the execution of a job (even while the job is still running).

• Any MATLAB process client or worker can write an entry to the ValueStore at any time. Any MATLAB process client or worker can then read this entry from the ValueStore at any time. However, the ordering of operations executed by different processes is not guaranteed.

• ValueStore is not held in system memory, so it can be used to store large results.

Creation

The ValueStore object is automatically created when you create:

• A job on a cluster, which is a parallel.Job object. To create a job, use the batch, createJob, or createCommunicatingJob function.

• A parallel pool of process workers on the local machine, which is a ProcessPool object. To create a process pool, use the parpool function.

• A parallel pool of thread workers on the local machine, which is a ThreadPool object. To create a thread pool, use the parpool function. (since R2023b)

• A parallel pool of workers on a cluster of machines, which is a ClusterPool object. To create a cluster pool, use the parpool function.

You can access the ValueStore object on a worker by using the getCurrentValueStore function. You can then retrieve the ValueStore object on a client by using the ValueStore property that is associated with the job or the parallel pool. For example, see Run Batch Job and Retrieve Data from Workers.

Properties

KeyUpdatedFcn — Callback executed when entry is added or replaced

function handle

Callback executed when an entry is added or replaced, specified as a function handle. The function handle must accept two input arguments that represent the ValueStore object and its key when an entry is added or replaced.

KeyRemovedFcn — Callback executed when entry is removed

function handle

Callback executed when an entry is removed, specified as a function handle. The function handle must accept two input arguments that represent the ValueStore object and its key when an entry is removed.

Object Functions

isKey	Determine if ValueStore or FileStore object contains keys
keys	Return all keys of ValueStore or FileStore object
put	Add key-value pairs to ValueStore object
get	Get values from ValueStore object
remove	Remove entries from ValueStore or FileStore object

Examples

Run Batch Job and Retrieve Data from Workers

Run a simulation on workers and retrieve the data storage of the job on a client. The data storage is a ValueStore object with key-value entries.

The following simulation finds the singular values of random matrices and stores the results in the ValueStore object.

type workerSvdCode

function workerSvdCode(models)
% Get the ValueStore of the current job
store = getCurrentValueStore;
for i = 1:numel(models)
    % Store simulation results in the ValueStore object
    pause(1)
    key = strcat("result_",num2str(i));
    store(key) = svd(rand(models(i)));
    store("progress") = i/numel(models);
end
end

The following callback function is executed when an entry is added to the ValueStore object.

type handleNewEntry

function handleNewEntry(store,key)
if strcmp(key,"progress")
    fprintf("Progress update: %.2f %%\n",store(key)*100);
else
    fprintf("Result %s added\n",key);
end
end

Run a batch job on workers using the default cluster profile.

models = [8,16,32,20];
c = parcluster;
job = batch(c,@workerSvdCode,0,{models});

Retrieve the ValueStore object on the client while the job is still running. Show the progress of the job.

store = job.ValueStore;
store.KeyUpdatedFcn = @handleNewEntry;
wait(job);

Result result_1 added
Progress update: 25.00 %
Result result_2 added
Progress update: 50.00 %
Result result_3 added
Progress update: 75.00 %
Result result_4 added
Progress update: 100.00 %

Get the entry value as specified by the key "result_1" from the object.

val1 = store("result_1")

val1 =

    4.3318
    1.2988
    1.1040
    0.8813
    0.5711
    0.3991
    0.2092
    0.1048

Run Simulation on Parallel Pool of Process Workers and Retrieve Data

Run simulation on a parallel pool of process workers and retrieve the data storage on a client.

The following simulation finds the inverse of random matrices and stores the results in the ValueStore object.

type workerInvCode

function workerInvCode(models)
% Get the ValueStore of the current job
store = getCurrentValueStore;
for i = 1:numel(models)
    % Store simulation results in the ValueStore object
    pause(1);
    key = strcat("result_",num2str(i));
    store(key) = inv(rand(models(i)));
end
end

The following callback function is executed when an entry is added to the ValueStore object.

type handleUpdatedEntry

function handleUpdatedEntry(store,key)
    fprintf("Result %s added\n",key);
end

Start a parallel pool of process workers.

pool = parpool('Processes');

Starting parallel pool (parpool) using the 'Processes' profile ...
Connected to parallel pool with 6 workers.

Get the ValueStore for this pool and assign the callback function to be executed when an entry is added.

store = pool.ValueStore;
store.KeyUpdatedFcn = @handleUpdatedEntry;

Run the simulation on the pool.

models = [4,8,32,20];
future = parfeval(@workerInvCode,0,models);
wait(future);

Result result_1 added
Result result_2 added
Result result_3 added
Result result_4 added

Run Independent Tasks and Retrieve Data and Files from All Tasks

Run a job of independent tasks. Then, retrieve the data and file storage of the job on a client.

The following simulation finds the permutations and combinations of a vector, and stores the results in the ValueStore and FileStore objects.

type taskFunction

function taskFunction(dataset,keyname)
% Get the ValueStore and FileStore of the current job
valueStore = getCurrentValueStore;
fileStore = getCurrentFileStore;
% Run the simulation to find permutation and combination
[result,logFile] = runSimulation(dataset);
% Store results in ValueStore to release system memory
valueStore(keyname) = result;
% Copy file to FileStore to retrieve the file from non-shared file system
copyFileToStore(fileStore,logFile,keyname);
end

function [result,logFile] = runSimulation(dataset)
    permutations = perms(dataset{1});
    combinations = nchoosek(dataset{1},dataset{2});
    result.N_perm = length(permutations);
    result.N_comb = length(combinations);
    logFile = strcat(tempname("C:\myLogFolder"),".mat");
    save(logFile,"permutations","combinations")
end

Create a job using the default cluster profile.

c = parcluster;
job = createJob(c);

Create independent tasks for the job. Each task runs the simulation with the given input.

set_1 = {[12,34,54],2};
set_2 = {[45,33],1};
set_3 = {[12,12,12,13,14],3};
tasks = createTask(job,@taskFunction,0,{{set_1,"sim_1"},{set_2,"sim_2"},{set_3,"sim_3"}});

Run the job and wait for it to finish.

submit(job);
wait(job);

Retrieve the data and file storage of the job.

valueStore = job.ValueStore;
fileStore = job.FileStore;

Show the result of the third task that is stored in the ValueStore object.

result_3 = valueStore("sim_3")

result_3 = struct with fields:
    N_perm: 120
    N_comb: 10

Copy files from the file storage as specified by the corresponding keys "sim_1" and "sim_2" to the local files "analysis_1.mat" and "analysis_2.mat".

copyFileFromStore(fileStore,["sim_1" "sim_2"],["analysis_1.mat" "analysis_2.mat"]);

Display all the information on the variables stored in the local files.

whos -file 'analysis_1.mat'

  Name              Size            Bytes  Class     Attributes

  combinations      3x2                48  double              
  permutations      6x3               144  double              

whos -file 'analysis_2.mat'

  Name              Size            Bytes  Class     Attributes

  combinations      2x1                16  double              
  permutations      2x2                32  double              

Limitations

• When using parallel.cluster.Generic clusters with 'HasSharedFileSystem' set to false, the visibility of modifications made to ValueStore while a job is running depends on your specific implementation. Without additional synchronization between the MATLAB client and worker JobStorageLocation, changes might only be visible once the job has completed.

Version History

Introduced in R2022a

R2023b: Use ValueStore on thread-based parallel pools

You can now use ValueStore on ThreadPool objects.

ValueStore is not supported on MATLAB backgroundPool.