How to plot data from JSON Structure array

Hi

I have used HTTP to download a large amount structued data from an API.

This is my code

function n02()

downloaded_data = device_data("864475047548549","2022-04-08T10:31:00Z","2022-05-08T11:31:00Z");
%struct2table(downloaded_data)
plot(downloaded_data.Timestamp,downloaded_data.iot_battery);

end

This is the output i am getting

Error using plot
Invalid first data argument.

Error in n02 (line 6)
plot(x,y);

What am i doing wrong

Answer 1

[Jon]

I don't understand what you are expecting to happen in your example code. Your variable downloaded_data is assigned to an array of 3 strings. The variable downloaded_data doesn't have any fields and in particular not the ones you are trying to plot.(Timetstamp or iot_battery).

Here's a small example of how you should be working with the json data and plotting it. Hopefully you can follow from this and adapt your code accordingly

If you have some json text, txt, you can turn it into a structure using jsondecode(txt)

So for example

    txt = '{"t":[1, 2, 3, 4, 5],"x":[1, 4, 9, 16, 25]}'
    mydat = jsondecode(txt)
    plot(mydat.t,mydat.x)

[Jon]

Actually, I'm looking at this thread now and see it is completly confused. You kept on entering things as new answers rather than just continuing with comments to the original thread of my answer. Then I added comments to your answer thinking it was all still the original thread.

Regarding the grouping and hourly averages. Please see for example. https://www.mathworks.com/matlabcentral/answers/706998-hourly-and-6-hourly-average-of-data-in-timeable

[Dharmesh Joshi]

Thanks sorry for creating the confussion. I am getting the hang of the forum now,

[Jon]

Sorry I didn't catch it earlier either. Anyhow would still be clearer to accept my answer as answering your question, rather than as you have answered the question yourself. Thanks

[Dharmesh Joshi]

Ok, done, sorry for the time duration was trying to see how to do this. Then noticed that i had to undo the previous answer.

[Jon]

Thanks, good luck with your project

Answer 2

[Dharmesh Joshi]

Hi Jon

Yes, if i call the plot as

plot([downloaded_data.iot_battery])

it works. It seems that i had to enter [ ]. Is there a reason for this?

My data also includes a timestamp, which i belive it in a text formate as

'2022-04-08T10:41:33.3189578+00:00'

How do get this information on my x axis?

[Jon]

Looking more carefully, I see now that you have a function, device_data, that returns some data. Is this data in json format or is it already a structure. I can't tell enough from what you show to help you further. If you could please attach a small self contained (one that I can run) example of what you are trying to do that demonstrates the problem you are having then I might be able to give you more help.

Regarding why it might work with the square brackets, it seems like you may have data that is in a comma separated list, so putting it in square brackets turns it into an array that can be plotted.

[Dharmesh Joshi]

My API , which i do not want to disclose publically outputs a JSON as string and this is my function below.

function  data_from_table_api = device_data(deviceid,start_date_time,end_date_time)

import matlab.net.*;
import matlab.net.http.*; 
import matlab.net.http.field.*

% deviceid="864475047548549";
% start_date_time="2022-04-08T10:31:00Z";
% end_date_time="2022-05-08T11:31:00Z";

api = "my api";
%httpOpts = HTTPOptions;
headers = AcceptField('application/json;odata=nometadata'); 
r = RequestMessage('GET',headers);
        try                                   
            tic;
            resp = r.send(api);                                    
            toc;
            tic;
            data_from_table_api= jsondecode(resp.Body.Data); 
            toc;
            %data_from_table_api(1)
            size_of_struct = size(data_from_table_api)                         
        catch exc
            disp(exc);            
        end  
end

[Jon]

If you can't provide a runnable example to show the problem, then lets go back to your attempt to plot the two variables, downloaded_data.Timestamp,downloaded_data.iot_battery.

Please show the first few elements of downloaded_data.Timestamp, and downloaded_data.iot_battery so that I can see exactly what you are trying to plot.

[Dharmesh Joshi]

downloaded_data(1)

ans = 

  struct with fields:

                                       Timestamp: '2022-05-08T11:30:47.7909324+00:00'
                                     iot_battery: 100
                                      iot_signal: 24
                                  iot_temperture: 24677
                                    iot_humidity: 32791
                                       iot_light: 0
                                  iot_light_test: 0
                                  iot_mass_con_a: 2.9120
                                  iot_mass_con_b: 3.3380
                                  iot_mass_con_c: 3.5480
                                  iot_mass_con_d: 3.5890
                                   iot_num_con_a: 19.7610
                                   iot_num_con_b: 22.9010
                                   iot_num_con_c: 23.2270
                                   iot_num_con_d: 23.2890
                                   iot_num_con_e: 23.2890
                        iot_temperture_extension: 24677
                          iot_humidity_extension: 32791
                             iot_mass_con_a_test: 0
                             iot_mass_con_b_test: 0
                             iot_mass_con_c_test: 0
                             iot_mass_con_d_test: 0
                              iot_num_con_a_test: 0
                              iot_num_con_b_test: 0
                              iot_num_con_c_test: 0
                              iot_num_con_d_test: 0
                              iot_num_con_e_test: 0
                   iot_temperture_extension_test: 0
                     iot_humidity_extension_test: 0
                        iot_moisture_a_extension: 0
                        iot_moisture_b_extension: 0
                        iot_moisture_c_extension: 0
                                      iot_N02_F1: -10.9359
                                      iot_N02_F2: -20.7669
                          iot_N02_Ref_Sensitvity: 0.2760
                            iot_input_port1_flow: 0
                            iot_input_port2_flow: 0
                            iot_input_port3_flow: 0
                           iot_input_port_status: 0
                              iot_we_voltage_raw: 226.8767
                             iot_aux_voltage_raw: 229.8950
                             iot_N02_Ref_we_zero: 226
                            iot_N02_Ref_aux_zero: 228
                         iot_N02_Ref_we_zero_pcb: 231
                        iot_N02_Ref_aux_zero_pcb: 229
                              iot_N02_Ref_kelvin: 295.2793
    iot_n02_temperture_compensation_algrorithum3: 1
    iot_n02_temperture_compensation_algrorithum4: 0

[Dharmesh Joshi]

This the data, i have about 3.5K datasamples, so i have shown only 1

[Jon]

OK I'm starting to understand. Here is an approach:

% make up a little bit of example data just to show how to manipulate it
downloaded_data(1).Timestamp = '2022-05-08T11:30:47.7909324+00:00';
downloaded_data(2).Timestamp = '2022-05-08T11:40:27.4909628+00:00';
downloaded_data(3).Timestamp = '2022-05-08T11:50:36.2309354+00:00';

downloaded_data(1).iot_battery = 100;
downloaded_data(2).iot_battery = 104;
downloaded_data(3).iot_battery = 305;


% get values from array of structures to arrays
TimeStampRaw = {downloaded_data.Timestamp}; % cell array of character arrays
iot_battery = [downloaded_data.iot_battery] % array of doubles

% need to clean up time stamps, get rid of letter T in middle and +00:00 at
% end, use cellfun to perform operation on every element in the cell array
TimeStamp = cellfun(@(x)[x(1:10),' ',x(12:27)],...
    TimeStampRaw,'uniformOutput',false);

% convert TimeStamp strings to datetime array
t = datetime(TimeStamp)

% plot result
plot(t,iot_battery)

[Dharmesh Joshi]

Thanks I will give this a go.

[Jon]

OK Good luck, let me know how it goes

[Dharmesh Joshi]

Thanks

I have one last questions in relation to this topic.

I have temperture data , which i need it to be processed before it goes into a plot

iot_temperture=[(((downloaded_data.iot_temperture/65536)*165)-40)];
plot(t,iot_temperture)

But when i call this , i get the following error:

Error using  / 
Too many input arguments.

Error in n02 (line 18)
iot_temperture=[(((downloaded_data.iot_temperture/65536)*165)-40)];

[Jon]

Yes, the problem here is again that when accessing one field, in this case .iot_temperture, in an array of structures, in this case, downloaded_data, you must first turn the comma separated list into a vector. You do this by surrounding it with square brackets.

You also have a lot of unecessary round parenthesis.

So, modify your code above to:

iot_temperture=[downloaded_data.iot_temperture]/65536*165-40;

One editorial comment, I like your use of meaningful names for your variables, but you might want to rename your field .iot_temperture to .iot_temperature. That is, correct the spelling of temperature

[Dharmesh Joshi]

Thanks I got it working

One issue i noticed on the plots is that there is some blank spots. I presume at that plot is assume certain time stamp and if there is no data for that timestamp it showing blank space. Please see the attached image.

[Jon]

The blank spots probably correspond to points where the temperature value is nan (not a number), so as you say missing data.

Answer 3

[Dharmesh Joshi]

Is there away to confirm if there is a nan, as my device are updating every 1 minute, if there is any issue , there would be not entry for that minute or a perticular field would be 0.

I noticed the following warning

Warning: Error updating Text.

 String scalar or character vector must have valid interpreter syntax:
Temperature $^\circ \mathrm{C}$
 
> In defaulterrorcallback (line 12)
In matlab.graphics.axis.decorator.DatetimeRuler.get.TickLabelFormat
In matlab.graphics.axis.decorator/DatetimeRuler/format
In matlab.graphics.internal.makeNonNumeric
In matlab.graphics.internal.makeNonNumeric
In n02 (line 33) 

[Dharmesh Joshi]

It seems the x-axis is assumed 1 min internval, and if a peritcular sample is missing(not avilable ) it is missing that place and adding blank.

Does this sound possible?

[Jon]

The error you report above is just due to some problem in the formating of the text for your y axis in the plot command.

Do you know how to use the debugging tools?

If so you could set a breakpoint just before the call to plot the data and then you could look at the contents of iot_temperture

Alternatively you can just temporarily modify your code to remove the semicolon at end of the line where iot_temperature is calculated (as shown below) and you will see the values on your screen.

iot_temperture=[downloaded_data.iot_temperture]/65536*165-40 % remove semicolon to see output

If you don't know how to use the debugger, it is really an essential tool, here's an intro https://www.mathworks.com/help/matlab/matlab_prog/debugging-process-and-features.html

Finally, if you are still revising the program, you may want to change your variable names to correctly spell temperature. Currently you have variables/fields such as downloaded_data.iot_temperture, for me it would be nicer if that were correctly spelled (doesn't affect the calculation just better style) as downloaded_data.iot.temperature

[Dharmesh Joshi]

Hi Jon,

I am new to Matlab, so i am learning as i progresse.

My data structure is about 34K data points, so reading each point could be a bit an issue.

In regards to the spelling issue, that is the JSON from the API i am working on. I would not be able to correct this at this stage.

[Jon]

If you just wanted to know if there were any nan values in iot_temperture, you could insert this line just before you plot

disp(any(isnan(iot_temperture)))

If you have nan's then before the data is plotted you will see a 1 displayed on your console

[Dharmesh Joshi]

Thanks let me give that a go

[Dharmesh Joshi]

The output is 0

[Dharmesh Joshi]

Is there limit to how may data points can be on plot?

[Jon]

I don't think there is a limit to how many points can be plotted.

You may also have missing values in your plot if there are "not a time" values in the time vector. You can determine if there are any of these using the isnat function. So in a similar way as you checked if there were any nan values in the temperature vector you could check if there are any not a time values in the datetime vector using

iot_temperture=[downloaded_data.iot_temperture]/65536*165-40;
disp(any(isnat(t)))
plot(t,iot_temperature)

You will see a 1 displayed if there are any not a time values, othererwise it will display a zero.

To go further I strongly suggest you learn how to use the debug tool to set a breakpoint just before the plot and then you can look directly at the contents of those variables just before they are plotted.

[Dharmesh Joshi]

Hi Jon

After some debugging it seems that issue is related to the timestamp string being converted to timestamp object.

TimeStampRaw = {downloaded_data.Timestamp}; % cell array of character arrays
TimeStamp = cellfun(@(x)[x(1:10),' ',x(12:27)],...
    TimeStampRaw,'uniformOutput',false);
t = datetime(TimeStamp);
disp(any(isnat(t)))

This outputted a 1.

If i was to look at location 34803 for both TimeStampRaw and TimeStamp

This is output i am getting:

>> TimeStampRaw(34803)
ans =
  1×1 cell array
    {'2022-04-08T11:42:03.496687+00:00'}
>> t(34803)
ans = 
  datetime
   NaT

[Jon]

Great job isolating the problem!

It looks like, for whatever reason, the raw date strings sometimes have less decimal places. So, the + character appears sooner. Cutting of at character 27 can therefore sometimes include a + in the string which leads to the NaT.

You could just try using one less decimal place and modify to

TimeStamp = cellfun(@(x)[x(1:10),' ',x(12:26)],...
    TimeStampRaw,'uniformOutput',false);

You could also take a more robust approach and make a little function to clean up the strings like this:

...
% end, use cellfun to perform operation on every element in the cell array
TimeStamp = cellfun(@(x)cleanTime(x),...
    TimeStampRaw,'uniformOutput',false);
% convert TimeStamp strings to datetime array
t = datetime(TimeStamp)
% plot result
plot(t,iot_battery)

% put function to clean the time stamps at end of script, or nested inside of your main function
function tstmp = cleanTime(str)
% clean up time stamps, get rid of letter T in middle and +00:00 at
% end
parts = strsplit(str,{'T','+'});
tstmp = [parts{1},' ',parts{2}];
end


    

[Dharmesh Joshi]

Hi Jon

Yes, I noticed that small difference as well. I will incorporate the clean time Function. But with the previous method, is it really necessary to add the decimal point and value?

Answer 4

[Dharmesh Joshi]

Hi Jon

I got it working now.

Its ploting well.

At the moment, the data sample are per minute. how can i group all the data per hour and work out the mean or average for that hour?

[Jon]

That's great to hear.

I think to keep this thread clear it would be good to accept my answer rather than accepting your own final note (which should probably just be another comment) as an answer. I think it is possible to modify which answer you accept.

Rather than starting up a new thread here I will anwer your question regarding the grouping of data as a further comment under my original answer