I have multiple nested JSON files that I need to combine into single CSV file using MATLAB

I need to combine 200 nested JSON files. Each JSON file consist of more than thousand data points. I am sharing here a sample of two data points each file contains to help in understanding how the data looks like. Let me know how can I achieve this task with MATLAB.

{"dataPointId":"0012bac3-f926-45ec-af61-af2e63a9ac0b","journeyId":"8ebce7ed0dac18983570c3b66bc931ac1ba094a8","capturedTimestamp":"2019-12-19T18:23:25.000-0600","location":{"latitude":45.275407,"longitude":-93.735603,"geohash":"cbj3df","postalCode":"55362","regionCode":"MN","countryCode":"US"},"metrics":{"speed":118.65,"heading":302.0},"vehicle":{"make":"CHEVROLET","model":"SILVERADO_HD","year":2016,"status":{"ignitionStatus":"MID_JOURNEY"}}}

{"dataPointId":"008c6966-35b0-4697-95ce-16b5f81e9c9d","journeyId":"4735c2b19fb44880e80b2052dc2f2f6a65b22a1f","capturedTimestamp":"2019-12-19T18:15:02.000-0600","location":{"latitude":45.295728,"longitude":-93.802924,"geohash":"cbj39m","postalCode":"55362","regionCode":"MN","countryCode":"US"},"metrics":{"speed":0.0,"heading":182.0},"vehicle":{"make":"CHEVROLET","model":"IMPALA","year":2016,"status":{"ignitionStatus":"MID_JOURNEY"}}}

[Reeshabh Ranjan]

@Salman Ahmad Could you add 2 sample JSON files each containing atleast 2 datapoints to be merged? It will help me understand the issue better.

[Salman Ahmad]

Thanks for your reply. As you asked I am sharing here two data points from two different JSON files.

File # 1 name: part-00001-27537ebc-6753-4447-8fa5-7f1ad9be10e4.c000

Datapoints:

{"dataPointId":"0018d399-f01c-442e-bdd5-927515b425a9","journeyId":"a3d849be148f0cf082894d85be82cd518512147f","capturedTimestamp":"2019-12-21T11:09:56.000-0600","location":{"latitude":45.220131,"longitude":-93.503014,"geohash":"cbj922","postalCode":"55327","regionCode":"MN","countryCode":"US"},"metrics":{"speed":67.96,"heading":178.0},"vehicle":{"make":"CHEVROLET","model":"SILVERADO_HD","year":2017,"status":{"ignitionStatus":"MID_JOURNEY"}}}
{"dataPointId":"00b2e29b-179b-4b21-ab55-f82097a49078","journeyId":"486b040071859fbb24f82ebc8a479f3d0ef71bcb","capturedTimestamp":"2019-12-21T11:38:05.000-0600","location":{"latitude":45.244178,"longitude":-93.516389,"geohash":"cbj3ru","postalCode":"55374","regionCode":"MN","countryCode":"US"},"metrics":{"speed":49.53,"heading":269.0},"vehicle":{"make":"CHEVROLET","model":"SUBURBAN","year":2018,"status":{"ignitionStatus":"MID_JOURNEY"}}}

File # 2 name: part-00002-27537ebc-6753-4447-8fa5-7f1ad9be10e4.c000

Datapoints:

{"dataPointId":"005a06d6-c0db-4071-814a-dbabfe955b97","journeyId":"0907355fe39773e7cfa4d0fdf5bbb939179dc1f5","capturedTimestamp":"2019-12-21T11:48:34.000-0600","location":{"latitude":45.305636,"longitude":-93.793677,"geohash":"cbj39x","postalCode":"55362","regionCode":"MN","countryCode":"US"},"metrics":{"speed":0.0,"heading":297.0},"vehicle":{"make":"CHEVROLET","model":"SILVERADO","year":2017,"status":{"ignitionStatus":"MID_JOURNEY"}}}
{"dataPointId":"01b97a66-5fb4-4571-b0f0-8b13c854bb37","journeyId":"611ffbc59a2e52e539b6c2fc20e166dccc774fb2","capturedTimestamp":"2019-12-21T11:16:25.000-0600","location":{"latitude":45.353868,"longitude":-93.819876,"geohash":"cbj610","postalCode":"55308","regionCode":"MN","countryCode":"US"},"metrics":{"speed":80.64,"heading":359.0},"vehicle":{"make":"CHEVROLET","model":"CRUZE","year":2016,"status":{"ignitionStatus":"MID_JOURNEY"}}}

Let me know if you need anymore information.

[Reeshabh Ranjan]

@Salman Ahmad, I assume the root of the JSON file is a JSON array? Like is it arranged in the following way:

[
    {"dataPointId":"0018d399-f01c-442e-bdd5-927515b425a9","journeyId":"a3d849be148f0cf082894d85be82cd518512147f","capturedTimestamp":"2019-12-21T11:09:56.000-0600","location":{"latitude":45.220131,"longitude":-93.503014,"geohash":"cbj922","postalCode":"55327","regionCode":"MN","countryCode":"US"},"metrics":{"speed":67.96,"heading":178.0},"vehicle":{"make":"CHEVROLET","model":"SILVERADO_HD","year":2017,"status":{"ignitionStatus":"MID_JOURNEY"}}},
    {"dataPointId":"00b2e29b-179b-4b21-ab55-f82097a49078","journeyId":"486b040071859fbb24f82ebc8a479f3d0ef71bcb","capturedTimestamp":"2019-12-21T11:38:05.000-0600","location":{"latitude":45.244178,"longitude":-93.516389,"geohash":"cbj3ru","postalCode":"55374","regionCode":"MN","countryCode":"US"},"metrics":{"speed":49.53,"heading":269.0},"vehicle":{"make":"CHEVROLET","model":"SUBURBAN","year":2018,"status":{"ignitionStatus":"MID_JOURNEY"}}}
]

And same for the other one:

[
    {"dataPointId":"005a06d6-c0db-4071-814a-dbabfe955b97","journeyId":"0907355fe39773e7cfa4d0fdf5bbb939179dc1f5","capturedTimestamp":"2019-12-21T11:48:34.000-0600","location":{"latitude":45.305636,"longitude":-93.793677,"geohash":"cbj39x","postalCode":"55362","regionCode":"MN","countryCode":"US"},"metrics":{"speed":0.0,"heading":297.0},"vehicle":{"make":"CHEVROLET","model":"SILVERADO","year":2017,"status":{"ignitionStatus":"MID_JOURNEY"}}},
    {"dataPointId":"01b97a66-5fb4-4571-b0f0-8b13c854bb37","journeyId":"611ffbc59a2e52e539b6c2fc20e166dccc774fb2","capturedTimestamp":"2019-12-21T11:16:25.000-0600","location":{"latitude":45.353868,"longitude":-93.819876,"geohash":"cbj610","postalCode":"55308","regionCode":"MN","countryCode":"US"},"metrics":{"speed":80.64,"heading":359.0},"vehicle":{"make":"CHEVROLET","model":"CRUZE","year":2016,"status":{"ignitionStatus":"MID_JOURNEY"}}}
]

[Salman Ahmad]

I don't think it is a JSON array because there is no square bracket surrounding the data in any file.

[Reeshabh Ranjan]

So the files just contain a list of JSON objects and each JSON object will be a row in the converted CSV, right?

[Salman Ahmad]

Yes, exactly.

[Salman Ahmad]

But we have to keep in mind that some of the objects in each file are under multi level nesting of strings that needs to be flattened to make a single header row in a table format.

[Rik]

Then you need to write a function that will detect if an object needs to be flattened and does that. Then it should be as simple as concatenating the char arrays.

[Salman Ahmad]

This is the part where I am struggling the most i.e. developing a function/code to read all the files at the same time and combine their data in a tabular format. Because I don't have a coding background so that's where I thought MATLAB could give me some advance/easy options to get this job done otherwise I would have to spend alot of my time learning some programming languages that can reduce my effort.

[Reeshabh Ranjan]

@Salman Ahmad see if this helps: https://in.mathworks.com/matlabcentral/answers/29837-processing-files-using-a-for-loop