Is there a best practice for enconding URI on the REST API communication from MATLAB?

Hi, I'm working on a project, where JSon::API-based communication is happening from MATLAB.

I'm getting to the point, where the URI, which I passing to the 'send' method gets encoded by MATLAB. Is there a way to disable this conversion?

Particularly, here is my code:

            request = matlab.net.http.RequestMessage();
            request.Method = options.method;
            request.Header = [ ...
                matlab.net.http.field.ContentTypeField( 'application/json' ), ...
                matlab.net.http.field.AcceptField( 'application/json' ), ...
                matlab.net.http.field.AuthorizationField('Authorization', ['Bearer ' PAT]), ...
                ];
            httpOptions = matlab.net.http.HTTPOptions('ResponseTimeout', remoteOperationTimeOut);

            [response, completedRequest] = request.send(finalPoint, httpOptions)

where finalPoint is 'https://<xxx-server>/rest/v1/projects/NickTest/spaces/Testing/documents/Big document for testing REST/parts'

Note the spaces in the URL.

When I check the completedRequest, I see that this URL was post-processed:

RequestMessage with properties:

         Method: GET
    RequestLine: 'GET /rest/v1/projects/NickTest/spaces/Testing/documents/Big%20document%20for%20testing%20REST/parts HTTP/1.1'
         Header: [1×7 matlab.net.http.HeaderField]
           Body: [0×0 matlab.net.http.MessageBody]
      Completed: 1
      

note, that the RequestLine has converted spaces to the '%20'. That's all fine so far.

However, I'm working with the pagination feature, and server returns to me some links how to request next page of the response.

And this links already contain %20 instead of the space, so passing to 'request.send(newURL)' makes a double encoding, and it works wrongly (not found).

For now, before sending the consequent request, I'm converting back %20 to spaces:

      nextPageLink = strrep(addition.links.next, '%20', ' ');

but this sounds wrong to me - on the client I don't want to alter any URLs, that server returns to me.

Is there any flag or option to prevent double conversion, and leaving URIs as-is?

Thank you in advance!

Answer 1

[Shubh]

Hi Nick,

In MATLAB, when using the "matlab.net.http.RequestMessage" class for HTTP requests, URL encoding (such as converting spaces to %20) is a standard procedure to ensure that URLs are valid and conform to the appropriate specifications. However, as you've noticed, this can cause issues when you receive a URL from the server that's already encoded, and MATLAB tries to re-encode it.

Unfortunately, MATLAB's HTTP interface doesn't provide a built-in option to disable automatic URL encoding in the send method of "matlab.net.http.RequestMessage". The send method automatically applies encoding to ensure that the URL is valid according to standard URL encoding rules.

To handle this, you can manually decode the URL received from the server before passing it to the send method. Although it might seem counterintuitive, this ensures that MATLAB will encode the URL only once.

Your current approach using "strrep" to replace "%20" with spaces is one way to do it, but a more robust method is to use MATLAB's "urldecode" function, which can decode all percent-encoded characters, not just spaces. This is useful if your URLs contain other encoded characters.

Here's how you can modify your code to include URL decoding:

% Your existing setup
request = matlab.net.http.RequestMessage();
request.Method = options.method;
request.Header = [ ...
    matlab.net.http.field.ContentTypeField('application/json'), ...
    matlab.net.http.field.AcceptField('application/json'), ...
    matlab.net.http.field.AuthorizationField('Authorization', ['Bearer ' PAT]), ...
];
httpOptions = matlab.net.http.HTTPOptions('ResponseTimeout', remoteOperationTimeOut);

% Sending the first request
[response, completedRequest] = request.send(finalPoint, httpOptions);

% Assuming you receive nextPageLink from the server
nextPageLink = response.Body.Data.nextPageLink; % Example, adjust according to your actual response structure

% Decode the URL received from the server
decodedNextPageLink = urldecode(nextPageLink);

% Send the next request
[nextResponse, nextCompletedRequest] = request.send(decodedNextPageLink, httpOptions);

This code snippet assumes that you're receiving nextPageLink in the response body and it's already URL encoded. The "urldecode" function will decode it back to its original form, ensuring that when it's passed to "request.send", MATLAB only encodes it once.

Hope this helps!

[Nick]

Thank you, @Shubh, for the confirmation and tip for the urldecode!

[Nick]

FYI: @Shubh, I found that

decodedNextPageLink = urldecode(nextPageLink);

is not a generic solution! Problem appears if the "nextPageLink" has spaces or other special characters in the parameters part.

Meaning, that if I have link like this:

'https://<xxx-server>/rest/v1/projects/NickTest/spaces/Testing/documents/Big%20document%20for%20testing%20REST/parts?fields%5Bprojects%5D=id%2C%20name%2C%20description&sort=name"

I can't do urldecode for the whole string, otherwise there will be exception (see below).

From this call and exception (a bit different URL, but it's not important):

K>> request.send(decodedEndpoint, httpOptions)

Invalid URL: 'https://xxx-server/rest/v1/projects?fields[projects]=id,name,description&page[number]=2&sort=name'.

Error in matlab.net.http.internal.HTTPConnector/set.URI (line 614)

obj.Connection.URL = char(uri); % this may throw if uri is bad

Error in matlab.net.http.internal.HTTPConnector (line 169)

obj.URI = uri;

Error in matlab.net.http.RequestMessage/send (line 456)

connector = HTTPConnector(completedURI, options, [], obj.Header);

I found the matlab.net.URI.assumeHost(str), which translates input URL to a struct, where the main part of the URL is stored as EncodedPath, and it seems to be result of encoding of the Path, and EncodedQuery, which doesn't seem to be processed, and left as is.

I mean, I know how to workaround the issue, but if you have influense on the API of MATLAB, it will be better to provide some helpers for such cases :-)