Main Content


Add token type details to documents



updatedDocuments = addTypeDetails(documents) detects the token types in documents and updates the token details. The function adds type details to the tokens with unknown type only. To get the token types from updatedDocuments, use tokenDetails.


updatedDocuments = addTypeDetails(documents,Name,Value) specifies additional options using one or more name-value pairs.


Use addTypeDetails before using the lower, upper, and erasePunctuation functions as addTypeDetails uses information that is removed by these functions.


collapse all

Convert manually tokenized text into a tokenizedDocument object, setting the 'TokenizeMethod' option to 'none'.

str = ["For" "more" "information" "," "see" "" "."];
documents = tokenizedDocument(str,'TokenizeMethod','none')
documents = 

   7 tokens: For more information , see .

View the token details using the tokenDetails function.

tdetails = tokenDetails(documents)
tdetails=7×2 table
               Token               DocumentNumber
    ___________________________    ______________

    "For"                                1       
    "more"                               1       
    "information"                        1       
    ","                                  1       
    "see"                                1       
    ""          1       
    "."                                  1       

If you set 'TokenizeMethod' to 'none' in the call to the tokenizedDocument function, then it does not detect the types of the tokens. To add the token type details, use the addTypeDetails function.

documents = addTypeDetails(documents);

View the updated token details.

tdetails = tokenDetails(documents)
tdetails=7×3 table
               Token               DocumentNumber       Type    
    ___________________________    ______________    ___________

    "For"                                1           letters    
    "more"                               1           letters    
    "information"                        1           letters    
    ","                                  1           punctuation
    "see"                                1           letters    
    ""          1           web-address
    "."                                  1           punctuation

Input Arguments

collapse all

Input documents, specified as a tokenizedDocument array.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: 'TopLevelDomains',["com" "net" "org"] specifies the top-level domains "com", "net", and "org" for web address detection.

Top-level domains to use for web address detection, specified as a character vector, string array, or cell array of character vectors.

If you do not specify TopLevelDomains, then the function uses the output of the topLevelDomains function.

Example: ["com" "net" "org"]

Data Types: char | string | cell

Option to discard previously computed details and recompute them, specified as true or false.

Data Types: logical

Output Arguments

collapse all

Updated documents, returned as a tokenizedDocument array. To get the token details from updatedDocuments, use tokenDetails.

Version History

Introduced in R2018b