C++ - Tokenization Of a String

Heya Amigos !! Hope you are competing properly with each other and staying in a loop of competitive programming.

This article is not going to tell you on how to do competitive programming, rather we will be discussing one technique of string algorithm, that can come handy while writing some code.

So the technique is Tokenization.

What is Tokenization of a string?

Let me start with an example here, suppose we have a string as below,

 string s = "I love writing blogs on hashnode";

Tokenization is breaking the above string according to a delimiter, which is given by the user.

Suppose the delimiter is " ", i.e. blank space. Then the string will break at every blank space and six tokens will be generated as below,

I 
love 
writing 
blogs 
on 
hashnode

All the six tokens generated above are independent of each other and they are separate string objects. Many a time in competitive programming we need to generate such types of tokens in order to do some string manipulations.

How to perform this tokenization?

Tokenization can be easily performed by a function named strtok() inside the <cstring> header-file in C++.

Syntax:

char* strtok(char* str,const char* delimiter)

Parameters

• str − The contents of this string are modified and broken into smaller strings (tokens).
• delim − This is the C string containing the delimiters. These may vary from one call to another.

Return Value

This function returns a pointer to the first token found in the string. A null pointer is returned if there are no tokens left to retrieve.

A small example

Main Code

#include <iostream>
#include <cstring>
#include <vector>

using namespace std;

int main()
{
    // declaring a demo array
    char s[100] = "I love writing blogs on hashnode";

    // initializing a character pointer
    char* ptr = strtok(s," ");

    // initializing a dynamic array
    vector<char*> v;

    // looping over and storing successively
    while(ptr!=nullptr)
    {
        v.push_back(ptr);
        ptr = strtok(nullptr," ");
    }

    // console out the container 
    for(auto i=0;i<v.size();i++)
    {
        cout << v[i] << endl;
    }

    return 0;
}

Output:

I 
love 
writing 
blogs 
on 
hashnode

I hope that helps as a good concept refresher ⭐🎊