{"id":30900,"date":"2025-08-29T12:52:20","date_gmt":"2025-08-29T12:52:20","guid":{"rendered":"https:\/\/parashift.io\/?p=27877"},"modified":"2025-11-30T22:47:20","modified_gmt":"2025-11-30T22:47:20","slug":"generative-vs-discriminative-ai-how-parashift-makes-document-automation-smarter","status":"publish","type":"post","link":"https:\/\/parashift.ai\/en\/generative-vs-discriminative-ai-how-parashift-makes-document-automation-smarter\/","title":{"rendered":"Generative vs. Discriminative AI: How Parashift makes Document Automation smarter"},"content":{"rendered":"\n

This article was written by our ML team.<\/em><\/p>\n\n\n\n

This is the beginning of a multi part blog mini-series where we shine some light onto how we use ML \/ AI at Parashift<\/a> for solving intelligent document understanding problems.<\/p>\n\n\n\n

The first two parts are a short introduction into general key topics and concepts in terms of AI and not specifically focused on Parashift. We start by discussing the difference between discriminative<\/strong> and generative models<\/strong>. In the third part we will talk about our current efforts related to grounding LLM answers in documents<\/strong>. The term grounding in this context means the ability to link the answer of an LLM to explicit tokens in the document. The fourth part touches on more use cases of both methods in our product and finally the last part is as an outlook into where we want to go next.<\/p>\n\n\n\n

\u00a0<\/div>\n\n\n\n

Discriminative Models vs Generative AI<\/h4>\n\n\n\n

First we examine two different types of AI models and start with an introduction to highlight conceptual differences. As we will see those differences influence how and where we want to use those models in practice.<\/p>\n\n\n\n

\u00a0<\/div>\n\n\n\n
Discriminative Models<\/h5>\n\n\n\n

These models learn how to discriminate \/ differentiate between “things”. Think of them as ‘labeling machines’ to which you show a thing and the machine will assign a specific label and a number to it. The label(s) determine what that thing was classified as and the number reflects the confidence in the label. A common example is an image classifier that identifies animals. If this model is trained on, say, 100 different animal species all it can do is look at images and assign the best matching label. The model can not come up with a new label but is restricted (and biased) by the labels it was exposed to during training. Such models are also referred to as classifiers.<\/p>\n\n\n\n

Many types of IDP tasks can be formulated in a way that a discriminative models can be trained to solve them. The approach always boils down to assigning a label to something. Use cases differ in the kind of label is assignd and to what “thing” they are assigned to. In the following examples we continuously increase the granularity of those label assignments and see how this solves different tasks.<\/p>\n\n\n\n

\u00a0<\/div>\n\n\n\n
Document Classification (one label per document)<\/strong><\/h6>\n\n\n\n

The most straight forward use case where during training the model is fed documents (and labels) of different types to learn how to distinguish them. Here the labels<\/code> are all the document-types we need to distinguish (i.e. ‘invoice’, ‘passport’, ‘gym membership card’ etc.). Despite the task being well defined and straight forward there is quite a lot of creative freedom on how the model is trained to perform the task. That could be on visual features, on textual features or a combination of both, to name a few design decisions.<\/p>\n\n\n\n

\"Document<\/figure>\n\n\n\n
\u00a0<\/div>\n\n\n\n
Page Sequence Separation (one label per page)<\/strong><\/h6>\n\n\n\n

Now we start from an ordered sequence of pages and the task is to evaluate which pages belong together to form an actual document. This is useful if one scans physical mail, resulting in one large PDF containing all documents. We opted for an approach to train a model that predicts two labels: this is a first page or this is a last page. Based on these predictions we can then infer where to split the page sequence into individual documents. See below a schematic example of a page sequence consisiting of 6 pages that results in 3 individual documents.<\/p>\n\n\n\n

\"Page<\/figure>\n\n\n\n
\u00a0<\/div>\n\n\n\n
Information Extraction (one label per token)<\/strong><\/h6>\n\n\n\n

Now lets see how to apply discriminative models to extract information such as the sender address, payable amounts, or delivery dates etc. If we look at this problem through the lens of a classifier, we must build a system that assigns labels to only specific parts of the document. As an example it can assign the label company name to the document text “Parashift AG”. As an additional step further one can train a model to predict which of these tokens belong together (so called link prediction). So the model has to decide if “Parashift AG” is one entity or if it is two separate ones like: “Parashift” & “AG”.<\/p>\n\n\n\n

\u00a0<\/div>\n\n\n\n
Embeddings and Tokenization<\/strong><\/h6>\n\n\n\n

Now we know how to conceptually formulate those IDP tasks for a discriminative model. There are at least two more fundamental concepts to understand to get a clearer picture of the workings of such models: embeddings & tokenization. Since the task boils down to “assign label to thing” we need a way to meaningfully represent the “things” for the models to distinguish them (where thing is a document, a page, or tokens of documents). This is where embeddings come into play.<\/p>\n\n\n\n

 <\/p>\n\n\n\n

Embeddings<\/strong><\/p>\n\n\n\n

AI models do not work with what we humans understand as images, audio, document, text or tokens directly, they only understand numbers. The act of transforming real world objects into “a list of numbers” is called embedding or vectorization. The important aspect is to figure out a way such that those numbers have meaning for the problem you attempt to solve.<\/p>\n\n\n\n

\u00a0<\/div>\n\n\n\n

RGB as example<\/strong><\/p>\n\n\n\n

This section is not tied in any way to IDP, but just a generic example how we can encode information into a list of numbers. A nice visual example of using numbers to represent information is colors. On your screen the color of each pixel can be represented as three RGB-values (where R: red, G: green, B: blue). We can look at this as a type of embedding consisting of 3 numbers representing the [R, G, B] values. For our example here we decide that each number takes on values from 0..1 (pixels usually are represented with numbers from 0..255, but we make our lives easier with 0..1). “Red” is represented as [1, 0, 0], green as [0, 1, 0] and [0, 0, 1] corresponds to blue.<\/p>\n\n\n\n

This enables us to express ANY color as a combination of these 3 numbers, for example yellow = red + green -> [1, 1, 0]. We pick the RGB example only because it is easy to visualize for humans (as color). Modern LLMs like ChatGPT, Claude or Llama use embeddings that are significantly larger than our example, typically in the range of a few thousand numbers per embedding!<\/p>\n\n\n\n

\n
\u00a0<\/div>\n\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n\n
\u00a0<\/div>\n<\/div>\n\n\n\n

With this in mind assume we build a token embedding model, and the model is only allowed to use 3 numbers for representing the meaning of a token (such that it is easy to visualize). We can see the output of the model below and that ‘addresses’ have a reddish tone, numbers appear green and dates have a blue note to them. It is unclear how the model learned this, but it is pure observation at this point. Upon further inspection we find exceptions to that observation, namely both the street numbers and zip codes in addresses are orange, not green. What this means is that our model is smart enough to change the embedding of a token based on its contex (= the tokens surrounding it). This means the embeddings are dynamic. In contrast static embeddings are fixed lookups where for example the token 9155 will always be assigned the same embedding, no matter the context.<\/p>\n\n\n\n

We will see how at Parashift we leverage static and dynamic embeddings when it comes to grounding LLM answers in the actual document in the third part of this blog series. By grounding we mean “to identify which tokens in the document make up the answer” which allows to provide users with exact coordinates that belong to the answer (down to the token level, not just paragraph level). Static embeddings are limited in their expressivenes as they for example fail to distinguish homonyms (words that have the same spelling but different meaning based on context). An example would be the word lead in: – My action lead to a consequence – I don’t like lead in my drinking water.<\/p>\n\n\n\n

\"AI<\/figure>\n\n\n\n
\u00a0<\/div>\n\n\n\n

Tokenization<\/strong><\/p>\n\n\n\n

Another crucial step for turning information extraction into a classification problem is to divide the document into smaller parts. This can be paragraphs, lines, words or tokens. Then we can train models to assign labels to those ‘parts’. The distinction between a word and a token is a bit blurry here but as a first approximation think of a token as “short words or sub-words”.<\/p>\n\n\n\n

Dividing a document or text into tokens is done by a tokenizer and it is important to note that there is no agreed upoon method on how to best split a text into tokens. Below is a visualization on how the GPT-4 tokenizer splits up an example sentence. Tokens are visualized as colored rectangles. You can also go to this interactive tool<\/a> <\/em><\/strong>and play with different tokenizers yourself. Once we have the document tokenized we can assign labels to them. One can see how assigning labels such as sender-name, sender-street, sender-house-number can finally serve for information extraction.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

To close the loop back to information extraction we can look at the embeddings above one can hopefully imagine to train an other model that takes those embeddings and uses them to assign labels to tokens one is interested in. For this we train a separate model on different training examples to figure out the relationship between the embeddings and the task the model is actually required to solve. It is easy to see that with such an embedding model it would be easy to extract information about “addresses” or “dates”. If we are after information different from these two cases we are probably out of luck since the model cannot distinguish most of the other tokens from each other (they all have the same color).<\/p>\n\n\n\n

This highlights the need for training embeddings that are generic enough for solving a wide range of potential downstream tasks but at the same time capture enough information to make it easier on the following classification models.<\/p>\n\n\n\n

\u00a0<\/div>\n\n\n\n

Summary of discriminative models<\/h4>\n\n\n\n

These models come with a few pros and cons when using them to solve tasks in real world applications.<\/p>\n\n\n\n

\u00a0<\/div>\n\n\n\n
The Pros<\/h6>\n\n\n\n