Embedding-Based Candidate Selection
Posted: Sat May 24, 2025 10:12 am
To enable fuzzy matching with any feature, we expanded the capabilities in both the candidate selection and matching steps.
The ultimate goal is to enable fuzzy matching with any subset of features, chosen by the user. But before getting to the matching step, we first need to create clusters of close candidates based on those features. And to do that, we need a flexible approach that can find similar candidates based on any subset of features.
To achieve this, we developed an embedding-based candidate selection approach. Our afghanistan phone number list method first passes all records into an embedding model, in this case the open-source model ByT5, to create embeddings, or vector representations, of each record. These vector representations capture meaningful information about the record, and vectors that are close geometrically indicate that the corresponding records are similar. For example, the vector representation for “James Smith, California, USA” will be geometrically close to the vector representation of “James R. Smith, CA, United States of America”.
For increased computational efficiency, we first quantized the embedding model, reducing the precision of the model weights from 32 bits down to 8 bits. This significantly improves embedding generation speed. After all embeddings are generated, we then apply local sensitivity hashing, which generates hash buckets, or clusters, of close embeddings.
The ultimate goal is to enable fuzzy matching with any subset of features, chosen by the user. But before getting to the matching step, we first need to create clusters of close candidates based on those features. And to do that, we need a flexible approach that can find similar candidates based on any subset of features.
To achieve this, we developed an embedding-based candidate selection approach. Our afghanistan phone number list method first passes all records into an embedding model, in this case the open-source model ByT5, to create embeddings, or vector representations, of each record. These vector representations capture meaningful information about the record, and vectors that are close geometrically indicate that the corresponding records are similar. For example, the vector representation for “James Smith, California, USA” will be geometrically close to the vector representation of “James R. Smith, CA, United States of America”.
For increased computational efficiency, we first quantized the embedding model, reducing the precision of the model weights from 32 bits down to 8 bits. This significantly improves embedding generation speed. After all embeddings are generated, we then apply local sensitivity hashing, which generates hash buckets, or clusters, of close embeddings.