sentence-transformers How to use Daria-best/stella_en_400M_v5_neurips_papers_fine-tuned with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Daria-best/stella_en_400M_v5_neurips_papers_fine-tuned", trust_remote_code=True)
sentences = [
"Classifier reduction techniques for improving prediction accuracy",
"INTRODUCTION While neural networks have proved a good tool for processing static patterns, classi fying sequential information has remained a challenging task. The problem involves recognizing patterns in a time series of vectors, which requires forming a good inter nal representation for the sequences. Several researchers have proposed extending the self-organizing feature map (Kohonen 1989, 1990), a highly successful static pattern classification method, to sequential information (Kangas 1991; Samara bandu and Jakubowicz 1990; Scholtes 1991). Below, three of the most recent of these networks are briefly described. The remainder of the paper focuses on a new architecture designed to overcome the shortcomings of these approaches. 578 Daniel L. James, Risto Miikkulainen Recently, Chappel and Taylor (1993) proposed the Temporal Kohonen Map (TKM) architecture for classifying sequences. The TKM keeps track of the activation his tory of each node by updating a value called leaky integrator potential, inspired by the membrane potential in biological neural systems. The activity of a node depends both on the current input vector and the previous input vectors, represented by the node's potential. A given sequence is processed by mapping one vector at a time, and the last winning node serves to represent the entire sequence. This way, there needs to be a separate node for every possible sequence, which is a disadvantage when the number of sequences to be classified is large. The TKM also suffers from loss of context. Which node wins depends almost entirely upon the most recent input vectors. For example, the string baaaa would most likely map to the same node as aaaaa, making the approach applicable only to short sequences. The SOFM-S network proposed by van Harmelen (1993) extends TKM such that the activity of each map node depends on the current input vector and the past activation of all map nodes. The SOFM-S is an improvement of TKM in that con textual information is not lost as quickly, but it still uses a single node to represent a sequence. The TRACE feature map (Zandhuis 1992) has two feature map layers. The first layer is a topological map of the individual input vectors, and is used to generate a trace (i.e. path) of the input sequence on the map . The second layer then maps the trace pattern to a single node. In TRACE, the sequences are represented by distributed patterns on the first layer, potentially allowing for larger capacity, but it is difficult to encode sequences where the same vectors repeat, such as baaaa. All a-vectors would be mapped on the same unit in the first layer, and any number of a-vectors would be indistinguishable. The architecture described in this paper, SARDNET (Sequential Activation Re tention and Decay NETwork), also uses a subset of map nodes to represent the sequence of vectors. Such a distributed approach allows a large number of repre sentations be \"packed\" into a small map-like sardines. In the following sections, we will examine how SARDNET differs from conventional self-organizing maps and how it can be used to represent and classify a large number of complex sequences. 2 THE SARDNET ARCHITECTURE Input to SARDNET consists of a sequence of n-dimensional vectors S V I, V 2 , V 3 , ... , VI (figure 1). The components of each vector are real values in the interval [0,1]. For example, each vector might represent a sample of a speech signal in n different frequencies, and the entire sequence might constitute a spoken word. The SARDNET input layer consists of n nodes, one for each component in the input vector, and their values are denoted as A (aI, a2, a3, ... , an). The map consists of m x m nodes with activation Ojk , 1 j, k m. Each node has an n-dimensional input weight vector Wjk, which determines the node's response to the input activation. In a conventional feature map network as well as in SARDNET, each input vector is mapped on a particular unit on the map, called the winner or the maximally responding unit. In SARDNET, however, once a node wins an input, it is made SARDNET: A Self-Organizing Feature Map for Sequences 579 Sequence of Input vectors S Previous winners Input weight vector wJk.l Winning unit jlc Figure 1: The SARDNET architecture. A sequence of input vectors activates units on the map one at a time. The past winners are excluded from further competition, and their activation is decayed gradually to indicate position in the sequence. INITIALIZATION: Clear all map nodes to zero. MAIN LOOP: While not end of seihence 1. Find unactivated weight vector t at best matches the input. 2. Assign 1.0 activation to that unit. 3. Adjust weight vectors of the nodes in the neighborhood. 4. Exclude the winning unit from subseent competition. S. Decrement activation values for all ot er active nodes. RESULT: Sequence representation activated nodes ordered by activation values Table 1: The SARDNET training algorithm. uneligible to respond to the subsequent inputs in the sequence. This way a different map node is allocated for every vector in the sequence. As more vectors come in, the activation of the previous winners decays. In other words, each sequence of length 1 is represented by 1 active nodes on the map, with their activity indicating the order in which they were activated. The algorithm is summarized in table 1. Assume the maximum length ofthe sequences we wish to classify is I, and each input vector component can take on p possible values. Since there are pn possible input vectors, Ipn map nodes are needed to represent all possible vectors in all possible positions in the sequence, and a distributed pattern over the Ipn nodes can be used to represent all pnl different sequences. This approach offers a significant advantage over methods in which pnl nodes would be required for pnl sequences. The specific computations of the SARDNET algorithm are as follows: The winning node (j, k) in each iteration is determined by the Euclidean distance Djk of the 580 Daniel L. James, Risto Miikkulainen input vector A and the node 's weight vector W jk: The unit with the smallest distance is selected as the winner and activated with 1.0. The weights of this node and all nodes in its neighborhood are changed according to the standard feature map adaptation rule: where a denotes the learning rate. As usual, the neighborhood starts out large and is gradually decreased as the map becomes more ordered. As the last step in processing an input vector, the activation 7]jk of all active units in the map are decayed proportional to the decay parameter d: As in the standard feature map , as the weight vectors adapt, input vectors gradually become encoded in the weight vectors of the winning units. Because weights are changed in local neighborhoods, neighboring weight vectors are forced to becom e as similar as possible, and eventually the network forms a topological layout of the input vector space. In SARDNET, however, if an input vector occurs multiple times in the same input sequence, it will be represented multiple times on the map as well. In other words, the map representation expands those areas of the input space that are visited most often during an input sequence. 3 EXPERIMENTS SARDNET has proven successful in learning and recognizing arbitrary sequences of binary and real numbers , as well as sequences of phonemic representations for English words. This section presents experiments on mapping three-syllable words. This data was selected because it shows how SARDNET can be applied to complex input derived from a real-world task. 3.1 INPUT DATA The phonemic word representations were obtained from the CELEX database of the Max Planck Institute for Psycholinguistics and converted into International Pho netic Alphabet (IPA)-compliant representation, which better describes similarities among the phonemes. The words vary from five to twelve phonemes in length. Each phoneme is represented by five values: place, manner, sound, chromacity and sonor ity. For example , the consonant p is represented by a single vector (bilabial, stop, unvoiced, nil, nil), or in terms of real numbers, (.125, .167, .750,0,0). The diph thong sound ai as in \"buy\" , is represented by the two vectors (nil, vowel, voiced, front, low) and (nil, vowel , voiced, front-center, hi-mid), or in real numbers , There are a total of 43 phonemes in this data set, including 23 consonants and 20 vowels. To represent all phonemic sequences of length 12, TKM and SOFM-S would SARDNET: A Self-Organizing Feature Map for Sequences 581 Figure 2: Accuracy of SARDNET for different map and data set sizes. The accuracy is measured as a percentage of unique representations out of all word sequences. need to have 4512 6.919 map nodes, whereas SARDNET would need only 45 x 12 540 nodes. Of course, only a very small subset of the possible sequences actually occur in the data. Three data sets consisting of 713,988, and 1628 words were used in the experiments. If the maximum number of occurrences of phoneme i in any single sequence is Cj I then the number of nodes SARDNET needs is C L:o Cj I where N is the number of phonemes . This number of nodes will allow SARDNET to map each phoneme in each sequence to a unit with an exact representation of that phoneme in its weights. Calculated this way, SARDNET should scale up very well with the number of words: it would need 81 nodes for representing the 713 3.2 DENSENESS AND ACCURACY A series of experiments with the above three data sets and maps of 16 to 81 nodes were run to see how accurately SARDNET can represent the sequences. Self-organization was quite fast: each simulation took only about 10 epochs, with a 0.45 and the neighborhood radius decreasing gradually from 5-1 to zero. Fig ure 2 shows the percentage of unique representations for each data set and map SARDNET shows remarkable representational power: accuracy for all sets is better than 97.7, and SARDNET manages to pack 1592 unique representations even on the smallest 16-node map. Even when there are not enough units to represent each phoneme in each sequence exactly, the map is sometimes able to \"reuse\" units to represent multiple similar phonemes . For example, assume units with exact representations for the phonemes a and b exist somewhere on the map, and the input data does not contain pairs of sequences such as aba-abb, in which it is crucial to distinguished the second a from the second b. In this case, the second occurrence of both phonemes could be represented by the same unit with a weight vector that is the average of a and b. This is exactly what the map is doing: it is finding the most descriptive representation of the data, given the available resources. 582 Daniel L. James, Risto Miikkulainen Note that it would be possible to determine the needed C L:f:o Cj phoneme representation vectors directly from the input data set, and without any learning or a map structure at all, establish distributed representations on these vectors with the SARDNET algorithm. However, feature map learning is necessary ifthe number of available representation vectors is less than C. The topological organization of the map allows finding a good set of reusable vectors that can stand for different phonemes in different sequences, making the representation more efficient. 3.3 REPRESENTING SIMILARITY Not only are the representations densely packed on the map, they are also descriptive in the sense that similar sequences have similar representations. Figure 3 shows the final activation patterns on the 36-unit, 713-word map for six example words. The first two words, \"misplacement\" and \"displacement,\" sound very similar, and are represented by very similar patterns on the map. Because there is only one m in \"displacement\" , it is mapped on the same unit as the initial m of \"misplacement.\" Note that the two IDS are mapped next to each other, indicating that the map is indeed topological, and small changes in the input cause only small changes in the map representation. Note also how the units in this small map are reused to represent several different phonemes in different contexts. The other examples in figure 3 display different types of similarities with \"mis placement\". The third word, \"miscarried\", also begins with \"mis\", and shares that subpart of the representation exactly. Similarly, \"repayment\" shares a similar tail and \"pessimist\" the subsequence \"mis\" in a different part or the word. Because they appear in a different context, these subsequences are mapped on slightly different units, but still very close to their positions with \"misplacement.\" The last word, \"burundi\" sounds very different, as its representation on the map indicates. Such descriptive representations are important when the map has to represent in formation that is incomplete or corrupted with noise. Small changes in the input sequence cause small changes in the pattern, and the sequence can still be recog nized. This property should turn out extremely important in real-world applications of SARDNET, as well as in cognitive science models where confusing similar pat terns with each other is often plausible behavior. 4 DISCUSSION AND FUTURE RESEARCH Because the sequence representations on the map are distributed, the number of possible sequences that can be represented in m units is exponential in m, instead of linear as in most previous sequential feature map architectures. This denseness together with the tendency to map similar sequences to similar representations should turn out useful in real-world applications, which often require scale-up to large and noisy data sets. For example, SARDNET could form the core of an isolated word recognition system. The word input would be encoded in duration normalized sequences of sound samples such as a string of phonemes, or perhaps representations of salient transitions in the speech signal. It might also be possible to modify SARDNET to form a more continuous trajectory on the map so that SARDNET itself would take care of variability in word duration. For example, a SARDNEf : A Self-Organizing Feature Map for Sequences 583 Figure 3: Example map representations. sequence of redundant inputs could be reduced to a single node if all these inputs fall within the same neighborhood. Even though the sequence representations are dense, they are also descriptive. Cat egory memberships are measured not by labels of the maximally responding units, but by the differences in the response patterns themselves. This sort of distributed representation should be useful in cognitive systems where sequential input must be mapped to an internal static representation for later retrieval and manipula tion. Similarity-based reasoning on sequences should be easy to implement, and the sequence can be easily recreated from the activity pattern on the map. Given part of a sequence, SARDNET may also be modified to predict the rest of the sequence. This can be done by adding lateral connections between the nodes in the map layer. The lateral connections between successive winners would be strengthened during training. Thus, given part of a sequence, one could follow the strongest lateral connections to complete the sequence. 584 Daniel L. James, Risto Miikkulainen 5 CONCLUSION SARDNET is a novel feature map architecture for classifying sequences of input vectors. Each sequence is mapped on a distributed representation on the map, making it possible to pack a remarkable large number of category representations on a small feature map . The representations are not only dense, they also represent the similarities of the sequences, which should turn out useful in cognitive science as well as real-world applications of the architecture. Acknowledgments Thanks to Jon Hilbert for converting CELEX data into the International Phonetic Alphabet format used in the experiments. This research was supported in part by the National Science Foundation under grant IRI-9309273. References Chappel , G. J., and Taylor, J. G. (1993). The temporal Kohonen map. Neural Kangas, J. (1991). Time-dependent self-organizing maps for speech recognition. In Proceedings of the International Conference on Artificial Neural Networks (Espoo, Finland), 1591-1594. Amsterdam; New York: North-Holland. Kohonen, T. (1989). Self-Organization and Associative Memory. Berlin; Heidelberg; New York: Springer. Third edition. Kohonen, T . (1990). The self-organizing map. Proceedings of the IEEE, 78:1464- Samarabandu, J. K., and Jakubowicz, O. G . (1990). Principles of sequential fea ture maps in multi-level problems. In Proceedings of the International Joint Conference on Neural Networks (Washington, DC), vol. II, 683-686. Hillsdale, NJ: Erlbaum. Scholtes, J. C. (1991). Recurrent Kohonen self-organization in natural language processing. In Proceedings of the International Conference on Artificial Neu ral Networks (Espoo, Finland), 1751-1754. Amsterdam; New York: North Holland. van Harmelen, H. (1993). Time dependent self-organizing feature map for speech recognition. Master's thesis, University of Twente, Enschede, the Netherlands. Zandhuis, J. A . (1992). Storing sequential data in self-organizing feature maps. Internal Report MPI-NL- TG-492, Max-Planck-Institute fur Psycholinguistik, Nijmegen, the Netherlands.",
"INTRODUCTION Measurement of facial expressions is important for research and assessment psychi atry, neurology, and experimental psychology (Ekman, Huang, Sejnowski, Hager, 1992), and has technological applications in consumer-friendly user interfaces, inter active video and entertainment rating. The Facial Action Coding System (FACS) is a method for measuring facial expressions in terms of activity in the underlying facial muscles (Ekman Friesen, 1978). We are exploring ways to automate FACS. 824 BARTLETI, VIOLA, SEJNOWSKI, GOLOMB, LARSEN, HAGER, EKMAN Rather than classifying images into emotion categories such as happy, sad, or sur prised, the goal of this work is instead to detect the muscular actions that comprise a facial expression. FACS was developed in order to allow researchers to measure the activity of facial muscles from video images of faces. Ekman and Friesen defined 46 distinct action units, each of which correspond to activity in a distinct muscle or muscle group, and produce characteristic facial distortions which can be identified in the images. Although there are static cues to the facial actions, dynamic information is a critical aspect of facial action coding. FACS is currently used as a research tool in several branches of behavioral science, but a major limitation to this system is the time required to both train human experts and to manually score the video tape. Automating the Facial Action Coding System would make it more widely accessible as a research tool, and it would provide a good foundation for human-computer interactions tools. Why Detect Facial Actions? Most approaches to facial expression recognition by computer have focused on clas sifying images into a small set of emotion categories such as happy, sad, or surprised (Mase, 1991; Yacoob Davis, 1994; Essa Pentland, 1995). Real facial signals, however, consist ofthousands of distinct expressions, that differ often in only subtle ways . These differences can signify not only which emotion is occurring, but whether two or more emotions have blended together, the intensity of the emotion(s), and if an attempt is being made to control the expression of emotion (Hager Ekman , An alternative to training a system explicitly on a large number of expression cat egories is to detect the facial actions that comprise the expressions. Thousands of facial expressions can be defined in terms of this smaller set of structural compo nents. We can verify the signal value of these expressions by reference to a large body of behavioral data relating facial actions to emotional states which have al ready been scored with FACS. FACS also provides a meanS for obtaining reliable training data. Other approaches to automating facial measurement have mistakenly relied upon voluntary expressions, which tend to contain exaggerated and redundant cues, while omitting some muscular actions altogether (Hager Ekman, 1995). 2 IMAGE DATABASE We have collected a database of image sequences of subjects performing specified facial actions. The full database contains over 1100 sequences containing over 150 distinct actions, or action combinations, and 24 different subjects. The sequences contain 6 images, beginning with a neutral expression and ending with a high in tensity muscle contraction (Figure 1). For our initial investigation we used data from 20 subjects and attempted to classify the six individual upper face actions illustrated in Figure 2. The information that is available in the images for detecting and discriminating these actions include distortions in the shapes and relative po sitions of the eyes and eyebrows, the appearance of wrinkles, bulges, and furrows, in specific regions of the face, and motion of the brows and eyelids. Prior to classifying the images, we manually located the eyes, and we used this information to crop a region around the upper face and scale the images to 360 x 240. The images were rotated so that the eyes were horizontal, and the luminance was normalized. Accurate image registration is critical for principal components based approaches. For the holistic analysis and flow fields, the images were further scaled Classifying Facial Action 825 to 22 x 32 and 66 x 96, respectively. Since the muscle contractions are frequently asymmetric about the face, we doubled the size of our data set by reflecting each image about the vertical axis, giving a total of 800 images. Figure 1: Example action sequences from the database. Figure 2: Examples of the six actions used in this study. AU 1: Inner brow raiser. 2: Outer brow raiser. 4: Brow lower. 5: Upper lid raiser (widening the eyes). 6: Cheek raiser. 7: Lid tightener (partial squint). 3 HOLISTIC SPATIAL ANALYSIS The Eigenface (Thrk Pentland, 1991) and Holon (Cottrell Metcalfe, 1991) representations are holistic representations based on principal components, which can be extracted by feed forward networks trained by back propagation. Previous work in our lab and others has demonstrated that feed forward networks taking such holistic representations as input can successfully classify gender from facial images (Cottrell Metcalfe, 1991; Golomb, Lawrence, Sejnowski, 1991). We evaluated the ability of a back propagation network to classify facial actions given principal components of graylevel images as input. The primary difference between the present approach and the work referenced above is that we take the principal components of a set of difference images, which we obtained by subtracting the first image in the sequence from the subsequent images (see Figure 3). The variability in our data set is therefore due to the facial distortions and individual differences in facial distortion, and we have removed variability due to surface-level differences in appearance. We projected the difference images onto the first N principal components of the dataset, and these projections comprised the input to a 3 layer neural network with 10 hidden units, and six output units, one per action (Figure 3.) The network is feed forward and fully connected with a hyperbolic tangent transfer function, and was trained with conjugate gradient descent. The output of the network was determined using winner take all, and generalization to novel subjects was determined by using the leave-one-out, or jackknife, procedure in which we trained the network on 19 subjects and reserved all of the images from one subject for testing. This process was repeated for each of the subjects to obtain a mean generalization performance across 20 test cases. 826 BARTLETI, VIOLA, SEJNOWSKI, GOLOMB, LARSEN, HAGER, EKMAN We obtained the best performance with 50 component projections, which gave 88.6 correct across subjects. The benefit obtained by using principal components over the 704-dimensional difference images themselves is not large. Feeding the difference images directly into the network gave a performance of 84 correct. 6 OUtputs I WT A Figure 3: Left: Example difference image. Input values of -1 are mapped to black and 1 to white. Right: Architecture of the feed forward network. 4 FEATURE MEASUREMENT We turned next to explicit measurement of local image features associated with these actions. The presence of wrinkles in specific regions of the face is a salient cue to the contraction of specific facial muscles. We measured wrinkling at the four facial positions marked in Figure 4a, which are located in the image automatically from the eye position information. Figure 4b shows pixel intensities along the line segment labeled A, and two major wrinkles are evident. We defined a wrinkle measure P as the sum of the squared derivative of the intensity values along the segment (Figure 4c.) Figure 4d shows P values along line segment A, for a subject performing each of the six actions. Only AU 1 produces wrinkles in the center of the forehead. The P values remain at zero except for AU 1, for which it increases with increases in action intensity. We also defined an eye opening measure as the area of the visible sclera lateral to the iris. Since we were interested in changes in these measures from baseline, we subtract the measures obtained from the neutral image. Pixel Image in Seqence Figure 4: a) Wrinkling was measured at four image locations, A-D. b) Smoothed pixel intensities along the line labeled A. c) Wrinkle measure. d) P measured at image location A for one subject performing each of the six actions. We classified the actions from these five feature measures using a 3-layer neural net with 15 hidden units. This method performs well for some subjects but not for Classifying Facial Action 827 Figure 5: Example flow field for a subject performing AU 7, partial closure of the eyelids. Each flow vector is plotted as an arrow that points in the direction of motion. Axes give image location. others, depending on age and physiognomy. It achieves an overall generalization performance of 57 correct. 5 OPTIC FLOW The motion that results from facial action provides another important source of information. The third classifier attempts to classify facial actions based only on the pattern of facial motion. Motion is extracted from image pairs consisting of a neutral image and an image that displays the action to be classified. An approximation to flow is extracted by implementing the brightness constraint equation (2) where the velocity (vx,Vy) at each image point is estimated from the spatial and temporal gradients of the image I. The velocities can only be reliably extracted at points of large gradient, and we therefore retain only the velocities from those locations. One of the advantages of this simple local estimate of flow is speed. It takes 0.13 seconds on a 120 MHz Pentium to compute one flow field. A resulting flow image is illustrated in Figure 5. We obtained weighted templates for each of the actions by taking mean flow fields from 10 subjects. We compared novel flow patterns, r to the template ft by the similarity measure S (3). S is the normalized dot product of the novel flow field with the template flow field. This template matching procedure gave 84.8 accuracy for novel subjects. Performance was the same for the ten subjects used in the training 6 COMBINED SYSTEM Figure 6 compares performance for the three individual methods described in the previous sections. Error bars give the standard deviation for the estimate of gener alization to novel subjects. We obtained the best performance when we combined all three sources of information into a single neural network. The classifier is a 828 BAR1LETI, VIOLA, SEJNOWSKI, GOLOMB, LARSEN, HAGER, EKMAN I 6 Output I WTA Classifier Figure 6: Left: Combined system architecture. Right: Performance comparisons. Holistic v. Flow Feature v. Row Feature v. Holistic Figure 7: Performance correlations among the three individual classifiers. Each data point is performance for one of the 20 subjects. feed forward network taking 50 component projections, 5 feature measures, and 6 template matches as input (see Figure 6.) The combined system gives a generalization performance of 92, which is an im provement over the best individual method at 88.6. The increase in performance level is statistically significant by a paired t-test. While the improvement is small, it constitutes about 30 of the difference between the best individual classifier and perfect performance. Figure 6 also shows performance of human subjects on this same dataset. Human non-experts can correctly classify these images with about 74 accuracy. This is a difficult classification problem that requires considerable training for people to be able to perform well. We can examine how the combined system benefits from multiple input sources by looking at the cprrelations in performance of the three individual classifiers. Combining estimators is most beneficial when the individual estimators make very different patterns of errors.1 The performance of the individual classifiers are com pared in Figure 7. The holistic and the flow field classifiers are correlated with a coefficient of 0.52. The feature based system, however, has a more independent pattern of errors from the two template-based methods. Although the stand-alone performance of the feature based system is low, it contributes to the combined system because it provides estimates that are independent from the two template-based systems. Without the feature measures, we lose 40 of the improvement. Since we have only a small number of features, this data does not address questions about whether templates are better than features, but it does suggest that local features plus templates may be superior to either one alone, since they may have independent patterns of errors. iTom Dietterich, Connectionists mailing list, July 24, 1993. Classifying Facial Action 829 7 DISCUSSION We have evaluated the performance of three approaches to image analysis on a dif ficult classification problem. We obtained the best performance when information from holistic spatial analysis, feature measurements, and optic flow fields were com bined in a single system. The combined system classifies a face in less than a second on a 120 MHz Pentium. Our initial results are promising since the upper facial actions included in this study represent subtle distinctions in facial appearance that require lengthy training for humans to make reliably. Our results compare favorably with facial expression recognition systems developed by Mase (1991), Yacoob and Davis (1994), and Pad gett and Cottrell (1995), who obtained 80, 88, and 88 accuracy respectively for classifying up to six full face expressions. The work presented here differs from these systems in that we attempt to detect individual muscular actions rather than emo tion categories, we use a dataset of labeled facial actions, and our dataset includes low and medium intensity muscular actions as well as high intensity ones. Essa and Pentland (1995) attempt to relate facial expressions to the underlying musculature through a complex physical model of the face. Since our methods are image-based, they are more adaptable to variations in facial structure and skin elasticity in the subject population. We intend to apply these techniques to the lower facial actions and to action com binations as well. A completely automated method for scoring facial actions from images would have both commercial and research applications and would reduce the time and expense currently required for manual scoring by trained observers. Acknow ledgments This research was supported by Lawrence Livermore National Laboratories, Intra University Agreement B291436, NSF Grant No. BS-9120868, and Howard Hughes Medical Institute. We thank Claudia Hilburn for image collection. References Cottrell, G., Metcalfe, J. (1991): Face, gender and emotion recognition using holons. In Advances in Neural Information Processing Systems 9, D. Touretzky, (Ed.) San Mateo: Ekman, P., Friesen, W. (1978): Facial Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto, CA: Consulting Psychologists Press. Ekman, P., Huang, T., Sejnowski, T., Hager, J. (1992): Final Report to NSF of the Planning Workshop on Facial Expression Understanding. Available from HIL-0984, UCSF, San Francisco, CA 94143. Essa, I., Pentland, A. (1995). Facial expression recognition using visually extracted facial action parameters. Proceedings of the International Workshop on Automatic Face- and Gesture-Recognition. University of Zurich, Multimedia Laboratory. Golomb, B., Lawrence, D., Sejnowski, T. (1991). SEXnet: A neural network identifies sex from human faces. In Advances in Neural Information Processing Systems 9, D. Touretzky, (Ed.) San Mateo: Morgan Kaufman: 572 - 577. Hager, J., Ekman, P., (1995). The essential behavioral science of the face and gesture that computer scientists need to know. Proceedings of the International Workshop on Automatic Face-and Gesture-Recognition. University of Zurich, Multimedia Laboratory. Mase, K. (1991): Recognition of facial expression from optical flow. IEICE Transactions Padgett, C., Cottrell, G., (1995). Emotion in static face images. Proceedings of the Institute for Neural Computation Annual Research Symposium, Vol 5. La Jolla, CA. Turk, M., Pentland, A. (1991): Eigenfaces for Recognition. Journal of Cognitive Neu Yacoob, Y., Davis, L. (1994): Recognizin human facial expression. University of Maryland Center for Automation Research Technical Report No. 706.",
"Introduction Certain classification problems, such as recognizing the digits of a hand written zip code, require the assignment of each object to a class. Others, involving relatively small amounts of data and high risk, call for indecision until more data become available. Examples in such areas as medical diagnosis, stock trading and radar detection are well known. The training data for the classifier in both cases will correspond to firmly labeled members of the competing classes. (A patient may be Presently a Senior Research Associate of the National Research Council at M . S. 210-9, NASA Ames Research Center, Moffett Field, CA 94035, on sabbatical leave from the Technion. Consistent Classification, Firm and Soft 327 either ill or healthy. A stock price may increase, decrease or stay the same). Yet, the classification of new objects need not be firm. (A given patient may be kept in hospital for further observation. A given stock need not be bought or sold every day). We call classification of the first kind \"firm\" and classification of the second kind \"soft\". The latter is not the same as training the classifier with a \"don't care\" option, which would be just another firm labeling option, as \"yes\" and \"no\", and would require firm classification. A classifier that correctly classifies the training data is called \"consistent\". Consistent classifier reductions have been considered in the contexts of the nearest neighbor criterion (Hart, 1968) and decision trees (Holte, In this paper we present a geometric approach to consistent firm and soft classifi cation. The classifiers are based on unions of local separators, which cover all the labeled points of a given class, and separate them from the others. We propose a consistent reduction of the nearest neighbor classifier and derive its expected design complexity and the expected classifier size. The nearest neighbor classifier and its consistent derivatives perform \"firm\" classification. Soft classification is performed by unions of maximal -volume spherical local separators. A domain of indecision is created near the boundary between the two sets of class-labeled points, and in regions where there is no data. We propose an economically motivated benefit func tion for a classifier as the difference between the probabilities of success and failure. Employing the respective benefit functions, the advantage of soft classification over firm classification is shown to depend on the rate of indecision. The performances of the proposed algorithms in predicting stock behavior are compared to those of the nearest neighbor method. 2 Consistent Firm Classification Consider a finite set of points X {X(i), i 1, ... , N} in some subset of Rn, the real space of dimension n . Suppose that each point of X is assigned to one of two classes, and let the corresponding subsets of X, having N1 and N2 points, respectively, be denoted Xl and X 2 We shall say that the two sets are labeled L1 and L 2 , respectively. It is desired to divide Rn into labeled regions, so that new, . unlabeled points can be assigned to one of the two classes. We define a local separator of a point x of Xl with respect to X 2 as a convex set, s(xI2), which contains x and no point of X2. A separator family is defined as a rule that produces local separators for class-labeled points. We call the set of those points of Rn that are closer to a point x E Xl than to any point of X2 the minimum-distance local separator of x with respect to X2. We define the local clustering degree, c, of the data as the expected fraction of data points that are covered by a local minimum -distance separator. The nearest neighbor criterion extends the class assignment of a point x E Xl to its minimum-distance local separator. It is clearly a consistent and firm classifier whose memory size is O(N). Hart's Condensed Nearest Neighbor (CNN) classifier (Hart, 1968) is a consis tent subset of the data points that correctly classifies the entire data by the nearest neighbor method. It is not difficult to show that the complexity of the algorithm 328 Y. Baram proposed by Hart for finding such a subset is O(N3). The expected memory re quirement (or classifier size) has remained an open question. We propose the following Reduced Nearest Neighbor (RNN) classifier: include a labeled point in the consistent subset only if it is not covered by the minimum distance local separator of any of the points of the same class already in the subset. It can be shown (Baram, 1996) that the complexity of the RNN algorithm is O(N2). and that the expected classifier size is O(IOgl(I-C) N). It can also be shown that the latter bounds the expected size of the CNN classifier as well. It has been suggested that the utility of the Occam's razor in classification would \"Given a choice between two plausible classifiers that perform identically on the data set, the simpler classifier is expected to classify correctly more objects outside the training set\". The above statement is disproved by the CNN and the RNN classifiers, which are strict consistent reductions of the nearest neighbor classifier, likely to produce more errors. 3 Soft Classification: Indecision Pays, Sometimes When a new, unlabeled, point is closely surrounded by many points of the same class, its assignment to the same class can be said to be unambiguously supported by the data. When a new point is surrounded by points of different classes, or when it is relatively far from any of the labeled points, its assignment to either class can be said to be unsupported or ambiguously supported by the data. In the latter cases, it may be more desirable to have a certain indecision domain, where new points will not be assigned to a class. This will translate into the creation of indecision domains near the boundary between the two sets of labeled points and where there is no data. We define a separntor S(112) of Xl with respect to X2 as a set that includes Xl and excludes X2. Given a separator family, the union of local separators S(x(i) 12) of the points is a separator of Xl with respect to X2. It consists of NI local separators. Let XI,c be a subset of Xl. The set will be called a consistent separator of Xl with respect to X2 if it contains all the points of X 1. The set XI,c will then be called a consistent subset with respect to the given separator family. Let us extend the class assignment of each of the labeled points to a local separator of a given family and maximize the volume of each of the local separators without Consistent Classification, Finn and Soft 329 including in it any point of the competing class. Let Sc(112) and Sc(211) be consis tent separators of the two sets, consisting of maximal-volume (or, simply, maximaQ local separators of labeled points of the corresponding classes. The intersection of Sc(112) and Sc(211) defines a conflict and will be called a domain of ambiguity of the first kind. A region uncovered by either separator will be called a domain of ambiguity of the second kind. The union of the domains of ambiguity will be des ignated the domain of indecision. The remainders of the two separators, excluding their intersection, define the conflict-free domains assigned to the two classes. The resulting \"soft\" classifier rules out hard conflicts, where labeled points of one class are included in the separator of the other. Yet, it allows for indecision in areas which are either claimed by both separators or claimed by neither. Let the true class be denoted y (with possible values, e.g., y1 or y2) and let the classification outcome be denoted y. Let the probabilities of decision and indecision by the soft classifier be denoted Pd and Pid, respectively (of course, P id 1 - Pd), and let the probabilities of correct and incorrect decisions by the firm and the soft classifiers be denoted Pfirm {y y}, Pfirm {y P y}, P soft {y y} and Psoft {y P y}, respectively. Finally, let the joint probabilities of a decision being made by the soft classifier and the correctness or incorrectness of the decision be denoted, respec tively, Psoft { d, Y y} and P soft { d, Y P y} and let the corresponding conditional probabilities be denoted Psoft {y y I d} and Psoft {y P y I d}, respectively. We define the benefit of using the firm classifier as the difference between the prob ability that a point is classified correctly by the classifier and the probability that it is misclassified: This definition is motivated by economic consideration: the profit produced by an investment will be, on average, proportional to the benefit function. This will become more evident in a later section, were we consider the problem of stock trading. For a soft classifier, we similarly define the benefit as the difference between the probability of a correct classification and that of an incorrect one (which, in an economic context, assumes that indecision has no cost, other than the possible loss of profit). Now, however, these probabilities are for the joint events that a classification is made, and that the outcome is correct or incorrect, respectively: Soft classification will be more beneficial than firm classification if Bsoft Bfirm' which may be written as For the latter to be a useful condition, it is necessary that Pfirm {y y} 0.5, Psofdy y I d} 0.5 and Psoft {y y I d} Pfirm {y y}. The latter will be normally satisfied, since points of the same class can be expected to be denser under the corresponding separator than in the indecision domain. In other words, 330 Y. Baram the error ratio produced by the soft classifier on the decided cases can be expected to be smaller than the error ratio produced by the firm classifier, which decides on all the cases. The satisfaction of condition (5) would depend on the geometry of the data. It will be satisfied for certain cases, and will not be satisfied for others. This will be numerically demonstrated for the stock trading problem. The maximal local spherical separator of x is defined by the open sphere centered at x, whose radius r(xI2) is the distance between x and the point of X2 nearest to x. Denoting by s(x, r) the sphere of radius r in Rn centered at x, the maximal local separator is then sM(xI2) s(x, r(xI2)). A separator construction algorithm employing maximal local spherical separators is described below. Its complexity is clearly O(N2). Let Xl Xl. For each of the points xci) of Xl, find the minimal distance to the points of X 2 Call it r(x(i) 12). Select the point x(i) for which r(x(i) 12) 2: r(x(j) 12), j f: i, for the consistent subset. Eliminate from Xl all the points that are covered by SM(X(i) 12). Denote the remaining set Xl. Repeat the procedure while Xl is non-empty. The union of the maximal local spherical separators is a separator for Xl with respect to X 2 . 4 Example: Firm and soft prediction of stock behaviour Given a sequence of k daily trading (\"close\") values of a stock, it is desired to predict whether the next day will show an increase or a decrease with respect to the last day in the sequence. Records for ten different stocks, each containing, on average, 1260 daily values were used. About 60 percent of the data were used for training and the rest for testing. The CNN algorithm reduced the data by 40 while the RNN algorithm reduced the data by 35. Results are show in Fig. 1. It can be seen that, on average, the nearest neighbor method has produced the best results. The performances of the CNN and the RNN classifiers (the latter producing only slightly better results) are somewhat lower. It has been argued that performance within a couple of percentage points by a reduced classifier supports the utility of Occam's razor (Holte, 1993). However, a couple of percentage points can be quite meaningful in stock trading. In order to evaluate the utility of soft classification in stock trading, let the predic tion success rate of a firm classifier, be denoted f and that of a soft classifier for the decided cases s. For a given trade, let the gain or loss per unit invested be denoted q, and the rate of indecision of the soft classifier ir. Suppose that, employing the firm classifier, a stock is traded once every day (say, at the \"close\" value), and that, employing the soft classifier, it is traded on a given day only if a trade is decided by the classifier (that is, the input does not fall in the indecision domain). The expected profit for M days per unit invested is 2(1 - 0.5)qM for the firm classifier and 2(s - 0.5)q(l-ir)M for the soft classifier (these values disregard possible com mission and slippage costs). The soft classifier will be preferred over the firm one if the latter quantity is greater than the former, that is, if which is the sample representation of condition (5) for the stock trading problem. Consistent Classification, Firm and Soft 331 ni . llIifip. llCce bene.fit Figure 1: Success rates in the prediction of rize and fall in stock values. Results for the soft classifier, applied to the stock data, are presented in Fig. 1. The indecision rates and the success rates in the decided cases are then specified along with a benefit sign. A positive benefit represents a satisfaction of condition (6), with ir, f and s replaced by the corresponding sample values given in the table. This indicates a higher profit in applying the soft classifier over the application of the nearest neighbor classifier. A negative benefit indicates that a higher profit is produced by the nearest neighbor classifier. It can be seen that for two of the stocks (xdssi and xelrnf) soft classification has produced better results than firm classification, and for the remaining eight stocks finn classification by the nearest neighbor method has produced better results. 5 Conclusion Solutions to the consistent classification problem have been specified in tenns of local separators of data points of one class with respect to the other. The expected complexities of the proposed algorithms have been specified, along with the ex pected sizes of the resulting classifiers. Reduced consistent versions of the nearest neighbor classifier have been specified and their expected complexities have been derived. A notion of \"soft\" classification has been introduced an algorithm for its implementation have been presented and analyzed. A criterion for the utility of such classification has been presented and its application in stock trading has been demonstrated. Acknowledgment The author thanks Dr. Amir Atiya of Cairo University for providing the stock data used in the examples and for valuable discussions of the corresponding results. 332 y. Baram References Baram Y. (1996) Consistent Classification, Firm and Soft, CIS Report No. 9627, Center for Intelligent Systems, Technion, Israel Institute of Technology, Haifa 32000, Israel. Baum, E. B . (1988) On the Capabilities of Multilayer Perceptrons, J. Complexity, Hart, P. E. (1968) The Condensed Nearest Neighbor Rule, IEEE Trans. on Infor Holte, R. C. (1993) Very Simple Classification Rules Perform Well on Most Com monly Used databases, Machine Learning, Vol. 11, No. 1 pp. 63 - 90. Rosenblatt, F. (1958) The Perceptron: A Probabilistic Model for Information Stor age and Organization in the Brain, Psychological Review, Vol. 65, pp. 386 - 408. Webb, G. 1. (1996) Further Experimental Evidence against the Utility of Occam's Razor, J. of Artificial Intelligence Research 4, pp. 397 - 147."
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4] 
