Browsing by Author "Soto A."

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Evaluation Benchmarks for Spanish Sentence Representations
    (European Language Resources Association (ELRA), 2022) Araujo Vasquez, Vladimir Giovanny; Carvallo, Andrés; Soto A.; Moens M.-F.; Kundu S.; Mercer R.E.; Canete J.; Bravo-Marquez F.; Mendoza M.
    © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.Due to the success of pre-trained language models, versions of languages other than English have been released in recent years. This fact implies the need for resources to evaluate these models. In the case of Spanish, there are few ways to systematically assess the models' quality. In this paper, we narrow the gap by building two evaluation benchmarks. Inspired by previous work (Conneau and Kiela, 2018; Chen et al., 2019), we introduce Spanish SentEval and Spanish DiscoEval, aiming to assess the capabilities of stand-alone and discourse-aware sentence representations, respectively. Our benchmarks include considerable pre-existing and newly constructed datasets that address different tasks from various domains. In addition, we evaluate and analyze the most recent pre-trained Spanish language models to exhibit their capabilities and limitations. As an example, we discover that for the case of discourse evaluation tasks, mBERT, a language model trained on multiple languages, usually provides a richer latent representation than models trained only with documents in Spanish. We hope our contribution will motivate a fairer, more comparable, and less cumbersome way to evaluate future Spanish language models.
  • No Thumbnail Available
    Item
    Inspecting the concept knowledge graph encoded by modern language models
    (Association for Computational Linguistics (ACL), 2021) Aspillaga C.; Soto A.; Mendoza Rocha, Marcelo Gabriel
    © 2021 Association for Computational LinguisticsThe field of natural language understanding has experienced exponential progress in the last few years, with impressive results in several tasks. This success has motivated researchers to study the underlying knowledge encoded by these models. Despite this, attempts to understand their semantic capabilities have not been successful, often leading to non-conclusive, or contradictory conclusions among different works. Via a probing classifier, we extract the underlying knowledge graph of nine of the most influential language models of the last years, including word embeddings, text generators, and context encoders. This probe is based on concept relatedness, grounded on WordNet. Our results reveal that all the models encode this knowledge, but suffer from several inaccuracies. Furthermore, we show that the different architectures and training strategies lead to different model biases. We conduct a systematic evaluation to discover specific factors that explain why some concepts are challenging. We hope our insights will motivate the development of models that capture concepts more precisely.