ELECTRICIDAD: The Spanish Electra Imgur

ELECTRICIDAD is a small Electra like model (discriminator in this case) trained on a Large Spanish Corpus (aka BETO's corpus).

As mentioned in the original paper: ELECTRA is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN. At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the SQuAD 2.0 dataset.

For a detailed description and experimental results, please refer the paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.

Model details ⚙

Param # Value
Layers \t12
Hidden 256 \t
Params 14M

Evaluation metrics (for discriminator) 🧾

Metric # Score
Accuracy 0.94
Precision 0.76
AUC 0.92

Benchmarks 🔨

WIP 🚧

How to use the discriminator in transformers

from transformers import ElectraForPreTraining, ElectraTokenizerFast
import torch

discriminator = ElectraForPreTraining.from_pretrained("mrm8488/electricidad-small-discriminator")
tokenizer = ElectraTokenizerFast.from_pretrained("mrm8488/electricidad-small-discriminator")

sentence = "el zorro rojo es muy rápido"
fake_sentence = "el zorro rojo es muy ser"

fake_tokens = tokenizer.tokenize(sentence)
fake_inputs = tokenizer.encode(sentence, return_tensors="pt")
discriminator_outputs = discriminator(fake_inputs)
predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)

[print("%7s" % token, end="") for token in fake_tokens]

[print("%7s" % int(prediction), end="") for prediction in predictions.tolist()[1:-1]]

# Output:
'''
el  zorro   rojo     es    muy    ser      0      0      0      0      0      1[None, None, None, None, None, None]
'''

As you can see there is a 1 in the place where the model detected the fake token (ser). So, it works! 🎉

Electricidad-small fine-tuned models

Acknowledgments

I thank 🤗/transformers team for answering my doubts and Google for helping me with the TensorFlow Research Cloud program.

Citation

If you want to cite this model you can use this:

@misc{mromero2020electricidad-small-discriminator,
  title={Spanish Electra (small) by Manuel Romero},
  author={Romero, Manuel},
  publisher={HF中国镜像站},
  journal={HF中国镜像站 Hub},
  howpublished={\url{https://huggingface.co/mrm8488/electricidad-small-discriminator}},
  year={2020}
}

Created by Manuel Romero/@mrm8488

Made with in Spain

Downloads last month
72
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no pipeline_tag.

Dataset used to train mrm8488/electricidad-small-discriminator

Collection including mrm8488/electricidad-small-discriminator