Questions about AI and ML - Atlas

[ademidun]

• what are the different ways a model can be open-sourced? Is there a difference between open-source weights vs open-source data?
• How come a model like gpt-j-6B is trained on 800 GB dataset but the model is less than that size
• How does one fine-tune a model?
  • Can we use unsuperverised learning so we don't need labelled data? https://www.sbert.net/examples/unsupervised_learning/README.html
  • What's the difference between domain adaptation and unsupervised learning? https://www.sbert.net/examples/domain_adaptation/README.html#domain-adaptation-vs-unsupervised-learning
  • How much better would the model be if we had labelled data?
  • What are some good resources for finetuning? Thoughts on this? https://towardsdatascience.com/fine-tune-transformer-models-for-question-answering-on-custom-data-513eaac37a80
• Should we use Bart LFQA or GB-j-6b for our question and answering? What should the prompt look like?
• What's the fastest way to A/B test 2 different models
• It looks like the model doesn't really understand the context of the generated answers
• Should we be using a Pre-Trained MS Marco model instead of a sentence embedding model because it's better for asymmetric search? https://www.sbert.net/examples/applications/semantic-search/README.html#symmetric-vs-asymmetric-semantic-search
• Should we apply for TRC and get some TPU? https://sites.research.google/trc/about/
  • Is TPU better than GPU? Is it compatible with existing GPU/CPU based code?