The Meta AI chatbot is more willing to share what data it was trained on than Meta is.
Meta, formerly known as Facebook, first released Meta AI last year amid its big push into the generative AI space to keep up with a wave of public interest prompted in late 2022 by the release of OpenAI's ChatGPT. It expanded Meta AI in April as a chat and image generator function across all its apps, including Instagram and WhatsApp. Still, it hasn't disclosed much about how its chatbot was trained.
When Business Insider asked Meta AI a series of questions about what data it was trained on and how Meta obtained such data, the chatbot gave some interesting answers.
Meta AI told Business Insider that it was trained on large datasets of transcriptions from YouTube videos. In addition, it said Meta has its own web scraper bot called "MSAE," an acronym for Meta Scraping and Extraction, which it said scrapes large amounts of data from the web to train AI models.
Meta had not disclosed the existence of this scraper before. YouTube's terms of service prohibit the use of bots and scrapers to collect its data, and the use of such data without its permission, something OpenAI has recently come under scrutiny for purportedly doing.
A Meta spokesman did not deny any of Meta AI's answers about its scraper or training data. BI provided Meta with the prompts it used. Instead, the spokesman suggested that Meta AI could be incorrect.
"As with all generative AI systems, the models could return inaccurate or inappropriate outputs," the spokesman said. "We'll continue to improve these features as they evolve, and more people share their feedback."
The spokesman also noted, "Like others in the industry, we use web crawlers," without specifying the MSAE bot Meta AI cited.
"Generative AI models take a large amount of data to effectively train, so a combination of sources are used for training, including information that's publicly available online and annotated data," the spokesperson added.
Meta AI initially said its training data included a third-party dataset of 3.7 million transcribed YouTube videos. The chatbot specified that it "did not use its web scraper bot to scrape YouTube videos directly."
In responding to further queries about its YouTube training data, Meta AI said its training data included another, larger dataset of transcriptions from 6 million YouTube videos also compiled by a third party. It added that its training data includes two additional sets of YouTube transcriptions or subtitles, one with 1.5 million videos and another with 2.5 million videos, as well as a dataset of transcriptions from 2,500 TED Talks posted to YouTube. These datasets were all compiled by third parties, according to Meta AI.
Meta's chatbot said it "takes steps to avoid collecting copyrighted data." In using Meta AI, it seems clear that the chatbot is scraping the web to some degree. Results for several queries cited sources like NBC News, CNN, and The Financial Times. Meta AI often did not include sources for its responses, unless specifically asked to do so.
Meta is currently considering new paid deals with media publishers to gain access to more AI training data, as BI reported, which could improve Meta AI's results.
Meta AI also said it respects robots.txt, a line of code website owners can use to ostensibly stop content from being scraped by bots that now leverage the content for AI training.
Meta developed the chatbot with its large language model, Llama. Although Llama 3 was released in April, around the time Meta AI was expanded, Meta has yet to publish an accompanying research paper for the new model or disclose the training data used. In a blog post, Meta said the massive set of 15 trillion tokens, or language units, that Llama 3 was trained on came from "publicly available sources."
Web scrapers, like OpenAI's GPTBot, Google's GoogleBot, and Common Crawl's CCBot, can effectively extract any and all content accessible on the web. The content is stored in massive datasets fed into LLMs and often regurgitated by generative AI tools like ChatGPT.
Several ongoing lawsuits concern owned and copyrighted content being freely absorbed by the world's biggest tech companies. The US Copyright Office is expected to release new guidance on acceptable use for AI companies later this year.
Are you a Meta employee or someone with a tip or insight to share? Contact Kali Hays at khays@bjinnox.com or on the secure messaging app Signal at 949-280-0267. Reach out using a non-work device.
