NotebookLM is my favorite AI tool by a long shot, but this doesn't mean it doesn't have flaws. While I think that NotebookLM is at risk of losing what made it special, I still enjoy experimenting with its new features and using it to summarize large amounts of information.

Its reliance on sources provided by the user is what sets it apart from other tools. But as I've played around with its features, my experience as a journalist and someone writing about tech has made me realize how this reliance can also be a weakness.

NotebookLM's source requirement is a major strength

It helps prevent hallucinations

If you've tried out various AI tools to summarize information, you'll be familiar with the problem of hallucinations. This comes down to the way LLMs are trained and the way they work. The text they generate is based on patterns rather than knowledge about what's correct or true.

While NotebookLM is powered by Gemini, it differs from LLMs because it is grounded in the sources added to a notebook. This means it pulls from a much smaller dataset than the average chatbot. If your notes don't include the information you request in a prompt, it simply states this.

However, chatbots with access to the internet may pull from dubious sources or simply try to 'guess' the answer. They are also sometimes vulnerable to sites and scammers that create content specifically to manipulate AI answers.

NotebookLM isn't immune to hallucinating. I've come across the issue when generating Audio Overviews even when using pretty simple data. But I have found that it hallucinates far less than other AI tools.

This dependence on sources can be a weakness too

The quality of sources matters

Since NotebookLM relies heavily on your sources, the quality of those you add to your notebook impacts its output significantly.

If your sources lack important context, include false information, or lean heavily towards a certain viewpoint, this will significantly shape the chatbot's output. It can't discern truth from fiction, so it will always take your sources as fact.

This is an issue I first really mulled over when looking at some of NotebookLM's featured notebooks. Some of the notebooks are from specific authors, publications, or businesses. While it can be useful to get insights from experts, I would also never recommend that someone learn about a topic from only a single outlet. Even the most neutral publication will still have blind spots.

To experiment with this idea, I played around with sources in different notebooks. For example, I have a notebook on fibromyalgia that I use to get recommended exercises. In this notebook, I included only high-quality sources from reputable outlets.

I then created a second notebook on fibromyalgia that included articles from a site that is known for medical misinformation. I then asked the chatbot in each notebook whether there is a cure for fibromyalgia. In the notebook with medically reviewed sources, the answer was no. In the dubious notebook, the answer was that certain treatments could heal or cure symptoms, maybe even completely.

I also have a notebook on the satirical Birds Aren't Real movement. This was set up to see whether the right (or wrong) sources could turn the chatbot into a conspiracy theorist. One source was the difference between presenting the movement as an earnest one or a satirical one.

Even mixing in a few bad-quality sources with decent ones significantly affects the output, since the chatbot doesn't weigh them differently as a search engine algorithm does. This is a big reason why I don't like using the feature to add sources from the web, since I find that NotebookLM doesn't always present high-quality sources from its search.

You can add context, but there's a limit

Free users can add fewer sources

The best remedy to ensure you have a well-rounded overview of a topic is to include a range of high-quality sources that approach the topic from different angles. But even then, there's a limit to how much you can add.

For free users, you can only add 50 sources to a notebook. This limit increases with different tiers of paid subscriptions. While casual users are unlikely to hit their limit easily, deep diving into a topic can easily result in dozens of sources. This will especially be a problem if you want to add news articles and the latest research on a specific subject.

Some users have also reported running into issues with performance when nearing the 50-source limit, with the chatbot taking longer to respond to queries. Some users have also reported that answers become more generic with increased sources. So in this way, adding additional context can become a double-edged sword.

However, I would still recommend finding a balance of well-rounded sources. For example, I found the responses about fibromyalgia were more well-rounded when I added some sources around the lived experience of patients rather than only the medical perspective.

You need to be an active contributor to your NotebookLM sources

While it's tempting to get NotebookLM to pull a few links from the web and call it a day, staying active in maintaining the quality of your sources is essential when using the AI tool.

This is less of an issue if you're just using NotebookLM while you're bored. But if you want to use it to gain a real understanding of a topic, you're going to have to be aware of the potential blind spots and biases of the sources you choose.