The Aboard Newsletter

I Keep Accidentally Building Search Engines

Ironically, the AI-ification of everything makes this possible—and necessary.

by -
Image of a search bar with a man peeking out of it.

I’m feeling lucky!

I think we all know that AI-generated text is hit or miss. It is good at summarizing things, or writing documentation about code, or generating a report after a meeting. On the other hand, I don’t let AI write articles for me, or do anything to my manuscripts. I’m not against you doing that, at all—it’s just that writing is how I think, and I don’t know what I think until I write it down. 

But that doesn’t mean that I don’t use AI prose. In fact, I find it incredibly useful, and consume thousands of words of AI-generated prose per day.

For example, I get a daily newsletter that summarizes and sorts industry news (for a couple different industries) to my exact specifications. It chews through about 500 articles a day and sends me bullet points about 20–50 of them. I then read the links on the web, pick out good paragraphs, and share those excerpts on my LinkedIn, typically about five a day.

I added a killer feature to the newsletter: I can reply to it. The other day, I saw an article about DeepSeek and I wrote back, “Please summarize the infrastructure being built to support Chinese AI companies.” It searched through all the articles it could find and sent me summaries and notes in response.

I’ve built other tools like this. One helps me find podcast guests. Another helps me find conferences where we might want to promote our company. In each case, it’s the same: I use AI to find and filter web pages, using a limited number of sources—Wikipedia and Wikipedia’s cited links come first, then Archive.org, then a few other paid and public sites that I trust, then well-known news organizations. I save it all in a database. I query it literally all day, often using AI to summarize the results.

One day, as I looked at my thousandth AI-generated bulleted list, I realized: These are just search results. I’ve gone and made a copy of huge, multi-gigabyte chunks of the web, locally, and had it searched all day long, because I don’t really count on Google to come up with trustworthy results. It’s not just those sloppy AI summaries at the top. It’s the preponderance of ads, and the slow decline of the open web, and, well, enshittification. So many things that used to be useful are no longer usable.

I think this is very weird! Huge web companies have every URL in their index. Giant AI companies absorb the web and use it to make LLMs. Why am I making my own little search engines and databases?

Partly, it’s because I need to be right about things when I write. Often I get fact-checked, and I need my facts to be trued up and cited and accurate. AI is just plain bad at that, and since search is so heavily AI-based now, I can’t really count on it. Part of it is I just want to see the actual links and click on the story because people did work and deserve some attention.

It’s wild to me that every single giant web and/or AI company could offer experiences like the tools I built and drive links back to publishers, who could then use that traffic to drive subscriptions and advertising revenue. But they do not want to, and at this point, I do not think we are going back.

That said, it really is a consolation that I can make my own search engines to solve problems now—I can spider the web, filter it, classify it, and search it, with full text, then click the link. My guess is lots of other people are doing stuff like this, too (always curious to hear if you are). 

Maybe the future is a bunch of big companies advertising to each other while the rest of us build our little cognitive islands and form a weird non-platform data archipelago. The last thing I ever expected to build was a personal search engine, but the AI-ification of everything simultaneously makes it possible, because it’s easier to build things than ever before, and makes it necessary, because the open web feels less and less reliable every day. 

Since AI keeps getting cheaper, it’s getting easier for me to share my search engine with other people—maybe eventually with the world. And as I go, so will many others. Eventually we could all federate our personal knowledge bases. Stranger things have happened! But ultimately good information does want to find its audience.