I’m beginning to see how well feedland fits into the a8c culture via Chuck Grimmett, as he ranges over Automattic, finding great collections of blogs that can be turned into reading lists and news products.

And of course that makes total sense. I feel like Rip Van Winkle in a way, you all picked up the ball from UserLand and ran with it for almost 20 years (so far). The seeds that were there when we were moving this stuff forward have developed into vast collections of people working on and with blogs and the web. And it’s very gratifying that the stuff I’ve built in FeedLand works to organize this vast collection of blogs as well as I hoped it would — and the reason is (forgive the all boldface) we respect open standards wherever they come from.

That’s the problem Tech had with RSS and related technology (XML-RPC, OPML).

Most people from Tech, from academia to Silicon Valley and much of the open source world, will reject anything that doesn’t have the aura of a standards body over it. I understand the cost of this very well, having worked with three Microsoft people for a few weeks to develop XML-RPC and then watching SOAP get continually more frustratingly complicated in pointless ways inside the W3C. I never argued with anyone, I just watched, I could afford to do that because we had XML-RPC there to build on. When the time was right, along with Jake Savin who worked at UserLand, we came up with a Busy Developer’s Guide to SOAP which basically brought it back to the simplicity of XML-RPC but was still 100% standard SOAP.

But the cool thing about Matt and therefore the culture that sprang from Matt, is that he never had any issue with RSS and XML-RPC. You’d have to ask him why he, along with a small number of others, didn’t seem to care. Thinking of Stewart Butterfield at Slack. And there are users who intuitively get this, like Om Malik and Doc Searls.

It’s the “wherever they come from” part that matters most. What’s so ironic is when the W3C was rolling out XML their pitch was “now you can create your own formats.” So that’s exactly what we did. And they burned us for doing that!

I offered to give them RSS many times, but they always looked the other way. So I found Harvard was willing to work with me, so I went down the street on Mass Ave, and now RSS and podcasting, blogging, BloggerCon, all sprung from there, and if you look at the history you’ll see that Automattic’s history came from that thread too. 😉

I’m glad I’m around to be part of this. Lovely.

The Bookmarks menu in FeedLand is there so it’s easy to get back to places you want to check regularly. Any URL in FeedLand can be bookmarked.

This is a screen shot of what’s in my Bookmarks menu right now.

BTW, it can be hierarchic, and you edit it in a tiny little outliner window that pops up inside FeedLand.

Also, Bookmarks are documented.

We noticed that when viewing the river page for a user that follows many feeds, it can take quite a while to load, around 6 seconds for approximately 1000 feeds. When checking the console it was clear the river building process was the culprit:

displayTraditionalRiver: getRiver took 4.463 secs.

The query we’re looking for is made on this line on feedlanddatabase.js and it looks like so:

SELECT * FROM items WHERE flDeleted = false AND feedUrl IN (SELECT feedUrl FROM subscriptions WHERE listName='screenname') ORDER BY pubDate DESC LIMIT 175;

We initially thought the issue was the lack of LIMIT on the query or maybe the subquery to get the list of feeds but we decided to focus on the ordering (ORDER BY pubDate). After digging a little bit on the code and checking the table structure (here) for indexes we could use to make the WHERE clause efficient, we decided to go with a compose index for the columns we’re querying. We ended up with two composite index ideas:

Add index on (pubDate, feedUrl, flDeleted)

Times for that query with the new index were virtually identical, it turns out MySQL was prioritizing the flDeleted index. We choose the order of the columns based on their cardinality but since we’re querying for flDeleted=false, and it’s the only = operation, it has a higher priority.

Add index on (flDeleted, pubDate, feedUrl)

After what we learned above, we dropped that index and created a new one, which essentially brought down the time needed to around 1/7th of the original query time for the same river page:

displayTraditionalRiver: getRiver took 0.626 secs.

The performance boost was so big, Dave found it was a good idea to add it to feedland.org as well and made a note about it. It’s also even in the original instructions for the database setup.

Future improvements

We can potentially remove the other indexes if we’re sure they’re not being used by other queries. That’s because the composite index can only be used if queries uses the columns from left to right, so other queries must at least use flDeleted, or flDeleted + pubDate, or flDeleted + pubDate + feedUrl.

Things I learned in the proccess

• MySQL chooses which index to used based on a bunch of factors, one of those being the operation and another one the cardinality (check the EXPLAIN command).
• Composite indexes are used if the columns are queried from left to right (explained above).
• MySQL has a neat EXPLAIN ANALYZE tool to profile queries.

Huge thanks to Chris for helping me out debugging that query.

Here’s a nice url to look at.

https://lists.feedcorps.org/

It returns an OPML file that you can open in Drummer.

https://drummer.land/?url=http://lists.feedcorps.org/

Each of the headlines in the outline is an include node, which means when you expand it, it gets the contents of the outline at the address in its url attribute, and inserts it below (subordinate to) the headline.

Each of the headlines in each of the outlines is a link to a feed that’s part of that subscription list.

https://lists.feedcorps.org/nyt.opml

FeedLand will have a command that loads the top level outline, and gives you a set of choices for our collection of reading lists. It will use the list-of-reading-lists viewer I wrote yesterday. Lots of partially finished pieces here, but you should be able to see how they begin to fit together.

The interface for people who create reading lists is GitHub.

First notes

1. I have a little more work to do on it, it must read the OPML files and cache information from them, like when the file was created, grab a description, and we may want to define some other head elements. We at least need a created element for each so the OPML works properly.

2. I have a few more hours of work to do on this.

3. I wanted to create it outside of FeedLand at least at first to make it easier to experiment with.

4. This is the piece I didn’t have enough control to do in config.json files in PagePark.

5. I published the code, which needs a bit of a cleanup.

Later notes

1. We’re now caching copies of all the outlines on the lists.feedcorps.org server, this makes it possible to include info from the lists in the master list, like the number of URLs in the list, needed for the readinglists page in FeedLand.

2. I believe I’m done with this app now, it’s deployed, seems to be running fine.

3. Did a code cleanup.

Late last week I quickly set up lists.feedcorps.org, as a way to view the collection of reading lists we’re creating on the new support repo on GitHub.

I have an idea where I want to go with this, and today I did some light cleanups, to make it work properly.

Note this site is served by PagePark, so most of the fixes are expressed in terms of PagePark features.

1. You no longer have to add ?format=OPML to the URL of a reading list, to prevent the server from rendering the list as HTML. In this context, we’re serving up OPML files, meant to be read by software. This was accomplished by setting flProcessOpmlFiles false in the config.json file for the site.

2. It was using the markdown template for the FeedLand Docs site, so it had the wrong title, and had a menu for the wrong website. I took that out, and we’re now using the default markdown template, which is very plain.

3. The home page currently lists each of the reading lists. I want something really nice here, and not sure how I will implement it. For now I’m going to add a readme.md file to the lists directory on the repo, and with any luck that should show up on the home page of the site. (Update: It worked, but changes to the readme.md file are being cached somewhere. More work to do here.)

4. Ultimately, we’re going to need a new feature in PagePark that lets us get some pages from the repo, and others from local files in the domain’s folder.

Last night I released new versions of feedland and feedlanddatabase with the implementation of reading lists. This morning I wrote the instructions for upgrading, and changed the setup instructions so it sets up the new version.

All of this has been very sparsely tested, and only by me, and esp with the upgrade instructions, by no one at all. So this is a high priority. 😉

I’m seeing some obvious huge holes in the user interface. For example.

1. I want to see a list of all the feeds from one reading list, in a Feed List view, so I can look at the first five posts, and click to get to the Feed Info page. I just added the newspack list and I wanted to see all of them in one place, but there’s no way to do it.
2. Similarly I want to see all the news from this list, without having to do it with categories. The information is available in the database, I think it’s a pretty simple query, but it doesn’t exist yet.
3. The NewsPack list is amazing. Just want to pause for a moment, this is the first time I’ve seen it in action.
4. Let’s use lists.feedcorps.org to serve our reading lists? I’d like the lists to come with docs, so when we present them, they’re like a landing page for a podcast. The OPML is of course the central most important component, but we can and should provide a friendly way of choosing them. This could even be integrated with FeedLand so you can see which ones you’re already subscribed to.

I currently have four news products powered by the A8c instance of FeedLand, with plans for more in the works:

1. Peekskill News
2. WordPress Project Updates
3. Newspack Sites
4. WP Special Projects Team Sites

Here is how I set them up:

Gathering Sources and making the OPML files

I started with the WP Special Projects site list. We have our own CLI with a command to export a site list, which I used. Then, since they are all WordPress sites, I appended /feed/ to the end each one.

Since two of these products contain hundreds of URLs, so I made a utility script in PHP to take CSVs and turn them into OPML files: https://gist.github.com/cagrimmett/5260af01f5f9642ff7677d52e88fdff4

For the Peekskill list the main hurdle was identifying the feeds. The City of Peekskill miraculously overhauled their main website a couple days later and exposed a bunch of new feeds, which I promptly added. I discovered the new site while paying my water bill online.

On Newspack, I needed to scrape the list of sites from an open bookmarking service where they keep their main list. Once scraped, I appended /feed/ and put the list into my csv-to-opml script.

For the WordPress Project News I grabbed the feeds pretty quickly from WordPress.org and make.WordPress.org.

Suggested improvement: Auto feed discovery, so users only need to add a domain they want to follow, not the full feed URL. Would be needed for the bookmarklet Dave suggested.

Importing & Categories

I added categories for each news product, then imported the OPML files. I added the categories as tabs in my main news feed to verify the imports worked and content made it into the rivers.

Suggested improvement: If categories are in an OPML file getting imported but do not exist in the user’s account, add them.

Outline-based templates

Dave implementing outline-based templates was a breakthrough and made spinning up news products a little easier for me. Here are the docs.

I implemented the four of these using Instant Outlines in Drummer.

Here are my four outlines:

1. https://drummer.land/chuck@grimmett.co/MakeWordPress.opml
2. https://drummer.land/chuck@grimmett.co/T51Partners.opml
3. https://drummer.land/chuck@grimmett.co/Peekskill.opml
4. https://drummer.land/chuck@grimmett.co/newspack.opml

I used Drummer for two reasons:

1. It is the easiest way to make OPML files!
2. Instant Outlines made iteration fast and simple. I make an edit, it autosaves, and I refresh the news product to see the change.

Long-term, once I stop making changes to the templates, I’ll probably download the stable OPML files from Drummer and stick them on one of my webservers.

Styling

Once I got a basic template up and running, I decided to try my hand at styling each one. Dave added support for loading custom CSS, so all that I needed to do was inspect the page to find the right selectors, write some styles to make my vision a reality, and add that to the OPML. Easy peasy.

Suggested improvement: While this works for developers and power uses, we’ll eventually need to support an easier way for customization. I suggested the WordPress Customizer to Dave as one idea we can borrow.

Mapping Domains

For Peekskill News and WordPress Project News, I mapped domains using PagePark. More on that here:

https://cagrimmett.com/development/2023/10/08/setting-up-pagepark-on-a-digitalocean-droplet/

Dave helped me with filter.js, the key to serving a news product to requests from the domain. More in this issue: https://github.com/scripting/a8c-FeedLand-Support/issues/18

Suggested improvement: Should we support mapping domains in the application? See https://github.com/scripting/a8c-FeedLand-Support/issues/19

---

Areas for further exploration:

• I don’t have tabs on any of my news products.
  • I could split out the Peekskill one into Official City Updates and Local Journalism.
  • I could add more blogs to the WordPress one and split out the Make WordPress Team Updates and Release News and General News
• The design of these pretty much look the same. How can I go wild and reimagine these templates? This is a time for experimentation. That’s what these are for.
• Add domains for the two without domains.
• Make more news products!
  • A8c Blogging Club feeds
  • More topical interests.
    • Christie Wright’s Customer-Centric Operations feeds
    • Apple bloggers (this is a product people might be interested in, even though I’m no longer that interested in the space)
    • My favorite food blogs

Something fun to close off today.

https://lists.feedcorps.org/nyt.opml?format=opml

This works too..

https://lists.feedcorps.org/wordpress-make-teams.opml?format=opml

A curious thing happens if you drop the ?format=opml at the end. 🙂

More on this to come, real soon now.

Every time I reload the All category in FeedLand, I spend the four seconds it takes to load the page thinking about ways around this problem.

And I think for premium users (ie paying customers) we will cache their page, and rebuild it every five minutes, so they are pretty much guaranteed it’ll be there, really fast, every time they visit.

We can still clear the cache based on a feed they’re following updating, but that just means we have to rebuild it immediately after.

It’ll be nice to have a way to spend a few bucks a month to make the system reallllly fast.

Reading Lists are another kind of open source.

We’re not locking people in to our tools for making and running news products.

We want ours to be the best, but not the only way to make news products.

And by setting this example, we’re encouraging those who come after to do the same.

It’s how OPML became the export format for subscription lists.

We want to make sure people don’t lose that feeling. 😉