Skip to main content

April 2024 Update

· One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Monthly Statistics

During March and April the Metron Project added the following to its database:

  • Users: 75
  • Issues: 4,495
  • Creators: 255
  • Characters: 1,632

Thanks to everyone that contributed!

Digital Chapters

Recently a new series type, Digital Chapters, was added to Metron. This was to facilitate adding digital comics, like DC Comics Bombshells. This series type appends the word Digital to the series display name and adds Chapter to the issue display name. When adding Digital Chapters series, the print version should not be added as an associated seres, but added as reprints to the appropriate issues. For an example of this, please refer to this issue.

Anyway, that's all I've got for this month. Take Care!

February 2024 Update

· 2 min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Monthly statistics

During February the Metron Project had a total of 5,635 issues added to its database during February bring the total number of issues to 112,135. We also had 65 new users join.

ComicTagger

Recently comictagger released the metron_talker plugin, but unfortunately it's got a few design flaws that is generating 2-4 times the number of API calls compared to other clients, and it's also downloading the cover for each issue, instead of using the issue's cover_hash exposed by the Metron API. The developers are working on addressing these issues, but if you are planning to tag a large number of comics we would appreciate it if you waited for a new release or use Metron-Tagger in the interim.

Server News

With the server load increase due to the comictagger bug, I've gone ahead and increased the CPU and RAM for the server and with any luck users shouldn't see a decrease in server quality of service.

Metron-Tagger

AJ Slater recently released a new library (comicfn2dict) for parsing comic filenames, and since the code for handling that in darkseid was fairly old and crusty it seemed like a good time to do a little work on Metron-Tagger. Some of the changes are as follows:

  • comicfn2dict

    So, I switched over to using comicfn2dict for filename parsing, and did some testing on around 1,000 or so comics. Overall, it's performed as well as the old code and has the added benefit of being used by multiple projects.

  • Improved Matching

    Spent some time improving comic matching by implementing the following changes:

    • Only use the series name and issue number when querying Metron. By doing this we are likely to get a result from Metron. Also, we are striping out any special characters (like ,, -, or &) from the series name.
    • Use the cover hash from Metron to verify the issue is correct, otherwise we'll ask the user for help.
    • Use existing ComicInfo.xml for matching. If a comic has already been tagged by comictagger (or Metron-Tagger) and has information to help identify the issue (like the Comic Vine ID), we'll use that before using the filename.

Miscellaneous

Right now, I'm thinking about making some changes to the API (like adding versioning), and if I do I'll ask the community for any suggestions of changes they'd like to see.

January 2024 Update

· One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Issues Indexed

The Metron Project had a total of 7,816 issues added to its database during January bring the total number of issues to 106,500. I wish to give big thanks to everyone who has helped with the project. Thanks!

Universes Support

Universe support has been added to Metron, which will allow users to add Publisher's universes to Issues, Characters, and Teams. I've used information from a variety of sources to populate the initial universes, but I'm sure there is still a lot to be added.

To aid updating existing issues faster with this information I've created a Django Management Command to update a series' issues. To make a request to update a series with Universe information, please provide the necessary information in this discussion page.

Universe Detail

Software Updates

New versions of Mokkari and Metron-Tagger have been released, which primarily deal with using Pydantic and adding Universe support.

Outstanding Bug Fixed

Fixed an outstanding bug where regular users were not able to add a new creator when adding a credit record to an Issue. Thanks to THElegend5 for reporting this bug!

Creator Widget

December 2023 Update

· One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Issues Indexed

The Metron Project had a total of 10,722 issues added to its database during December bring the total number of issues to 98,684. I wish to give big thanks to everyone who has helped with the project. Thanks!

2023 Database Statistics

We added the following number of entries during 2023:

  • Users: 319
  • Publishers: 33
  • Series: 2,430
  • Comics: 44,914
  • Characters: 7,082
  • Creators: 3,118
  • Teams: 691
  • Arcs: 521

HTMX and Streaming HTML

I'm playing around with using HTMX and Django's ability to stream HTML to improve the user experience. Basically, we'll load the static parts (like the footer, navbar, etc.) of the page templates as soon as possible, and then load the data as it becomes available and render its fragment of HTML. Not sure if this will be pushed to production or not until I do more testing, but it's something worth looking into.

November 2023 Update

· 2 min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Issues Indexed

Not much user-facing development was done in November, but 10,612 issues were added to the database bringing the total number of issues to 88,244. I wish to give big thanks to everyone who has helped with the project. Thanks!

2024 Tasks

I've started working on a list of items that I'm planning to work on in 2024. Here's a partial list:

  • Upgrade Server

    Currently, the server is fairly underpowered (1 GiB Memory, 1 CPU, 25 GiB SSD, 1,000 GiB Transfer) but is able to handle the current load without any particular issues. Based on the current user growth rate, I'm guessing I'll finally need to upgrade sometime in 2024, I'm thinking a fairly minor bump should be sufficient (2 GiB Memory, 1CPU, 50 GiB SSD, 2,000 GiB Transfer)

  • Container Deployment

    Before upgrading the server it makes sense to finally set up the project to use containers (preferably Podman) when deploying the website. If anyone wants to help with this, that would be awesome.

  • Look into Open Collective for Future Funding

    Eventually it will make sense to look at setting up something like Open Collective to help with future server costs. Right now the server costs around $14 a month (not including annual domain name & email hosting costs) so it's not really an issue, but down the road I probably won't want to cover all the costs of the project myself.

    Glancing at the criteria needed for acceptance to the Open Source Collective, it looks like Metron meets most of them, except having 100 stars on Github. So, if you'd like to help the project, one easy way would be to star the repository.

  • Manga / International Support

    It would be nice to make a final decision on whether to support Manga and Non-U.S. issues, right now I'm inclined not to but if we do add support it will necessitate some changes to the database for things like currencies, languages, and such.

I'm sure there will be other smaller items (like tweaking the site's css and such), but these are most items I'll be looking at in 2024. 🎉

October 2023 Update

· 2 min read
Brian Pepple
Founder of the Metron Project / Code Monkey

New Indexed Issues Milestone

The Metron Project reached a milestone this month by having more than 77,000 issues added to its database. I wish to give big thanks to everyone who has helped with the project. Thanks!

Metron-Tagger v1.6.4

Duplicate Page Removal

Just released a new version of Metron-Tagger, which adds a new experimental feature of finding / removing duplicate pages in a directory of comics. The feature will get the hashes for all the pages of the comics in a directory, and then ask the user if the image should be removed from the comics. This is done by using the user's system image viewer to display the duplicate image. Once the user has reviewed all the duplicated images it will ask the user whether to write the changes to the comics and then prompt them if they also want to update the ComicInfo.xml (if it exists) for the page changes.

This feature most likely will work best when running on a weekly basis, instead of on a whole comic collection, since it's asking the user to review all the duplicate images before writing the changes. I'm planning on reworking this to make it more feasible to use on a large comic collection, but that is pretty far down on my TODO list (but if someone wants to work on this I'm more than willing to review a Pull Request).

Note: I've been using this for the last month and haven't had any issues, but if you plan on using this I would suggest backing up any files when using this initially, and verifying it works correctly for you since once the changes are written they can't be undone! 💣

Miscellaneous Maintenance

The rest of v1.6.4 changes are primarily developer-side changes updating the toolsets for Metron-Tagger.

Comic-Tagger

Comic-Tagger recently released their Metron-Talker plugin which adds support for tagging any comic with metadata from Metron.

Latest Happenings

· One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

New Indexed Issues Milestone

The Metron Project reached a milestone today by having 65,000 issues added to its database. I wish to give big thanks to everyone who has helped with the project. Thanks!

POST API

Been spending the last few months working on a POST API to make it easier to add/update information on Metron. Currently it's only available to users with Adminstrators permissions, but later this year I'll look at the feasability of making it more readily available.

Comic Vine ID's

Another project I've been working on is adding Comic Vine identification numbers to the various resource items. This should help users match items between the two data sources. Currently, about 40% of the issues have been updated with this information and with any luck the majority of issues will be finished by the end of the month.

ComicTagger

One of the developers of Comic Tagger has been working on adding support for tagging digital comics with data from Metron, and hopefully in the next couple of months that merged.

Limited Series Type

· One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Made a minor change to the series types available by consolidating mini & maxi-series to the new Limited Series type. There isn't a real definite value for mini and maxi series, and really it makes more sense to not bother differentiating them. So, I've gone ahead and migrated the existing data for this change, and users should see this change. If you have any questions or thoughts, don't hesitate to contact me.

Using Podman to Run GCD Database Locally

· 6 min read
Brian Pepple
Founder of the Metron Project / Code Monkey

I've had a few users ask me how to run a local copy of the Grand Comics Database, so I figured I'd do a short write-up on how I do it.

I use Podman for containers instead of Docker, but the differences between them is fairly minimal and this should give you a good idea of how to do it. Anyway, let's get started!

First off, you'll want to download the most recent database dump from GCD which can be found here. Once it's finished downloading, unzip the file:

unzip current.zip -d /tmp/

You should see output similar to this:

Archive:  current.zip
  inflating: /tmp/2022-10-01.sql   

Ok, now that we've got the data from gcd, let's create a MySQL container. We need to first pull a MySQL image, which can be done by running:

podman pull docker.io/library/mysql:5.7

After it's done downloading the image, you can confirm:

podman images

which should return output similar to this:

bpepple@frodo ~ $ podman images mysql
REPOSITORY               TAG         IMAGE ID      CREATED       SIZE
docker.io/library/mysql  5.7         eb175b0743cc  19 hours ago  448 MB

Now we need to create the container. You'll want to replace user and mysql credentials with your own values. The volume flag is where on your filesystem you want to save the database. The Z suffix is used for systems, primarily Red Hat-based, that use SELinux. If you don't use SELinux that can be dropped:

podman run -d --name mysql -p 3306:3306 -e MYSQL_USER=bpepple -e MYSQL_PASSWORD=123456 -e MYSQL_DATABASE=gcd -e MYSQL_ROOT_PASSWORD=567890 --volume /home/bpepple/Storage/pods/mysql/:/var/lib/mysql:Z mysql:5.7

Now let's verify that the container was created and is running by using the following:

podman ps

You should see output similar to this:

bpepple@frodo ~ $ podman ps
CONTAINER ID  IMAGE                       COMMAND CREATED       STATUS           PORTS                  NAMES
400c957ae25d  docker.io/library/mysql:5.7 mysqld  7 seconds ago Up 7 seconds ago 0.0.0.0:3306->3306/tcp mysql

Now we just need to dump the data from GCD into the database. We can do this by running:

cat /tmp/2022-10-01.sql | podman exec -i mysql /usr/bin/mysql -u root --password=567890 gcd

Depending on the specs of your machine, this might be a good time to grab some coffee, since the SQL dump is very large and will take awhile to finish. Once it's done, we can start using the database by attaching to the container:

podman exec -it mysql mysql -ubpepple -p gcd

This command will prompt you for your password, and then you'll see the MySQL prompt where you can run a simple SQL statement like so:

 DESCRIBE gcd_issue;

Which should produce output similar to this:

+----------------------------+---------------+------+-----+---------------------+----------------+
| Field                      | Type          | Null | Key | Default             | Extra          |
+----------------------------+---------------+------+-----+---------------------+----------------+
| id                         | int(11)       | NO   | PRI | NULL                | auto_increment |
| number                     | varchar(50)   | NO   | MUL | NULL                |                |
| volume                     | varchar(50)   | NO   | MUL |                     |                |
| no_volume                  | tinyint(1)    | NO   | MUL | 0                   |                |
| display_volume_with_number | tinyint(1)    | NO   | MUL | 0                   |                |
| series_id                  | int(11)       | NO   | MUL | NULL                |                |
| indicia_publisher_id       | int(11)       | YES  | MUL | NULL                |                |
| indicia_pub_not_printed    | tinyint(1)    | NO   |     | NULL                |                |
| brand_id                   | int(11)       | YES  | MUL | NULL                |                |
| no_brand                   | tinyint(1)    | NO   | MUL | NULL                |                |
| publication_date           | varchar(255)  | NO   |     | NULL                |                |
| key_date                   | varchar(10)   | NO   | MUL | NULL                |                |
| sort_code                  | int(11)       | NO   | MUL | NULL                |                |
| price                      | varchar(255)  | NO   |     | NULL                |                |
| page_count                 | decimal(10,3) | YES  |     | NULL                |                |
| page_count_uncertain       | tinyint(1)    | NO   |     | 0                   |                |
| indicia_frequency          | varchar(255)  | NO   |     |                     |                |
| no_indicia_frequency       | tinyint(1)    | NO   | MUL | 0                   |                |
| editing                    | longtext      | NO   |     | NULL                |                |
| no_editing                 | tinyint(1)    | NO   | MUL | 0                   |                |
| notes                      | longtext      | NO   |     | NULL                |                |
| created                    | datetime      | NO   |     | 1901-01-01 00:00:00 |                |
| modified                   | datetime      | NO   | MUL | 1901-01-01 00:00:00 |                |
| deleted                    | tinyint(1)    | NO   | MUL | 0                   |                |
| is_indexed                 | tinyint(1)    | NO   | MUL | 0                   |                |
| isbn                       | varchar(32)   | NO   | MUL |                     |                |
| valid_isbn                 | varchar(13)   | NO   | MUL |                     |                |
| no_isbn                    | tinyint(1)    | NO   | MUL | 0                   |                |
| variant_of_id              | int(11)       | YES  | MUL | NULL                |                |
| variant_name               | varchar(255)  | NO   |     |                     |                |
| barcode                    | varchar(38)   | NO   | MUL |                     |                |
| no_barcode                 | tinyint(1)    | NO   |     | 0                   |                |
| title                      | varchar(255)  | NO   | MUL |                     |                |
| no_title                   | tinyint(1)    | NO   | MUL | 0                   |                |
| on_sale_date               | varchar(10)   | NO   | MUL | NULL                |                |
| on_sale_date_uncertain     | tinyint(1)    | NO   |     | 0                   |                |
| rating                     | varchar(255)  | NO   | MUL | NULL                |                |
| no_rating                  | tinyint(1)    | NO   | MUL | NULL                |                |
| volume_not_printed         | tinyint(1)    | NO   |     | NULL                |                |
| no_indicia_printer         | tinyint(1)    | NO   |     | NULL                |                |
+----------------------------+---------------+------+-----+---------------------+----------------+
40 rows in set (0.00 sec)

Great, we've got the database up and running! To logout from the container just type:

\q

To stop the container, you would just use the following:

podman stop mysql

And that's it! Podman has tons of niceties (like auto-updating images, using systemd for container start-ups, etc.) and I'd encourage you to read up on using some its many features. If it get some free time, I'll do a follow-up article about using Python to query the database and using that data for things like writing metadata to comic archives.

OS Upgrade

· One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Finally got around to upgrading the OS for Metron to Ubuntu 22.04.1 LTS, which overall, was fairly painless. Tests didn't find anything obviously broken, but if you do run across something that is, please open a bug for it. Thx!