Skip to main content

October 2023 Update

2 min read
Brian Pepple
Founder of the Metron Project / Code Monkey

New Indexed Issues Milestone

The Metron Project reached a milestone this month by having more than 77,000 issues added to its database. I wish to give big thanks to everyone who has helped with the project. Thanks!

Metron-Tagger v1.6.4

Duplicate Page Removal

Just released a new version of Metron-Tagger, which adds a new experimental feature of finding / removing duplicate pages in a directory of comics. The feature will get the hashes for all the pages of the comics in a directory, and then ask the user if the image should be removed from the comics. This is done by using the user's system image viewer to display the duplicate image. Once the user has reviewed all the duplicated images it will ask the user whether to write the changes to the comics and then prompt them if they also want to update the ComicInfo.xml (if it exists) for the page changes.

This feature most likely will work best when running on a weekly basis, instead of on a whole comic collection, since it's asking the user to review all the duplicate images before writing the changes. I'm planning on reworking this to make it more feasible to use on a large comic collection, but that is pretty far down on my TODO list (but if someone wants to work on this I'm more than willing to review a Pull Request).

Note: I've been using this for the last month and haven't had any issues, but if you plan on using this I would suggest backing up any files when using this initially, and verifying it works correctly for you since once the changes are written they can't be undone! 馃挘

Miscellaneous Maintenance

The rest of v1.6.4 changes are primarily developer-side changes updating the toolsets for Metron-Tagger.

Comic-Tagger

Comic-Tagger recently released their Metron-Talker plugin which adds support for tagging any comic with metadata from Metron.

Latest Happenings

One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

New Indexed Issues Milestone

The Metron Project reached a milestone today by having 65,000 issues added to its database. I wish to give big thanks to everyone who has helped with the project. Thanks!

POST API

Been spending the last few months working on a POST API to make it easier to add/update information on Metron. Currently it's only available to users with Adminstrators permissions, but later this year I'll look at the feasability of making it more readily available.

Comic Vine ID's

Another project I've been working on is adding Comic Vine identification numbers to the various resource items. This should help users match items between the two data sources. Currently, about 40% of the issues have been updated with this information and with any luck the majority of issues will be finished by the end of the month.

ComicTagger

One of the developers of Comic Tagger has been working on adding support for tagging digital comics with data from Metron, and hopefully in the next couple of months that merged.

Limited Series Type

One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Made a minor change to the series types available by consolidating mini & maxi-series to the new Limited Series type. There isn't a real definite value for mini and maxi series, and really it makes more sense to not bother differentiating them. So, I've gone ahead and migrated the existing data for this change, and users should see this change. If you have any questions or thoughts, don't hesitate to contact me.

Using Podman to Run GCD Database Locally

6 min read
Brian Pepple
Founder of the Metron Project / Code Monkey

I've had a few users ask me how to run a local copy of the Grand Comics Database, so I figured I'd do a short write-up on how I do it.

I use Podman for containers instead of Docker, but the differences between them is fairly minimal and this should give you a good idea of how to do it. Anyway, let's get started!

First off, you'll want to download the most recent database dump from GCD which can be found here. Once it's finished downloading, unzip the file:

unzip current.zip -d /tmp/

You should see output similar to this:

Archive:  current.zip
  inflating: /tmp/2022-10-01.sql   

Ok, now that we've got the data from gcd, let's create a MySQL container. We need to first pull a MySQL image, which can be done by running:

podman pull docker.io/library/mysql:5.7

After it's done downloading the image, you can confirm:

podman images

which should return output similar to this:

bpepple@frodo ~ $ podman images mysql
REPOSITORY               TAG         IMAGE ID      CREATED       SIZE
docker.io/library/mysql  5.7         eb175b0743cc  19 hours ago  448 MB

Now we need to create the container. You'll want to replace user and mysql credentials with your own values. The volume flag is where on your filesystem you want to save the database. The Z suffix is used for systems, primarily Red Hat-based, that use SELinux. If you don't use SELinux that can be dropped:

podman run -d --name mysql -p 3306:3306 -e MYSQL_USER=bpepple -e MYSQL_PASSWORD=123456 -e MYSQL_DATABASE=gcd -e MYSQL_ROOT_PASSWORD=567890 --volume /home/bpepple/Storage/pods/mysql/:/var/lib/mysql:Z mysql:5.7

Now let's verify that the container was created and is running by using the following:

podman ps

You should see output similar to this:

bpepple@frodo ~ $ podman ps
CONTAINER ID  IMAGE                       COMMAND CREATED       STATUS           PORTS                  NAMES
400c957ae25d  docker.io/library/mysql:5.7 mysqld  7 seconds ago Up 7 seconds ago 0.0.0.0:3306->3306/tcp mysql

Now we just need to dump the data from GCD into the database. We can do this by running:

cat /tmp/2022-10-01.sql | podman exec -i mysql /usr/bin/mysql -u root --password=567890 gcd

Depending on the specs of your machine, this might be a good time to grab some coffee, since the SQL dump is very large and will take awhile to finish. Once it's done, we can start using the database by attaching to the container:

podman exec -it mysql mysql -ubpepple -p gcd

This command will prompt you for your password, and then you'll see the MySQL prompt where you can run a simple SQL statement like so:

 DESCRIBE gcd_issue;

Which should produce output similar to this:

+----------------------------+---------------+------+-----+---------------------+----------------+
| Field                      | Type          | Null | Key | Default             | Extra          |
+----------------------------+---------------+------+-----+---------------------+----------------+
| id                         | int(11)       | NO   | PRI | NULL                | auto_increment |
| number                     | varchar(50)   | NO   | MUL | NULL                |                |
| volume                     | varchar(50)   | NO   | MUL |                     |                |
| no_volume                  | tinyint(1)    | NO   | MUL | 0                   |                |
| display_volume_with_number | tinyint(1)    | NO   | MUL | 0                   |                |
| series_id                  | int(11)       | NO   | MUL | NULL                |                |
| indicia_publisher_id       | int(11)       | YES  | MUL | NULL                |                |
| indicia_pub_not_printed    | tinyint(1)    | NO   |     | NULL                |                |
| brand_id                   | int(11)       | YES  | MUL | NULL                |                |
| no_brand                   | tinyint(1)    | NO   | MUL | NULL                |                |
| publication_date           | varchar(255)  | NO   |     | NULL                |                |
| key_date                   | varchar(10)   | NO   | MUL | NULL                |                |
| sort_code                  | int(11)       | NO   | MUL | NULL                |                |
| price                      | varchar(255)  | NO   |     | NULL                |                |
| page_count                 | decimal(10,3) | YES  |     | NULL                |                |
| page_count_uncertain       | tinyint(1)    | NO   |     | 0                   |                |
| indicia_frequency          | varchar(255)  | NO   |     |                     |                |
| no_indicia_frequency       | tinyint(1)    | NO   | MUL | 0                   |                |
| editing                    | longtext      | NO   |     | NULL                |                |
| no_editing                 | tinyint(1)    | NO   | MUL | 0                   |                |
| notes                      | longtext      | NO   |     | NULL                |                |
| created                    | datetime      | NO   |     | 1901-01-01 00:00:00 |                |
| modified                   | datetime      | NO   | MUL | 1901-01-01 00:00:00 |                |
| deleted                    | tinyint(1)    | NO   | MUL | 0                   |                |
| is_indexed                 | tinyint(1)    | NO   | MUL | 0                   |                |
| isbn                       | varchar(32)   | NO   | MUL |                     |                |
| valid_isbn                 | varchar(13)   | NO   | MUL |                     |                |
| no_isbn                    | tinyint(1)    | NO   | MUL | 0                   |                |
| variant_of_id              | int(11)       | YES  | MUL | NULL                |                |
| variant_name               | varchar(255)  | NO   |     |                     |                |
| barcode                    | varchar(38)   | NO   | MUL |                     |                |
| no_barcode                 | tinyint(1)    | NO   |     | 0                   |                |
| title                      | varchar(255)  | NO   | MUL |                     |                |
| no_title                   | tinyint(1)    | NO   | MUL | 0                   |                |
| on_sale_date               | varchar(10)   | NO   | MUL | NULL                |                |
| on_sale_date_uncertain     | tinyint(1)    | NO   |     | 0                   |                |
| rating                     | varchar(255)  | NO   | MUL | NULL                |                |
| no_rating                  | tinyint(1)    | NO   | MUL | NULL                |                |
| volume_not_printed         | tinyint(1)    | NO   |     | NULL                |                |
| no_indicia_printer         | tinyint(1)    | NO   |     | NULL                |                |
+----------------------------+---------------+------+-----+---------------------+----------------+
40 rows in set (0.00 sec)

Great, we've got the database up and running! To logout from the container just type:

\q

To stop the container, you would just use the following:

podman stop mysql

And that's it! Podman has tons of niceties (like auto-updating images, using systemd for container start-ups, etc.) and I'd encourage you to read up on using some its many features. If it get some free time, I'll do a follow-up article about using Python to query the database and using that data for things like writing metadata to comic archives.

OS Upgrade

One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Finally got around to upgrading the OS for Metron to Ubuntu 22.04.1 LTS, which overall, was fairly painless. Tests didn't find anything obviously broken, but if you do run across something that is, please open a bug for it. Thx!

Esak-1.3.2

One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Did a quick release this morning of the python wrapper for the Marvel API, Esak, which adds an option to have the cache results expire after a user-defined number of days. To update to the latest version just run the following command:

pip3 install -U --user esak

Darkseid-1.3.3

One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Did a quick release today of Darkseid that fixes a bug which prevented exporting rar (.cbr) comics to zip files (.cbz). If you use Metron-Tagger and export rar files, it would be worthwhile to update Darkseid which can be done with the following command:

pip3 install -U --user darkseid

As always if you run into any problems, don't hesitate to contact us on Matrix.

Trade Paperback / Collections Support

One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

Marvel Masterworks Cover

A user contacted me about adding Marvel Masterworks: The Uncanny X-Men series to Metron since we have recently added support for trade paperbacks/collected editions. I figured this would be useful exercise since it would show me what parts of the process could be improved to make it easier.

Currently, the process requires the editor to add all creators, characters, teams, etc. manually for the reprinted issues associate with the collection, much like a regular issue. One obvious way to simplify this is to automatically add any characters, team, story titles, etc. for the issues being reprinted. Currently, I've written an admin action to do this, but sometime in the future I need to expose this functionality so that anyone adding a Trade Paperback/Collection can save some time.

When I was creating Metron one of the choices I made was to add creators at an issue-level, and not at a story-level, since I wanted the process for adding an issue to be fairly simple. The downside to this approach is that adding information to a collection from an issue with multiple stories isn't really possible.馃槩

50,000 issues indexed

One min read
Brian Pepple
Founder of the Metron Project / Code Monkey

The Metron Project reached a milestone today by having 50,000 issues added to its database. I wish to give big thanks to everyone who has helped with the project. Thanks!

MetronInfo.xml

2 min read
Brian Pepple
Founder of the Metron Project / Code Monkey

So, one of the new projects I've been working on in my spare time is a new comic book xml schema called MetronInfo.

Currently, most applications use ComicInfo which originated from the ComicRack application, but due to it's history this format has some limitations like:

  • Poor handling of data. For example, cover date information that is split into separate integer types, instead of just using a date type.
  • Limited types metadata stored. For example, when tagging a comic book archive with an application like ComicTagger it would be useful to know where the metadata information was retrieved from (i.e. Comic Vine, GCD, Metron, etc.) and the resources identification number.
  • Lack of documentation. The Anansi Project is working on improving this, tho.

For now, the Anansi Project is working on documenting and extending (non-breaking changes) the ComicInfo schema, but in my opinion that time spent would be better spent on creating a new format not tied to a dead application. Due to the fairly unstructured nature of comic book archives it's trivial to tag one with the existing ComicInfo format and also a new format. That way users who use applications that only support ComicInfo.xml can still use them, but also gives them the option to use applications that could support the MetronInfo format.

My goals for MetronInfo.xml are:

  • Fix some of the deficiencies of the ComicInfo format.
  • Add information to make it easier for Plex-like application to identify data (series, character, etc/) when importing comics into their application.
  • Document the elements clearly so their use is not ambiguous.

Currently I'm working right now on finalizing the new format, and would appreciate any help or feedback from the comic book community in this. Feel free to leave suggestions either at the Discussions Page or contact use over on Matrix.

Once the new format is finalized I plan on writing a tool that will create a MetronInfo.xml file for any existing comic book archive that was tagged with ComicTagger or Metron-Tagger, and also retrieve any missing data from the source of information (Metron or Comic Vine).