Inspired by a post in the Cast/Crew Edit 2 thread, I started to look for AI APIs that would allow me too look up IMDb entries. ChatGPT said it could, but the API wouldn't be free. Well, for a freeware program that is a no-no, in my opinion. So I turned to Microsoft Copilot. It said that it couldn't provide IMDb data via the API. I didn't bother to check if the API was free. Finally I asked Gemini if it could do it. It said that it could, and that there was a free API (with some limitations).

That sounded promising, so I did some initial testing, and it worked.  I instructed Gemini that I wanted all cast and crew, and I wanted the credits exactly as in IMDb. On closer testing I found that Gemini failed in both aspects. Running the same movie multiple times would produce different number of credits, and slight variation of roles and crew jobs - not as credited. I tried changing the prompt to impress the criteria on Gemini, but that didn't help. That isn't good enough.

So, at least for now, this idea has been shot down.  Disappointing, because I was really hoping to be able to mine IMDb data without having to resort to screen scraping. So I guess it's still either Cast/Crew Edit 2, or being content with TMDB data using TmdbInfo. 

That may be true, but that’s beside the point. The data should be checked against the credits anyway.  You use it as a starting point. However, if the data is not complete, you’re no better off than if you use data from TMDb.

Also, transforming crew jobs to Profiler format is hopeless if the source isn’t consistent.

I may take another look at this in the future. But it feels like too little, too late. I’m not sure that there are enough users who care, in order for this to be worth the effort.

Quoting GSyren:

Quote:
That may be true, but that’s beside the point. The data should be checked against the credits anyway.  You use it as a starting point. However, if the data is not complete, you’re no better off than if you use data from TMDb.

Also, transforming crew jobs to Profiler format is hopeless if the source isn’t consistent.

I may take another look at this in the future. But it feels like too little, too late. I’m not sure that there are enough users who care, in order for this to be worth the effort.

What about going direct to the IMDb Non-Commercial Datasets here.

Just for some perspective, I ran a line count on all the IMDb files:

title.principals.tsv - 99 738 115 lines
title.akas.tsv - 57 439 274 lines
name.basics.tsv - 15 378 039 lines
title.basics.tsv - 12 539 092 lines
title.crew.tsv - 12 537 727 lines
title.episode.tsv - 9 685 416 lines
title.ratings.tsv - 1 676 413 lines

I guess it would be theoretically possible to write a program that loads (some of) these files into a database, and get full cast and crew listings from it. title.principals.tsv and title.basics.tsv would probably suffice for movies, add title.episode.tsv for TV shows. But would anyone be interested?

Quoting ObiKen:

Quote:
My mantra is the solution has got to be simple, when I ask for a banana, I don't want a gorilla holding that banana.

I take it that you didn’t realize you were suggesting a gorilla? 
I think I may build this for myself, to see if it climbs the Empire State Building with the banana.
"T’was fruit killed the beast" 

I tested loading the largest file into an Sqlite table. I found that I could load about 200 line per second. That would mean that it would take over 5 days (!) to complete.

I'm a fairly patient guy, but I would go bananas waiting that long, so I aborted the test. 

It's nice of IMDb to dump their database tables into downloadable files (for non-commercial use), but I can't help but wonder who actually finds a use for them.

So, this whole project was pretty much a waste of time. Oh, well, I did get a little refresher on how to use Sqlite, so at least that's something. I wonder if I can think of something else where I can use it ... 

I didn't think that it could be sped up enough to make it palatable, but how wrong I was! 
I changed the program to use transactions of 10,000 records each. Now it took 36 minutes instead of (the estimated) 5 days! Together with the other two files I needed, building the database took about an hour (not counting the coding time).

Now I just have to build a program that queries that database. But that's a job for another day.

Thanks for pushing me, ObiKen! 

PS The database became almost 7 GB.