How to choose what AI to use

My last guide to Chinese AI lasted 48 hours. Less than!

I published it on Friday. It became one of the best-performing videos on my YouTube channel - people want to know about Chinese AI eh? By Sunday Alibaba had announced Qwen3.8 and it was already out of date…

Then this landed in my WhatsApp group:

"It's pretty overwhelming Kyle. How do you exactly distinguish which model is best for which task?"

Yep. That is the actual story here. Let’s try to zoom out a bit from the specific releases (and boy there are a lot of them!) and instead look at the whole business of how to treat a new model.

The problem is trying to run a business while another "best model ever" arrives every few days. How do we deal with that noise?

And by god it is very noisy at the moment…

A launch post has three layers

Alibaba announced Qwen3.8 on Sunday. It says the model has 2.4 trillion parameters, that open weights are coming soon and that its performance is second only to Claude Fable 5.

Big claim. Maybe true. Maybe not.

First lesson: don’t listen to the launch post.

Guess what. The company promoting their new model is likely to say it’s pretty good. Funny that.

Qewn is still in preview. The full weights are not out. There is no model card, named independent benchmark or stable global API listing yet. A Max Preview is available through selected Alibaba products, including its China Personal Token Plan, Qoder and QoderWork.

So the announcement is real. You can get limited access. But the #2 ranking is still Alibaba talking about Alibaba. Obviously they reckon it is brilliant (!!). OpenAI, Anthropic and every other lab do the exact same thing.

Company launch posts tell you what to test. The proof comes later.

A quick reminder about open-source/weights

Qwen says the weights are coming. Cool. That means we can download the WHOLE model on our computers. Unless, you know, Trump blocks it. Which is looking likely…

But here is your daily reminder. This does not mean you can just “run it for free at home!”. I’m seeing a lot of posts on social media about this. It’s a great hook - you can run this Chinese model for free and never pay for Claude or ChatGPT again. Sounds great. But it’s entirely untrue.

At 2.4 trillion parameters, a rough four-bit version would still be about 1.2TB before runtime overhead. You are not chucking that on a MacBook. You need serious server hardware or (more likely) somebody else's cloud that you rent.

Run every model through four gates

OK back to the new models. How do you know what to smash and what to pass. Gotta be a better way to say that. Oh well.

When a new model appears, I like to ask four questions:

Was it announced? Find the first-party source, not a screenshot of a screenshot on Twitter. You’d be surprised how many new models don’t actually exist officially yet. This week everyone is talking about Opus 5 and its (apparently?) imminent release. So you need to be careful and make sure you are looking at an actually extant model and not random social noise!

Can you actually use it? Check the exact model, product, region and access route. A preview in a Chinese subscription plan is not the same thing as a normal global API. Models also release at different rates in different geographies - here in the EU we tend to get things last. Which is…awesome.

Has anyone independent tested it? Wait for proper benchmarks and people running real work. Producer numbers are where the evidence starts. I like to check out people I trust like Ethan Mollick, Simon Willison and Andrew Curran for their thoughts. Then benchmarks on sites like Artificial Analysis. Remember that the labs will always cherry-pick benchmarks that make them look good. So wait for independent reports.

Does it beat your current model on your work? This is the big one. Can the model do the work you want to use it for? And can it not only match your current model but instead BEAT it? Don’t shift your work for a passing grade here - only shift if there is a substantial delta. Otherwise you incur switching costs for…what?

Qwen3.8 passes gate one. It sorta passes gate two. Gates three and four ehhh not yet. So: we wait.

Make your workflow the leaderboard

If you still want to have a poke around, pick three low-risk tasks you already do. Run each one ten times on Qwen and your current model. Compare quality, time, cost and failures.

This is not based on vibes. Not somebody else's coding benchmark. Your actual boring arse work. Which, remember, is all that really matters here.

And do not switch because the new model is about as good. Moving prompts, context and workflows has a cost. It needs to be substantially better and/or substantially cheaper before the faff makes sense.

This is why I keep banging on about building one shared AI vault. Keep your context, files and processes outside a single model and trying a new provider is much less of a pain.

Honestly thought? Most of the time? Stick with what works and get the job done. I know that’s boring! I know you want to play with shiny new cool toys. But remember you have WORK to do! Important work. And blindly shifting from model to model chasing the new best thing gets you no closer to your goals.

I said it last week and apparently need to say it again: do not marry the model.

Until then…scroll on.

To the Task,

Kyle