In fact, the area north of the Kawarthas, bounded by Gravenhurst, Huntsville, Bancroft and Madoc, could support a fair number of larger communities. It is on the Shield and not much use for agriculture. It used to be logging and mining country. I don't know about now. But it has a lot of lakes and rivers for waterfront communities.
View attachment 80554
One problem, common to many non-urban areas, is employment. Unless you want to turn them into bedroom, long commute communities, there's not much to do, year-round, in places like Bancroft or MacTier. There is effectively no more logging or ag any more (there never was a non-pioneer agricultural base except in southern Kawartha Lakes). Even larger communities struggle to maintain a non-retail commercial tax base.
An additional problem if there ever was an attempt to create large urban areas would be water and sanitation. Without drawing on surface water (other than Georgian Bay, there are no really big bodies of water) I don't know how much water could be sucked out of the Pre-Cambrian Shield, or re-absorbed back into it.
It's a long commute, particularly in the winter. Communities on existing rail lines that could be enhanced for commuter passenger rail may have a bit of a future edge, but there is no rationale economic argument to carve a passenger-only rail line to Haliburton.
Affordability is relative to supply.
Kingston and the surrounding counties have some of the lowest vacancy rates in North America. While the average is far below the GTA, bidding wars occur for some of the shabbiest of homes that really and truly aren't worth the foundations they're sitting on.
Also, the fact that the GTA refugees artificially inflate the market doesn't help maintain affordability for locals or anyone coming from not the GTA.
Plenty of CAF members are still living in HWY 2 motels because of this.
The prime example of that is Prince Edward County. Long before the more recent work-from-home, Covid-induced price inflation, The County used to be cute rural area with a few small towns. Then it was 'found' by city folks. Wineries, bistros, art galleries, etc. were built to attract more city folks. The economy is probably stronger (I don't know if it is year-round - I get the sense that is quite seasonal), but prices of everything has increased to match. As other areas found during Covid, the expat/commuting/escaping home buyer makes it tough for the existing mechanic, store clerk or farmer.