My thinking was that towed and rifled mortars would be under the purview of artillery, and smoothbore mortars that can be packed up be infantry. So the Soltam K6/M120 120mm mortar, L16 81mm, M224 or M6 60mm and maybe even a 51mm would all be infantry, and the M327 would be artillery.
Remember what the fundamental differences are as between a gun and a mortar. Guns generally fire at a flatter trajectory, have a stronger chamber to allow heavy pressure which equals faster time of flight and range. The heavier barrel of the gun also leads to better accuracy. The rifling in a mortar is immaterial as is its being towed.
The thing is to think of indirect fires in both layers and range bands. The initial layer and the shortest range goes to mortars which provide intimate support to their battalion. Its guaranteed to be there ready to support the battalion at all times. It does not integrate well into the fire of other battalion's mortars. They basically play in their own space.
Artillery comes with a host of capabilities which an infantry battalion can't afford to support - it has other priorities and tasks. With its longer range, accuracy and communications network, artillery can range widely across the front and mass fires well with other guns. A brigade and even a division or corps can concentrate the fires of all of its artillery wherever it wants on a moments notice. Add in to that surveillance and target acquisition artillery, air defence artillery and rocket and missile artillery all integrated with a communication and coordination capability that allows artillery to strike deep into the rear of an opponent and break them up before they get close enough to be engaged by mortars. In effect guns are the next layer and rockets and missiles the third.
The point is that just because mortars are a type of indirect fire, doesn't mean they are part of an artillery function. They work well doing what they are designed to do, work with the direct fire and manoeuvre elements that are organic to the battalion. They are part of a complete team at the battalion commanders beck and call which are able to operate well within the battalion's area of operations.
Sure, you can have gunners man mortars - and some countries do - but why would you? I'll go back to the early parts of Advancing with Purpose and mortars were taken away from the infantry (and mech battalions still don't have them back.) No one was happy with that - neither the grunts nor the gunners. The grunts didn't want to lose the mortars, and the gunners didn't want to have them.
It was a fiscal PY exercise taking people from the infantry for various other roles including creating CMTC. The artillery was given no extra people to take over 9 mortar platoons (and in fact they didn't take over 9 platoons. The initial plan was for each RegF artillery regiment to have one mortar battery which meant 12 mortars for a brigade where there had been 24. But even worse came about in 2005. Each regiment had 18 guns at the time - 12 M109s and 6 LG1s - but they were then cut back to two six-gun batteries who also had to man the mortars in a weapon's locker format. When the M777 appeared, no extra people came with them so the six seven-man detachments evolved into four ten-man detachments - so eight guns per regiment plus a few mortars in the stores for "just in case".
Once again, the concept turned around deploying battle groups. In 2002 for Op Apollo the artillery was only required to provide a 4-tube mortar group. The winds and weather in the mountains made the mortars a poor choice to deploy. Much later in 2006 and thereafter each 10-man M777 detachment also had two 81mm mortars which were used whenever appropriate for close in targets that would benefit from high angle fire. Batteries went out with either four or six guns and eight or twelve mortars. It's a very silly system that barely worked for an insurgency and would collapse in a matter of hours in high intensity combat.
The 81mm mortars are still hanging around but haven't gone back to the mech battalions because there simply aren't any infantry PYs for them as it stands. I mean does anyone in their right mind actually think that a bunch of gunners in a soft skinned truck (all the TLAVs are gone and who knows how many ACSVs the arty will get) and an 81mm mortar are going to accompany a LAV battalion? Whose minding the M777s while they are swanning around trying not to be killed?
Honestly, speaking as a former gunner, I don't want to see any mortars going to the artillery. It would perpetuate a stupid idea and probably lead some of the bean counters to conclude "that's good enough; we don't need any of the expensive guns or their ammunition." And folks will die. Right now we have more than enough 81mm mortars for the light battalions. Get some new mortars for the mech battalions - none of that towed shit - get them a proper modified ACSV with a 120mm Nemo mortar turret.