I wonder if this is a very simple relationship. On a gasser as you advance spark you get quicker/easier starting, but the power band moves up slightly with each degree of advancing. Retarding the spark contributes to more bottom end power and a loss of top end. At a certain point the spark is advanced too far, causing detonation/pre-ignition.
Possibly when advancing the pump too far a person is approaching or arriving at detonation, causing an increase in combustion temperatures but not concurrent power increase? Very hard to hear detonation in a diesel power cycle.
Interesting that water/meth equipped trucks can take some pretty wild advance, as the W/M douses some of the combustion fire, cools mixture and adds power through ignited methanol.
I believe that part of it is very similar to a petrol engine. Except for the detonation and the added complication of turbocharging.
The whole idea is to get the peak combustion pressure occuring at the crank angle where it generates the most torque. With diesel burning speed not changing a whole lot with engine rpm and maybe the injection pump advance not quite advancing enough to compensate entirely, leaving the final pump timing to set the best operating point.
Detonation on a diesel can't happen as there's no fuel present before the injection. But what can happen is peak pressure occuring too early so the charge cools through the cylinder walls and head before it can exert it's max torque on the crank.
You also have the negative work of the piston working to compress a gas mixture which is burning and trying to expand sooner than usual.
Which brings me back to the question "how do we know where on the best efficiency point we currently are"?
I'm thinking cylinder head temperature is a very good clue, but it only works as a comparison (change timing, see if it's better or worse).