dmg conversation in 80% case

In PoE2, Conversion happens in two steps. Conversion applied to a skill from it's own skill gem applies separately first before other conversion. This allows conversion from gear/tree to apply to damage that was converted by a skill gem.

In the case you describe, if the active skill gem converts 80% of it's physical damage to fire, assuming for simplicity it started with 100 physical damage, it will have 80 fire damage and 20 physical damage after skill conversion, then Avatar of Fire's non-skill conversion will apply, converting 75% of the physical damage to fire (converting 75% of the fire damage to fire has no effect - converting something to the same thing is ignored entirely), resulting in 95 fire damage and 5 physical - which becomes just 95 fire damage since Avatar of Fire prevents dealing the physical damage at all.