math.floor((HP/5+Defense+Power+Poison Damage+Fire Power)/3)
This would seem off because, as I said before, it favors HP over the other stats. That and it would make glass cannons more favorable kills. Let me show you what I mean with a little exercise. Let's say a monster's HP is 250, its defense is 50, and its power is 50. The formula would work like this:
(250/5+50+50)/3=50
That's 50 EXP for one monster. Should we double either defense or power, this is what we would get.:
(250/5+100+50)/3=66.666
(250/5+50+100)/3=66.666
The monster would give us 66 EXP when killed. Doubling power would make a monster easier to kill, but more risky to fight against since its damage output is quadrupled. Doubling defense, on the other hand, would make a monster harder to kill since its endurance is quadrupled even though its power wouldn't seem all that effective.
If we doubled HP instead, here's what would we get.:
(500/5+50+50)/3=66.666
Pretty much the same deal as defense, but unlike defense which quadrupled endurance when doubled, HP merely doubles a monster's endurance when it gets doubled.
This is a little weird because, well, where the issue is most apparent is between Grunts and Leaflings (which both have roughly similar power levels). Here are their stats.
Grunt
HP: 290
Defense: 24
Power: 54
No elemental bonuses
Leafling
HP: 165
Defense: 34
Power: 52
No elemental bonuses
With how player-to-monster damage works currently, here are their endurance ratings.:
Grunt: (290/5)*24²=33408
Leafling: (165/5)*34²=38148
Essentially a monster's endurance is equivalent to how much damage output you would need before you can one-shot a monster exactly 50% of the time. This is essentially important because it is commonly recommended by the community to grind a mob as soon as you can three-shot it. To 3-shot a Grunt, you would need to deal 97 damage. I don't want to get into too many details this time, so I'll provide a visualization of how it would look if you can 3-shot a Grunt 50% of the time.:

The key for this chart would go as follows:
Green: 3 hits or less
Yellow: 4 hits
Orange: 5 hits
Red (not shown): 6 or more hits (purple would've gone for 11 or more hits, but we're not looking at Stalagmites)
Given you would have to round up the amount of hits you need to kill a monster instead of down, while you do have a 50% chance of 3-shotting a Grunt, you're more likely to four-shot (and oddly even five-shot as shown here) a Leafling. Yet, Grunts hand out 45 EXP while Leaflings hand out 39 EXP. Just making the Leafling's Power equal to the Grunt's power would only bring its EXP value up to 40. That wouldn't exactly sit to well statistically speaking. (Before any of y'all bring up poison damage, you would need to upgrade poison 3 or 4 times before you start seeing any significant changes going off the math done here; otherwise the results wouldn't change much. Even then, a P5 weapon would've needed to land two 125-damage or 126-damage hits for a slim chance of two-shotting a Grunt while it would've been impossible to two-shot a Leafling under the exact same conditions).
I decided to create both a chart and a formula for monster EXP rewards: two to be precise.
Both formulas make calculations based around a monster's overall endurance, which is as follows:
Monster Endurance=HP/5*Defense²
Taking that into account, Formula 1 looks like this:
Monster EXP Formula 1: math.ceiling((√(Endurance)+Power+Poison Damage*1.2+Fire Power*1.2)/5)
Formula 2, on the other hand, looks like this:
Monster EXP Formula 2: math.ceiling(((√(Endurance)/2)+Power+Poison Damage+Fire Power)/3)
I went for the Math Ceiling function in order to allow Squishers to provide 1 EXP (going with the Math Floor function would have them give out 0 EXP).
Putting both formulas together, this was the chart I got.:

Personally I like Formula 2 (listed as "New 2") more because it gives monster power and elemental boosts some relevance while at the same time still taking a monster's overall endurance into account. Personally, I would amend the formula with the following variables:
Boss Modifier: 3x (bosses take 12 minutes to spawn and were intended to require a group of 3 people working together to kill. Petunias, by comparison, took at most 130 seconds to spawn)
Artificial Monster Modifier: 0.5x (Monsters that come from artificial spawners like the Leafling Button and the Singing Trees are likely to cause influxes in EXP gains, so halving their EXP reward should make naturally-spawning monsters more favorable. In other words, a natural Leafling would provide 50 EXP while a Leafling Button Leafling would instead provide 25 EXP)
Yep. There ya have it.