Gentlefish has a good post, but I did want to touch upon a few things.
...our valuing the gold is pretty arbitrary as well
Gold actually has qualities that explain at least some of its value. It is a "noble" metal, which means it is naturally resistant to oxidation and corrosion. It also is a softer metal, making it easier to work this. This was in addition to being aesthetically pleasing to our ancestors. These are actually good reasons to seek it out and use it. True if it were more readily available it would have had a lower value, but because it did have both practical and decorative/symbolic applications, its scarcity on top of its other properties then made it an excellent store of value. "Money" makes it easier to labor by being a "store of value". You could use any goods, but most are perishable to one degree or another and/or subject to wild swings in value. So using "gold" or other precious metals could give you a store of value that would rot or otherwise significant degrade and also was unlikely to be subject to wild inflation or deflation just because of a bumper crop or famine... problems with using an otherwise more logical form of value like "food".
Edit: Oh an in the modern world, gold is one of the reasons certain electronics or similar devices cost so much as it is a good conductor of electricity. Combined with some of its other physical properties and this has made it a useful building component!
I remember seeing a discussion or two* on the practice of sharking (offering trades to/seeking trades offered by people who are unaware of the market price of an item and thus willing to trade it for significantly less) in virtual economies in which someone defended it using this exact argument - that the value is determined by how much worth the individual sees in it - in addition to the argument that if everyone comes away from the deal content everything's fine. Pretty shady, if you ask me.
This is something to be careful with because of an aspect you leave out entirely: personal accountability. There is also the distinction between things that are "questionable" or "shady" but those that are flat out "right" or "wrong". There are sometimes things that are "wrong" but trying to enforce what is "right" would actually cause a greater "wrong". "Sharking" is a good example; I do consider it "wrong" to unduly enrich one's self by taking advantage of the ignorant or the desperate
but where does one draw the line? "Price controls" have a nasty tendency to cause economic stagnation
and to backfire horribly as someone figures out how to manipulate them.
The buyer has a responsibility to be informed; if anything the solution on (as an example) the PTCGO would be to flat out tell people to visit the In Game Item Exchange subforum on the official PTCGO message boards and read for themselves what the going rates are for most cards. If on the other hand you've got someone commanding a premium for trades but earning it because (like the effective trading houses) you can come to them essentially 24/7 and get a trade accomplished within the hour, well they've added value to that trade through their effort.
If you've got someone savvy (being savvy is a
good thing) that
knows the current prices are inflated just because of novelty (like when
XY: Double Crisis was new) and thus are making offers well below what the trading companies are asking, they aren't ripping anyone off either. Yeah I might be biased because that was me, but I made sure over on another website that our Card of the Day section told it straight (some of the lowest scores we've ever had for a Top X list). When the set was new many of the prices were double what they are now with the most in demand cards going for like 6 boosters per copy, and if you're patient
even when the sets were new you could get the rarest cards from
XY: Double Crisis for just one or two booster packs, so if you're in an absolute hurry to get them it is just barely worth the price of going through a trading house.
Still... if we want to be technical then we could amend it to say "It is what well informed buyers are willing to pay consistently pay."