FORTUNE — You’re not supposed to discuss religion or politics at the dinner table, and you really shouldn’t ever bring up gold in polite company either.
For many boosters, gold takes on a kind of religious political importance, as some investors feel they can turn to it for deliverance from seemingly wasteful and duplicitous governments who resort to inflation to sustain ever-growing bureaucracies.
It’s the political extremism of gold’s fervent supporters that often leads people to ignore their claims of gold market manipulation. But there has been increasing evidence that something fishy is going on in the market for gold bullion, and in so-called paper gold, or various financial derivatives backed by gold. No, there’s no smoking gun, but there is a ton of circumstantial evidence of shenanigans that should give any potential gold investor pause before jumping into the market.
MORE: Why gold might drop another 50%
1. Allegations of LIBOR-like price fixing: Much like the LIBOR benchmark interest rate is used to value various financial products, the “London Gold Fix” is a price used by central banks, gold miners, and jewelers to value gold. It is set by a consortium of gold-dealing banks — Barclays (BCS), Deutsche Bank (DB), Societe Generale, HSBC (HSBC), and Bank of Nova Scotia (BNS) — based on the demand those banks are seeing from their clients.
Bloomberg News reported on Friday about a paper being drafted by Rosa Abrantes-Metz at NYU’s Stern School of Business and Albert Metz, a managing director at Moody’s Investors Service, that analyzes gold price movements around the times of these price-setting conference calls between banks. According to the Bloomberg report:
From 2004, they observed frequent spikes in spot gold prices during the afternoon call. The moves weren’t replicated during the morning call and hadn’t happened before 2004, they found.
Large price moves during the afternoon call were also overwhelmingly in the same direction: down. On days when the authors identified large price moves during the fix, they were downwards at least two-thirds of the time in six different years between 2004 and 2013. In 2010, large moves during the fix were negative 92 percent of the time, the authors found.
If these suspicious price movements are indeed evidence of price-fixing, these banks could hypothetically use information about the trajectory of gold prices to profit from that knowledge, at the expense of ordinary investors.
2. The mysterious case of German gold repatriation: In January of last year, the German central bank announced that it would repatriate half of the Bundesbank’s gold holdings, which were then being held in the United States, Britain, and France. Germany had built up such gold stashes through many years of trade surpluses, but the gold remained in foreign central banks because of the difficulty of safely shipping billions of dollars of heavy gold holdings from one country to the next.
MORE: 4 reasons inflation is finally about to take off
According to the Bundesbank’s original plan, it would repatriate 300 metric tons of gold from the Federal Reserve of New York over the course of eight years. But according to a January report in the German newspaper Die Welt, only five metric tons of gold had been transferred from New York to Frankfurt by the end of last year. Will Becker, an analyst at research firm Behind the Numbers wrote in a note to clients earlier this month:
But doesn’t the U.S. Supposedly have the world’s largest gold reserves? 8,133 MTs, right? We are then told by various central bankers that the reason for the paltry and delayed shipments is “logistical” in nature, due to the “challenges” of moving so much gold. OK, so millions of barrels of oil can be exported out of the Middle East every day, refined and distributed around the world. But a few gold bars in convenient, liftable sizes out of the U.S. … uh, impossible … Something doesn’t add up here.
3. Suspicious inventory levels in Gold ETFs: Becker’s suspicions of the physical gold market has also led him to question gold ETFs, like the popular SPDR Gold Shares Trust (GLD).
These ETFs have allowed the average investor to get exposure to the bull market in gold, which began around 2000 and lasted until gold’s tough 2013. But digging into SPDR’s prospectus has led analysts like Becker to question whether these shares would protect investors against financial calamity, which is ostensibly the entire point of including gold in an investment portfolio.
Becker points out that the only agents authorized to actually redeem shares of GLD are a dozen or so “authorized participants,” or large banks like HSBC, which is the custodian of the trust. The gold in the trust can also be accessed by six sub-custodians, and those banks can select other representatives to hold the gold. In other words, there’s no way for an investor in GLD to really know where the gold is. Writes Becker:
Since only a portion of GLD’s physical gold bars are held by the Custodian and the Trustee is unable to visit the premises and examine GLD’s gold or any records maintained by subcustodians, a comprehensive audit that covers all of GLD’s physical gold is not possible.
Furthermore, while the price of gold is up roughly 10% in 2014, GLD only increased their holdings by 1%. Becker worries that this lag may indicate that large-bank authorized participants would rather sell physical gold than buy shares of a gold ETF.
4. Strange trades in gold futures: The holy grail for gold conspiracy theorists is to connect the suspicious behavior listed above with central banks across the wealthy world, which presumably wish to suppress the price of gold because it threatens the legitimacy of central-bank backed fiat currency.
Paul Craig Roberts, an economist and former Assistant Treasury Secretary, believes he has done just that. In an article published on his website last month, he pointed to several instances in December and January in which massive amounts of gold, represented by futures contracts, were dumped onto the market within a matter of minutes. Only sellers who wished to manipulate the price of gold downward would want to operate in such a manner, he argues. Otherwise, you would try to sell your contracts in small batches to make sure you got the best price for them.
MORE: For investors, diamonds might be the new gold
To be sure, none of the above proves manipulation in the gold markets on the part of large banks or the government. But in light of recent revelations that central banks and large financial institutions were manipulating benchmark rates like LIBOR during and after the financial crisis, it’s fair to view the gold market with suspicion.
