These cases would be where appeals could come it handy. While it makes sense that actual hackers get slapped with bans, but no way of getting back into the game, it seems unfair not to provide innocent players who made an honest mistake a sort of safety-net to protect their progress and game privileges.
One way to ensure this kind of thing doesn't happen even though you've modded single-player before is to get a clean install of the game. That way there is a guarantee that no stray mod files stay in the system, and that you'll be playing with a clear, stock version of the game that the anti-cheat system will recognize as fair-play. This would, however, require you to uninstall and re-install the game entirely.
Another "common" cause of false-positive bans, as far as reports are to be believed, are various third-party programs with in-game overlays. The problem here is that a number of overlays function on a similar technical basis as GTA Online mod-menus and hacks, thus ticking off the watchdog. Not all overlays cause bans, however there is no way to know for sure whether a particular program will trigger a ban unless you have a fantastic grasp of coding principles and access to the source code.
That said, the community has compiled a list of high-risk programs. Discord's overlay is often implicated, and the Rivatuner overlay for the MSI Afterburner program is almost guaranteed to get you banned. Teamspeak, Bandicam and OBS are suspects, though many claim to use them frequently without any issue.
Funnily enough, the official Windows 10 Recorder is said to be a high-risk program, as is Fraps, Mirillis Action and XSplit. If you want to err on the side of caution, stay away from D3DGear and Dxtory, though these fall into the category of being suspect, as again, they are often used with no ill effect.
While the overlays of these programs might tick off the anti-cheat, keep in mind that using them without the overlay is perfectly safe. Sure, you might need to Alt+Tab out of GTA Online once or twice, but they are manageable without the overlays under normal circumstances, so you can still play safely without having to give these utilities up.
Another thing to avoid, no matter how tempting, are certain glitches that pop-up every now and then. It might be tough deciding whether a glitch is safe or not, however a good way to gauge risk is to see how directly it is related to the acquisition of cash. To be clear in the distinction, glitches are errors in the game that arise from programming bugs and do not use third-party tampering, like hacks.
Money glitches, as rare as they are, and duplication glitches, both of which lead to the near instant acquisition of high amounts of cash, are likely to tick off the anti-cheat. There is a general limit to how much cash one can make legitimately in GTA Online in a given time-span without buying Shark Cards, and if this limit is exceeded, the anti-cheat will assume foul play.
Other glitches can be safe. Outfit glitches, which have nothing to do with money and simply allow you to wear special outfits in Freemode, will never get you banned. Various glitches used to circumvent cooldowns, such as this method used in Import/Export, are also safe, since they generally don't throw large amounts of cash at you.
You might want to keep these suggestions in mind for future sessions of GTA Online, or you might not. Even if you purposefully do the exact opposite of what is written in this guide, but don't outright hack, you're still probably going to be playing to your merry with nary a ban in sight. These are just suggestions, not rules. That said, consider how annoyed you would be, should you be wrongfully slapped with a ban costing you all your progress.
Have you ever heard of verified cases of false-positive bans in GTA Online?