What are the rules for substituting の with ん?

As a Phenomena of Fast/Casual Speech

As Tsujimura(2007) describes, this a non-mandatory phonological process (not a syntactical one) called nasal syllabification. Consider these examples and a non-example as they might be written in native orthography:

1. 来るのなら → 来るんなら
2. 君のうち → 君んち
3. しらない → しんない
4. かえらない → かえんない
5. おくらない → おくんない
6. 学者になる → 学者んなる
7. 僕のうち → 僕んち
8. 君の名前 *→ * 君ん名前

Now compare IPA phonemic transcriptions:

1. /kuɾunonaɾa/ → /kuɾunnaɾa/
2. /kiminoutci/ → /kimintci/
3. /ɕiɾanai/ → /ɕinnai/
4. /kaeɾanai/ → /kaennai/
5. /okuɾanai/ → /okunnai/
6. /ɡakuɕaninaɾu/ → /ɡakuɕannaɾu/
7. /bokunoutɕi/ → /bokuntɕi/
8. /kiminonamae/ *→ * /kiminnamae/

Tsujimaru says this:

Certain functional words containing /n/ often lose their accompanying vowel, and as a result become a moraic nasal. In addition to the vowel loss, if there is an another second word-initial vowel immediately following then it too gets deleted.

And so isn't clear about the following:

• Which words are eligible for this phonological alternation
• Why the last example is unacceptable

Although Tsujimura didn't explicitly say so, I believe the nasal syllabification targets only mora.

As Vance(2008) observes about /n/:

When /n/ is word-medial the following phoneme can be any vowel or any syllable-initial consonant.

which means the alternations /nV/ → /n/ and /ɾa/ → /n/ will not produce an inadmissible phoneme sequence. Together with the fact that /n/ can be a word-final syllable, we see that all three relevant processes in nasal syllabification (defined below) will never result in an inadmissible sequence of phonemes. This fact has a crucial implication in that nasal syllabifications are phonological (alternational) processes not phonetic (derivational) processes. Meaning that once /n/ emerges, nasal syllabification has nothing more to do with it. Namely, rules such as the following have nothing to do with nasal syllabification, and are covered by other independent processes that happen afterwords:

ん becomes [n] before /t/, /d/, /n/ and /r/

That's as far as I've read on the topic so far. I've never seen any formal rules but I might propose three myself (the last rule describes the actual alternations):

• Comparing the last non-example to the first and second, either the eligibility criteria must discriminate on a semantic basis or that we should just dismiss this exception.
• The eligibility criteria does discriminate on a morphological basis, specifically that particles of the form /nV/ and the mora /ɾa/ when belonging to non-past negative vowel roots ending in /ɾ/ are eligible (these statements pertain to morphemes).
• If /nV/ is eligible, and a second V follows on the right (ignoring morpheme boundaries), then /nVV/ is eligible, and /nVV/ → /n/ (see examples 2 and 7). Otherwise /nV/ → /n/. The second vowel deletion process ignores morpheme boundaries. Lastly, if /ɾa/ is eligible then /ɾa/ → /n/.

As for @sawa's examples:

• こんなの → こんなん Phonemically we have /konnano/ → /konnan/ which is not problematic because the /n/ emerges after the syllabification has occurred. The resulting word-final (moraic) /n/ is admissible in Japanese (and is in fact realized as [ɴː] in this case).
• "sentence-final particles/nominalizers as in 食べたの →　食べたん" Basically the same thing as above; The /no/ in /tabetano/ is eligible and the resultant /n/ will be followed by either a vowel, a consonant, or a morpheme/word/phrase boundary. But all such resultant sequences are admissible.
• 食べぬ → 食べん Same thing as こんなの → こんなん because in either case we have /nV/->/n/ (the vowel V is not discriminated).
• にて → んて → で Only the first alternation is relevant to nasal syllabification, but to attribute the alternation to NS, that に must be eligible. I only stipulated that particles (postpositions) of the form /nV/ are eligible, so for NS to work here there would have to be a morpheme boundary between the に and て.
• それで → そんで The source word is /soɾede/ but all I know for sure is that /ɾa/ is eligible. Considering that in /nV/ → /n/ the vowel wasn't discriminated, we could potentially have the rule /ɾV/ → /n/.

And for @Pacerier question:

Btw when you say "does happen in speech", do you mean that the speaker knowingly (with intention) does a 「おれん…」 instead of a　「おれの…」?

Nasal syllabification is not mandatory; given the the appropriate context (fast/casual speech) an eligible mora undergoing syllabification is entirely optional. Whether or not a speaker is aware or will remember I don't know.

I'm going to assume that you specifically mean the "genitive" の, since all your examples are of this kind (i.e. not substantivizing の, or just random のs).

I'm guessing here, but I think it's likely that the substitution can only happen when the substituted ん would thereby be pronounced as a phonetic [n] (sorry, I can't write IPA, but feel free to edit my post as necessary). In the examples you have given, where the substitution is possible, the pronunciation will be

heyan'naka
orenchi
orentoko

To recap, ん becomes [n] before /t/, /d/, /n/ and /r/ (I'm not sure if this varies slightly depending on speaker. I'm open to comments or edits if this is not precise).

Also, it doesn't seem to happen when it would case two んs in a row, something that normally doesn't happen in Japanese.

In most of the examples in the second batch you've given, the substitution is unlikely to happen, since the pronunciation of ん would not be [n]:

ore(uvular N)kaasan
ore(nasal e)ie
ore(uvular N)sensee
enu(nasal u)enu

As for

orenteeburu

I think this one does happen in speech, although it wouldn't appear in writing. It's probably a phrase not occuring often enough to have become a fixed phrase like the first batch of examples you gave.

I was going to leave a comment since maybe I'm way out on a limb here, but it got long so I'll just make it an answer..

In these cases I think it's important to remember that languages evolve. They aren't really built from rules, that's just how we analyze them. Contractions like these happens naturally when this happens, and there may not be any specific rules to cover it. It's simply people being lazy.

This is also why it's discouraged to write ん in place of の, because written words are supposed to adhere to the rules we've made for them, and this is strictly speaking "wrong". What this really means of course is that it just hasn't been widely accepted as proper yet. (and maybe never really will)

So tl;dr; Substitute の with ん in speech or "casual writing" when it simply rolls easier of your tongue that way. (or copy natives, of course) It's fine as long as there's no ambiguity. (and you don't sound rude for being too sloppy)