This is an impressive FIRST substack post on a topic so entangled with long term thinking that is so central to the question of whether to post or not in the first place <3
Relational stuff first: I had lost track of Tim Tyler, and it is neat to see that he is still writing!
Another citation on this topic might be from Abram Demski in 2020.
Abram's focus is aimed at the way that so many seemingly objectively good things (reason, markets, science, etc) have a backstory of visibly arising in the context of "horizontal transmission" and yet also quite a few of them are like... uh... good? If an innovation is a parasite, then shouldn't the results be... like... bad or icky or something?
There's one coherent temptation(?) to argue that the horizontally transmitted thing has already hacked the as-implemented utility function of whoever is calling various things "good" to make itself seem good, so that they THINK it deserves to be transmitted. This hypothetical "motivated damage" to an agent's goodness detector could "explain away" the detector's (corrupted?) outputs.
There's another temptation(?) then (arising from second order doubts) to look at the idea transmission or acceptance system itself, and try to examine whether or how it filters things, and judge the transmission system's operation by reference to deeper principles that might be less amenable to being hacked. (Such re-examination could be part of the life cycle of one of these things however, so maybe only do it on 1/3rd of the decision/transmission apparatus at a time, kinda like how the US Senate only re-elects 1/3rd of the Senators in any given election?)
If numerous objective and unhacked judgement systems agree on an object level question, that feels safer somehow. Points for being simple. Points for empirical grounding. Points for self-consistency. Points for being old. Points for long term viability in older copies. Etc.
(Also, looking at lots of biological examples from the history of nucleic acid innovations, it seems like many clever tricks in complex healthy systems can be traced back to the vertical capture of a horizontal innovation, so it could be that there's a deeper idea where one should "watch the carnage", with filters, from a distance, then think carefully and double check things, but then eventually cautiously copy the probably-good parts of the show afterwards? This is just a hunch of mine, not something I've kicked the tires on very hard.)
The Credit Assignment Problem has never been clearly solved and tied up with a bow and a label on the tin (though it has been a named challenge in AI research since at least the 1980s).
Maybe I'm just ignorance of some key result somewhere?
Even lacking a fully general result, assuming one has "some decent ideas" about how credit assignment "should" work, then one one can check to see if existing credit accounting frameworks exist for various idea transmission systems such that their operation "in the loop" could account for some goodness of the ideas they are generating and sifting?
Like... Is there a utility function? Is its provenance hidden or not? How hard is it to find "the skeletons in the closet" in the history of the life and results of people holding tightly to this or similar “old ideas”? Has the idea re-skinned itself? If it wears new skin, is this for camouflage (to avoid blame for past mistakes), or did the old thing become corrupted somehow and a reboot was necessary? Are there active systems for hiding or censoring the history? And so on.
Applying this filter to itself as a check... one interesting thing here is that the idea of the credit assignment problem is *itself* hard to trace. I still have never managed to read Holland's original Bucket Brigade paper, which everyone seems to agree is causally important, but rather have only read summaries, like this:
I can go much further back, but not with clear tracery of a coherent idea... it requires a tolerance for "historical spice" and you kind of have to squint, but then the earliest causally central citation I can find of an attempt to apply credit assignment to ideas sort of systematically is... The Sermon On The Mount's metaphor regarding trees with good and bad fruit?
Matthew 7:16-20 (King James Version)
16 Ye shall know them by their fruits. Do men gather grapes of thorns, or figs of thistles?
17 Even so every good tree bringeth forth good fruit; but a corrupt tree bringeth forth evil fruit.
18 A good tree cannot bring forth evil fruit, neither can a corrupt tree bring forth good fruit.
19 Every tree that bringeth not forth good fruit is hewn down, and cast into the fire.
20 Wherefore by their fruits ye shall know them.
(However, an intensely literal reading might partly explain why the Holy Land turned into a freakin' desert? Come on! What about soil management? What about soil management as a metaphor? For such reasons my current tendency is to be more "live and let live" when other people want to engage in dangerous self-experimentation. Also, I often let weeds grow in between the rows of my little gardens so long as the weeds behave themselves and don't take over. The line I try to draw is when something repeatedly insists on experimenting on MY stuff without MY permission.)
There's a lot more to say potentially, but... my inclinations are to try to minimize the saying of things where Google can easily see until I've thought with at least some care about the possible results of such publishing <3
This is an impressive FIRST substack post on a topic so entangled with long term thinking that is so central to the question of whether to post or not in the first place <3
Relational stuff first: I had lost track of Tim Tyler, and it is neat to see that he is still writing!
Another citation on this topic might be from Abram Demski in 2020.
https://www.lesswrong.com/posts/H3wdw2cLNLpcF8pXA/social-capital-paradoxes
Abram's focus is aimed at the way that so many seemingly objectively good things (reason, markets, science, etc) have a backstory of visibly arising in the context of "horizontal transmission" and yet also quite a few of them are like... uh... good? If an innovation is a parasite, then shouldn't the results be... like... bad or icky or something?
There's one coherent temptation(?) to argue that the horizontally transmitted thing has already hacked the as-implemented utility function of whoever is calling various things "good" to make itself seem good, so that they THINK it deserves to be transmitted. This hypothetical "motivated damage" to an agent's goodness detector could "explain away" the detector's (corrupted?) outputs.
There's another temptation(?) then (arising from second order doubts) to look at the idea transmission or acceptance system itself, and try to examine whether or how it filters things, and judge the transmission system's operation by reference to deeper principles that might be less amenable to being hacked. (Such re-examination could be part of the life cycle of one of these things however, so maybe only do it on 1/3rd of the decision/transmission apparatus at a time, kinda like how the US Senate only re-elects 1/3rd of the Senators in any given election?)
If numerous objective and unhacked judgement systems agree on an object level question, that feels safer somehow. Points for being simple. Points for empirical grounding. Points for self-consistency. Points for being old. Points for long term viability in older copies. Etc.
(Also, looking at lots of biological examples from the history of nucleic acid innovations, it seems like many clever tricks in complex healthy systems can be traced back to the vertical capture of a horizontal innovation, so it could be that there's a deeper idea where one should "watch the carnage", with filters, from a distance, then think carefully and double check things, but then eventually cautiously copy the probably-good parts of the show afterwards? This is just a hunch of mine, not something I've kicked the tires on very hard.)
The Credit Assignment Problem has never been clearly solved and tied up with a bow and a label on the tin (though it has been a named challenge in AI research since at least the 1980s).
Maybe I'm just ignorance of some key result somewhere?
Even lacking a fully general result, assuming one has "some decent ideas" about how credit assignment "should" work, then one one can check to see if existing credit accounting frameworks exist for various idea transmission systems such that their operation "in the loop" could account for some goodness of the ideas they are generating and sifting?
Like... Is there a utility function? Is its provenance hidden or not? How hard is it to find "the skeletons in the closet" in the history of the life and results of people holding tightly to this or similar “old ideas”? Has the idea re-skinned itself? If it wears new skin, is this for camouflage (to avoid blame for past mistakes), or did the old thing become corrupted somehow and a reboot was necessary? Are there active systems for hiding or censoring the history? And so on.
Applying this filter to itself as a check... one interesting thing here is that the idea of the credit assignment problem is *itself* hard to trace. I still have never managed to read Holland's original Bucket Brigade paper, which everyone seems to agree is causally important, but rather have only read summaries, like this:
https://people.idsia.ch/~juergen/bucketbrigade/node2.html
I can go much further back, but not with clear tracery of a coherent idea... it requires a tolerance for "historical spice" and you kind of have to squint, but then the earliest causally central citation I can find of an attempt to apply credit assignment to ideas sort of systematically is... The Sermon On The Mount's metaphor regarding trees with good and bad fruit?
Matthew 7:16-20 (King James Version)
16 Ye shall know them by their fruits. Do men gather grapes of thorns, or figs of thistles?
17 Even so every good tree bringeth forth good fruit; but a corrupt tree bringeth forth evil fruit.
18 A good tree cannot bring forth evil fruit, neither can a corrupt tree bring forth good fruit.
19 Every tree that bringeth not forth good fruit is hewn down, and cast into the fire.
20 Wherefore by their fruits ye shall know them.
(However, an intensely literal reading might partly explain why the Holy Land turned into a freakin' desert? Come on! What about soil management? What about soil management as a metaphor? For such reasons my current tendency is to be more "live and let live" when other people want to engage in dangerous self-experimentation. Also, I often let weeds grow in between the rows of my little gardens so long as the weeds behave themselves and don't take over. The line I try to draw is when something repeatedly insists on experimenting on MY stuff without MY permission.)
There's a lot more to say potentially, but... my inclinations are to try to minimize the saying of things where Google can easily see until I've thought with at least some care about the possible results of such publishing <3