This is an impressive FIRST substack post on a topic so entangled with long term thinking that is so central to the question of whether to post or not in the first place <3
Relational stuff first: I had lost track of Tim Tyler, and it is neat to see that he is still writing!
Another citation on this topic might be from Abram Demski in 2020.
Abram's focus is aimed at the way that so many seemingly objectively good things (reason, markets, science, etc) have a backstory of visibly arising in the context of "horizontal transmission" and yet also quite a few of them are like... uh... good? If an innovation is a parasite, then shouldn't the results be... like... bad or icky or something?
There's one coherent temptation(?) to argue that the horizontally transmitted thing has already hacked the as-implemented utility function of whoever is calling various things "good" to make itself seem good, so that they THINK it deserves to be transmitted. This hypothetical "motivated damage" to an agent's goodness detector could "explain away" the detector's (corrupted?) outputs.
There's another temptation(?) then (arising from second order doubts) to look at the idea transmission or acceptance system itself, and try to examine whether or how it filters things, and judge the transmission system's operation by reference to deeper principles that might be less amenable to being hacked. (Such re-examination could be part of the life cycle of one of these things however, so maybe only do it on 1/3rd of the decision/transmission apparatus at a time, kinda like how the US Senate only re-elects 1/3rd of the Senators in any given election?)
If numerous objective and unhacked judgement systems agree on an object level question, that feels safer somehow. Points for being simple. Points for empirical grounding. Points for self-consistency. Points for being old. Points for long term viability in older copies. Etc.
(Also, looking at lots of biological examples from the history of nucleic acid innovations, it seems like many clever tricks in complex healthy systems can be traced back to the vertical capture of a horizontal innovation, so it could be that there's a deeper idea where one should "watch the carnage", with filters, from a distance, then think carefully and double check things, but then eventually cautiously copy the probably-good parts of the show afterwards? This is just a hunch of mine, not something I've kicked the tires on very hard.)
The Credit Assignment Problem has never been clearly solved and tied up with a bow and a label on the tin (though it has been a named challenge in AI research since at least the 1980s).
Maybe I'm just ignorance of some key result somewhere?
Even lacking a fully general result, assuming one has "some decent ideas" about how credit assignment "should" work, then one one can check to see if existing credit accounting frameworks exist for various idea transmission systems such that their operation "in the loop" could account for some goodness of the ideas they are generating and sifting?
Like... Is there a utility function? Is its provenance hidden or not? How hard is it to find "the skeletons in the closet" in the history of the life and results of people holding tightly to this or similar “old ideas”? Has the idea re-skinned itself? If it wears new skin, is this for camouflage (to avoid blame for past mistakes), or did the old thing become corrupted somehow and a reboot was necessary? Are there active systems for hiding or censoring the history? And so on.
Applying this filter to itself as a check... one interesting thing here is that the idea of the credit assignment problem is *itself* hard to trace. I still have never managed to read Holland's original Bucket Brigade paper, which everyone seems to agree is causally important, but rather have only read summaries, like this:
I can go much further back, but not with clear tracery of a coherent idea... it requires a tolerance for "historical spice" and you kind of have to squint, but then the earliest causally central citation I can find of an attempt to apply credit assignment to ideas sort of systematically is... The Sermon On The Mount's metaphor regarding trees with good and bad fruit?
Matthew 7:16-20 (King James Version)
16 Ye shall know them by their fruits. Do men gather grapes of thorns, or figs of thistles?
17 Even so every good tree bringeth forth good fruit; but a corrupt tree bringeth forth evil fruit.
18 A good tree cannot bring forth evil fruit, neither can a corrupt tree bring forth good fruit.
19 Every tree that bringeth not forth good fruit is hewn down, and cast into the fire.
20 Wherefore by their fruits ye shall know them.
(However, an intensely literal reading might partly explain why the Holy Land turned into a freakin' desert? Come on! What about soil management? What about soil management as a metaphor? For such reasons my current tendency is to be more "live and let live" when other people want to engage in dangerous self-experimentation. Also, I often let weeds grow in between the rows of my little gardens so long as the weeds behave themselves and don't take over. The line I try to draw is when something repeatedly insists on experimenting on MY stuff without MY permission.)
There's a lot more to say potentially, but... my inclinations are to try to minimize the saying of things where Google can easily see until I've thought with at least some care about the possible results of such publishing <3
Interesting! I never really thought about it from an application to epistemology point of view.
I think this is for a few reasons.
For one, I see explanation and prediction as quite different. I understand they are theoretically equivalent in some sense. But practically, I think it should be a Talebian barbell between "useless" explanation on one side and "senseless" anticipation on the other, and avoiding the middle. Explanations should furnish broad structural features of your epistemology, but should say nothing about its detailed dynamics. There is a temptation with the middle to wirehead on applying general principles to concrete situations, losing the nuance of the local context.
Secondly, I don't think we are all that coherent agents anyway, and so there really isn't a utility function to speak of. Ultimately there is no vertical or horizontal transmission in the general process of universal darwinism. They are just certain structural constraints which cause some things to be selected in some contexts. As to the desirability of such things, who knows? The only way to find out is the messy embodied participation in cultural life.
Credit assignment can be an important factor in AI and social epistemology for sure. But in deciding whether any specific idea is true or useful, I don't think it is a good frame. Ultimately the ideas are to be judged in and of themselves. There is a time and place to be meta of course, but that is when we have developed the argument in some current frame to the point where we sense systemic contradictions, and so must adopt a slightly larger frame in order to resolve them.
Ultimately the difference in attitude is that I adopt a barbelled bottom-up and top-down, instead of a "vertically integrated" approach I see as characteristic of the lesswrong mileau. Perhaps explaining why I have drifted away over the years :)
This is an impressive FIRST substack post on a topic so entangled with long term thinking that is so central to the question of whether to post or not in the first place <3
Relational stuff first: I had lost track of Tim Tyler, and it is neat to see that he is still writing!
Another citation on this topic might be from Abram Demski in 2020.
https://www.lesswrong.com/posts/H3wdw2cLNLpcF8pXA/social-capital-paradoxes
Abram's focus is aimed at the way that so many seemingly objectively good things (reason, markets, science, etc) have a backstory of visibly arising in the context of "horizontal transmission" and yet also quite a few of them are like... uh... good? If an innovation is a parasite, then shouldn't the results be... like... bad or icky or something?
There's one coherent temptation(?) to argue that the horizontally transmitted thing has already hacked the as-implemented utility function of whoever is calling various things "good" to make itself seem good, so that they THINK it deserves to be transmitted. This hypothetical "motivated damage" to an agent's goodness detector could "explain away" the detector's (corrupted?) outputs.
There's another temptation(?) then (arising from second order doubts) to look at the idea transmission or acceptance system itself, and try to examine whether or how it filters things, and judge the transmission system's operation by reference to deeper principles that might be less amenable to being hacked. (Such re-examination could be part of the life cycle of one of these things however, so maybe only do it on 1/3rd of the decision/transmission apparatus at a time, kinda like how the US Senate only re-elects 1/3rd of the Senators in any given election?)
If numerous objective and unhacked judgement systems agree on an object level question, that feels safer somehow. Points for being simple. Points for empirical grounding. Points for self-consistency. Points for being old. Points for long term viability in older copies. Etc.
(Also, looking at lots of biological examples from the history of nucleic acid innovations, it seems like many clever tricks in complex healthy systems can be traced back to the vertical capture of a horizontal innovation, so it could be that there's a deeper idea where one should "watch the carnage", with filters, from a distance, then think carefully and double check things, but then eventually cautiously copy the probably-good parts of the show afterwards? This is just a hunch of mine, not something I've kicked the tires on very hard.)
The Credit Assignment Problem has never been clearly solved and tied up with a bow and a label on the tin (though it has been a named challenge in AI research since at least the 1980s).
Maybe I'm just ignorance of some key result somewhere?
Even lacking a fully general result, assuming one has "some decent ideas" about how credit assignment "should" work, then one one can check to see if existing credit accounting frameworks exist for various idea transmission systems such that their operation "in the loop" could account for some goodness of the ideas they are generating and sifting?
Like... Is there a utility function? Is its provenance hidden or not? How hard is it to find "the skeletons in the closet" in the history of the life and results of people holding tightly to this or similar “old ideas”? Has the idea re-skinned itself? If it wears new skin, is this for camouflage (to avoid blame for past mistakes), or did the old thing become corrupted somehow and a reboot was necessary? Are there active systems for hiding or censoring the history? And so on.
Applying this filter to itself as a check... one interesting thing here is that the idea of the credit assignment problem is *itself* hard to trace. I still have never managed to read Holland's original Bucket Brigade paper, which everyone seems to agree is causally important, but rather have only read summaries, like this:
https://people.idsia.ch/~juergen/bucketbrigade/node2.html
I can go much further back, but not with clear tracery of a coherent idea... it requires a tolerance for "historical spice" and you kind of have to squint, but then the earliest causally central citation I can find of an attempt to apply credit assignment to ideas sort of systematically is... The Sermon On The Mount's metaphor regarding trees with good and bad fruit?
Matthew 7:16-20 (King James Version)
16 Ye shall know them by their fruits. Do men gather grapes of thorns, or figs of thistles?
17 Even so every good tree bringeth forth good fruit; but a corrupt tree bringeth forth evil fruit.
18 A good tree cannot bring forth evil fruit, neither can a corrupt tree bring forth good fruit.
19 Every tree that bringeth not forth good fruit is hewn down, and cast into the fire.
20 Wherefore by their fruits ye shall know them.
(However, an intensely literal reading might partly explain why the Holy Land turned into a freakin' desert? Come on! What about soil management? What about soil management as a metaphor? For such reasons my current tendency is to be more "live and let live" when other people want to engage in dangerous self-experimentation. Also, I often let weeds grow in between the rows of my little gardens so long as the weeds behave themselves and don't take over. The line I try to draw is when something repeatedly insists on experimenting on MY stuff without MY permission.)
There's a lot more to say potentially, but... my inclinations are to try to minimize the saying of things where Google can easily see until I've thought with at least some care about the possible results of such publishing <3
Interesting! I never really thought about it from an application to epistemology point of view.
I think this is for a few reasons.
For one, I see explanation and prediction as quite different. I understand they are theoretically equivalent in some sense. But practically, I think it should be a Talebian barbell between "useless" explanation on one side and "senseless" anticipation on the other, and avoiding the middle. Explanations should furnish broad structural features of your epistemology, but should say nothing about its detailed dynamics. There is a temptation with the middle to wirehead on applying general principles to concrete situations, losing the nuance of the local context.
Secondly, I don't think we are all that coherent agents anyway, and so there really isn't a utility function to speak of. Ultimately there is no vertical or horizontal transmission in the general process of universal darwinism. They are just certain structural constraints which cause some things to be selected in some contexts. As to the desirability of such things, who knows? The only way to find out is the messy embodied participation in cultural life.
Credit assignment can be an important factor in AI and social epistemology for sure. But in deciding whether any specific idea is true or useful, I don't think it is a good frame. Ultimately the ideas are to be judged in and of themselves. There is a time and place to be meta of course, but that is when we have developed the argument in some current frame to the point where we sense systemic contradictions, and so must adopt a slightly larger frame in order to resolve them.
Ultimately the difference in attitude is that I adopt a barbelled bottom-up and top-down, instead of a "vertically integrated" approach I see as characteristic of the lesswrong mileau. Perhaps explaining why I have drifted away over the years :)
Thanks for your very thought provoking comment!