I think it’s pretty clear that program synthesis good enough to replace programmers requires AGI.
This solely text based approach is simply “easy” to do, and that’s why we see it. I think it’s cool and results are intriguing but the approach is fundamentally weak and IMO breakthroughs are needed to truly solve the problem of program synthesis.
There's a few decades worth of work on program synthesis and it works very well. You don't need AGI.
You need either a) a complete specification of the target program in a formal language (other than the target language) or b) an incomplete specification in the form of positive and negative examples of the inputs and outputs of the target program, and maybe some form of extra inductive bias to direct the search for a correct program [edit: the latter setting is more often known as program induction].
In the last few years the biggest, splashiest result in program synthesis was the work behind FlashFill, from Gulwani et al: one-shot program learning, and that's one shot, from a single example, not with a model pretrained on millions of examples. It works with lots of hand-crafted DSLs that try to capture the most common use-cases, a kind of programming common sense that, e.g. tells the synthesiser that if the input is "Mr. John Smith" and the output is "Mr" then if the input is "Ms Jane Brown" the output should be "Ms". It works really, really well but you didn't hear about it because it's not deep learning and so it's not as overhyped.
Copilot tries to circumvent the need for "programming common sense" by combining the spectacular ability of neural nets to interpolate between their training data with billions of examples of code snippets, in order to overcome their also spectacular inability to extrapolate. Can language models learned with neural nets replace the work of hand-crafting DSLs with the work of collecting and labelling petabytes of data? We'll have to wait and see. There are also many approaches that don't rely on hand-crafted DSLs, and also work really, really well (true one-shot learning of recursive programs without an example of the base case and the synthesis terminates) but those generally only work for uncommon programming languages like Prolog or Haskell, so they're not finding their way to your IDE, or your spreadsheet app, any time soon.
But, no, AGI is not needed for program synthesis. What's really needed I think is more visibility of program synthesis research so programmers like yourself don't think it's such an insurmountable problem that it can only be solved by magickal AGI.
I said program synthesis good enough to replace programmers requires AGI. Program synthesis based off of informal specifications in natural language. Not talking about highly constrained environments with formal specs.
I am not belittling the work going in this space, and I’m sure for highly constrained and narrow use cases a lot can be done even now. But I believe solving the general problem of program synthesis based on informal spec requires AGI. I am hardly the only one who thinks this.
>> I am not belittling the work going in this space, and I’m sure for highly constrained and narrow use cases a lot can be done even now.
No. Program synthesis approaches work very well for a broad array of problems, not for "highly constrained and narrow use cases"- that is a misconception of the kind that results from lack of familiarity with modern program synthesis.
Sumit Gulwani, that I mentioned in my previous comment, is an author. To clarify, I'm not in any way affiliated with him or his collaborators. I'm actually from a rival camp, if you will, but the paper I link to is a very good summary of the state of the art. It should help you if you wish to understand where program synthesis is at.
>> I said program synthesis good enough to replace programmers requires AGI. Program synthesis based off of informal specifications in natural language.
Program synthesis from natural language is hard to make work because it's difficult to translate natural language specifications to specifications that a program synthesiser can use. But that is a limitation of current natural language analysis, specifically natural language understanding, approaches - not a limitation of program syhtesis approaches.
I think you equate formal specifications, or specification by example, with "narrow use cases". There's no connection between the two.
If program synthesis is as far advanced as you say it is, how come I make six figures doing something that you seem to be arguing can be totally automated?
The reality seems to disagree with your statements. Program synthesis is as of right now limited to academic research and highly narrow use cases. If the opposite was true, I’d be out of a job.
I think copilot is probably the first product of its type that might make its way into the hands of users en masse.
Edit:
Btw I was referring to program synthesis based off informal natural language spec. Spec inference is part of the synthesis pipeline, I think it’s not fair to just ignore that problem.
The purpose of program synthesis is not to get programmers out of a job. Rather,
it's a tool to help programmers better do their job. I think it's easy to see
why you're not using it. With few exceptions, advances in research take many
years to percolate down to the industry. And of course the industry is famous
for following trends without real understanding of anything.
Anyway the review I linked to has some examples of real-world applications of
program synthesis. Don't be afraid to read it- it's light on formal notation and
you don't need special skills to understand it. I appreciate that it's a long
document but there's a Table of Contents at the start and you should be able to
skim through in a short time just to get a general idea of the subject.
Anyway I can see you're trying to "wing it" and reason from first principles
about something you know nothing about, in true SWE style. Yet, you don't know
what you don't know, so you start from the wrong assumptions ("fully automated"
etc) and arrive at the wrong conclusions. That's no way to understand anything.
It's certainly not going to give you any good idea about what's going on in an
entire field of research you know nothing about.
Of course you're not obliged to know anything about program synthesis, but in
that case, maybe consider sitting back and listening rather than expressing
strong opinions with absolute conviction that is not supported by your
knowledge? I think that will make a better conversation, and a better internet,
for everyone.
I think you're holding text-based approaches and synthesis based approaches to radically different expectations. Copilot isn't approaching replacing programmers; presumably a programmer is invoking it, deciding what to keep or change, etc, i.e. generating parts of programs under the guidance of a human programmer. Synthesis can work at the level of providing an expression or a helper function, as a useful tool under the guidance of a programmer.
Copilot suggests some code snippets, and not necessarily good ones. To be dismissive of another approach to generate parts of programs because they cannot replace programmers is like saying that belt-drive bikes aren't worth considering over chains because a belt-drive bike isn't a replacement for a Learjet.
To be clear I wasn’t dismissing anything, that was not my intent. I think as a programmer assist text based approaches work and I really like copilot for what it is.
I was merely saying that for the holy grail, program synthesis from informal spec generalised to any domain, the approach will have to be different.
Your links say that program synthesis in general is the holy grail of computer science etc, you said that the holy grail is "program synthesis from informal spec" and by that you meant a natural language specification as per the context of our conversation so far. Are you now trying to subtly shift the goalposts?
If so, please leave them alone. You have no reason to assume that program synthesis from a natural language specification is "the holy grail" of anything. But I'm glad that our conversation at least made you look up a few links, even if only to try and win the internet conversation from what it looks like.
So what did you learn, from what you read about program synthesis? Can you see why your assumptions earlier on, about "narrow use cases" and the like were wrong?
Edit: btw, the Freuder paper you linked is about constraint prorgamming, not program synthesis.
And did you notice that one of your links above is the abstract of the review paper I proposed you read, earlier?
Look, I think the main disagreement here is that you don’t seem to consider spec inference as part of the program synthesis process, whereas I do. Your position may very well be correct from an academic point of view.
From my perspective, spec inference from informal spec is the main thing to solve. Because for formal specs, I’d just be programming in a declarative language to create the formal spec.
Spec by example won’t scale because you can’t provide examples across the entire domain for apps of real world complexity.
Once spec inference is solved, then you are just left with a search problem. I understand that the search space is freakin huge but I’d still say the latter problem is easier to solve than the former.
And I’d guess that the problem of inferring a spec from an informal description is what requires AGI.
I hope this clarifies my POV. I don’t think we disagree, we just have different perspectives.
Thank you for a level-headed and well thought-out response! I'm glad to see that
our communication isn't "ratcheting" to more and more forceful forms.
Now. I think the root of our disagreement was the necessity or not of AGI for
various kinds of program synthesis, which I think we 've probably pared down to
program synthesis from an informal specification, particularly a natural
language one.
I don't agree that AGI is necessary for that. I think that, as many AI tasks,
such a complete and uncompromising solution can be avoided and a more
problem-specific solution found instead.
In fact, we already had almost a full solution to the problem in 1968, with
SHRDLU [1], a program that simulated a robotic hand directed by a human user in
natural language to grasp and rearrange objects. This was in the strict confines
of a "blocks world" but its capability, of interpreting its user's intent and
translating it accurately to actions in its well-delineated domain remains
unsurpassed [2]. Such a natural language interface could well be implemented
for programming in bounded domains, for example to control machinery or run
database queries etc. This capability remains largely unexploited, because the
trends in AI have shifted and everybody is doing something else now. That's a
wider-ranging conversation though. My main point is that a fully-intelligent
machine is not necessary to communicate with a human in order to carry out
useful tasks. A machine that can correctly interpret statements in a subset of
natural language suffices. This is technology we could be using right now, only
nobody has the, let's say, political will to develop it because it's seen as
parochial, despite the fact that its capabilities are beyond the capabilities of
modern systems.
As to the other kind of informal specifications, by examples, or by
demonstration, what you say, that it won't scale etc, is not true. I mentioned
earlier one-shot learning of recursive hypotheses without an example of the base
case [3]. To clarify, the reason why this is an important capability is that
recursive programs can be much more compact than non-recursive ones and still
represent large sets of instances, even infinite sets of instances. In fact, for
some program learning problems, only recursive solutions can be complete
solutions (this is the case for arithmetic and grammar learning for example).
The ability to learn such solutions from a single example I think should
conclusively address concerns about scaling.
To be fair, this is a capability that was only recently achieved by Inductive
Logic Programming (ILP) systems, a form of logic program synthesis (i.e.
synthesis where the target language is a logic programming language, usually
Prolog, but not necessarily). The Gulwani survey I linked mentions this recent
advance in passing only. But ILP systems in general have excellent sample
complexities and can routinely generalise robustly from a handful of examples
(in the single digits), and have been doing this since the 1990's.
The cost of searching a large hypothesis space is, indeed, an issue. There are
ways around it however. Here, I have to tout my on horn and mention that my
doctoral research is exactly about logic program learning without search. I'd go
as far as to say that what's really been keeping program synthesis back is the
ubiquity of search-based approaches, and that program synthesis will be solved
conclusively only when we can construct arbitrary programs without searching. But you
can file that under "original research" (I mean, that's literally a description
of my job). Anyway, no AGI is needed for all of this.
So, I actually understand your earlier skepticism, along the lines of "if all
this is true, why haven't I heard of it?" (well, you said why do you still have
a job but it's more or less the same thing). The answer remains: because it's
not the trendy stuff. Trends drive both industry and research directions. Right
now the trend is for deep learning, so you won't hear about different approaches
to program synthesis, or even program synthesis in general. There's nothing to
do about it. Personally, I just grin and bear it.
Parsing intent in a programing context is easier than others. Also most the code is written to be parsed for a machine anyway. So with ASTs and all other static and maybe even some dynmaic checks it should be possible.
We already some of it with type detection , intellisense etc
It is hard set of problem with no magic solutions like this with years of development time needed. That approach will not happen commercially, only incrementally in the community.
Also the goal doesn't need to be "to replace programmers". As with copilot, the point of a program synthesis tool can be to assist the programmer. The point of the system in the video linked above is partly that interactively using such a system can aid development. My main point is this can be a lot better in combination with approaches from outside the ML community, which may involve much tighter integration to specific languages, as well as some awareness of a goal for the synthesized portion.
To "replace programmers", an organization would need to have a way of specifying to the system a high level program behavior, and to confirm that an output from the system satisfies that high level behavior. I think for specifications of any complexity, producing and checking them would look like programming just of a different sort.
This solely text based approach is simply “easy” to do, and that’s why we see it. I think it’s cool and results are intriguing but the approach is fundamentally weak and IMO breakthroughs are needed to truly solve the problem of program synthesis.