Python's import system would be my biggest complaint about the language. I've used Python for more than ten years, but there are so many pitfalls with regards to imports that I still have to look things up on a regular basis. What makes things even worse is that packages sometimes even meddle with how imports work. (See e.g. Tensorflow – I recently had to debug the import magic they're doing. It was a nightmare.)
Don't get me wrong, I LOVE Python! It's my go-to language for both small and big tasks. But sometimes I do wish Python had a function like PHP's require()… (Obviously for package-local imports, not for importing other packages.)
Lua uses a "require()" function instead of an import statement for loading modules. I think that this is a crucial difference from a lot of programming languages that is vastly understated. What this means is that you can completely replace "require()" with your own function that has complete control over how the module importing system works. Because of this it is possible to build your own runtime code hotloader entirely in Lua without needing to modify the source of the compiler, and thus Lua is the only "mainstream" programming language listed in the list of notable live coding environments on Wikipedia[0]. (I'm surprised that Clojure isn't listed there, though.)
The only way this can be accomplished is by designing the import system from the start to use a function instead of a statement. It's too late for Python to redo things and change its import system, unfortunately. It really is the one language feature you have to get right the first time or you'll miss out on those kinds of enhancements forever.
Sorry, I think you're right. I think the actual problem I was trying to get at is that Python makes it easy to both import from a module and bind the imported functions/values to local names at the same time. This means the values get bound privately in the module, instead of always being late bound by accessing the module as a dictionary. And because the code in libraries uses this way if importing things, you can't get at the old names if you try to hotload a module.
The docs for Python's importlib say as much:
> If a module imports objects from another module using from … import …, calling reload() for the other module does not redefine the objects imported from it — one way around this is to re-execute the from statement, another is to use import and qualified names (module.name) instead.
But Lua isn't actually different here since you can still bind the members of an imported module to local names manually, it's just that there is no "from" keyword that does this for you. My personal theory is because of the lack of something like "import * from" in Lua, it became more common for library authors to use late binding and access the returned module by indexing into it. However this is just my personal theory.
Also, now that I think about it you still have to make adjustments to your code in order to have hotloading be feasible, even in Lua. I deliberately chose to use late binding in the code I wrote with "qualified names" (really just table indexing). And all the third party dependencies are versioned in the project's repository, and don't have dependencies on other external modules, so there was no need to worry about a dependency breaking hot reloading because it uses imports differently. I think this might be a cultural thing. A lot of Lua libraries are written in a single-file, no-dependency manner which makes them much easier to integrate in the ways I want. Python seems to have a much larger ecosystem with many external dependencies depending on others.
So yes, I think it makes sense that Python could have module reloading, but the problem is that the language encourages the use of local binding from imports which is incompatible with it, and there is already a lot of code that imports things like that, so it's not as practical if you're using libraries from pip or use "import <...> from" anywhere in your codebase.
I think you have problems if you just write local foo = require ("foo") in two different files and you want to replace the module object in both of them at once.
Edit: the below is probably wrong about what you're doing. It sounds like your require-replacement merges the new return value of a module into the old one, so code that accesses stuff through the module's top level table will get the right result.
Wrong stuff: Maybe the hot reloading system you use relies on modules mutating the module object rather than making a new one. This is pretty unusual among Lua modules though. I usually see people unconditionally make a table and return it.
Yes, the table merging is what mine actually does. It clears the old table reference and inserts the contents of the new module into it. It seems to work well enough for my use-case.
Unless something has changed or there's a detail involving builtins I'm not aware of, I don't think this would work, for the same reason as the most common pitfall when mocking functions in python: You have to replace it where it's used, not where it's defined. To do that on a module, you have to first import that module - and then all the imports at the top of it would've already run.
> To do that on a module, you have to first import that module - and then all the imports at the top of it would've already run.
Why not just replace the `__import__` function at program startup, when you haven't imported anything? Monkey patching `__import__` at any other time seems like a bad idea.
I recently wrote a hot reloader for Lua[0]. It's actually a port of something I had done a decade ago in Python! Last time Python hot reloading came up on HN, I asked the author about this approach, and the only problem seemed to be that it was not so great if one was accessing the values one wanted replaced from multiple threads. Lua avoids this problem only by not having a multithreaded interpreter.
`require` is even better than you've indicated: it simply calls the contents of `package.loaders`, in order, on any new string it's given.
So in many instances, there's no need to replace the require builtin: one may simply add a new function to `package.loader`, anywhere in that order that's useful.
I've done this, and draw most (basically all) modules out of a SQLite database. It works like a charm.
If one has need to invalidate the cache for a given module, as comes up in live coding environments, simply set the require string in `package.loaded` to `nil`. Small caveat: the `package.loaded` table is 'special', in that the C which `require` is written in hard-codes a reference to it: one may not simply replace `package.loaded` in the global namespace and expect the new one to be referenced.
> `require` is even better than you've indicated: it simply calls the contents of `package.loaders`, in order, on any new string it's given.
Python’s import goes through the configured finders (sys.meta_path) and invokes them in-order with the name of the package being imported until one of them returns an import spec. Sound ‘bout the same to me?
I’ve written loaders which created “virtual” modules on-demand in the past.
The discussion is about python and the comment you’re replying to specifically (and largely incorrectly) trying to contrast lua’s require with python’s import?
I recently spent an entire day trying to import AllenNLP and its dependencies. I gave up after trying multiple interpreters, virtual environments, and operating systems.
That was probably the first things that attracted me to Docker.
If I started a new job or new project, it was the same story every time. “Do these five things and then you should be able to run the code” after half a day of back and forth I’d have a list of ten things, and the next new person would discover it is actually 12.
Docker doesn’t just change your experience, it forces the author to actually capture most of their prerequisites. To the point where even if you don’t use docker, you benefit from a project having at least tried to use it.
Frankly I was just trying to get a pre-trained ELMO model deployed via Cortex. As best I could tell the problem boiled down to a failure related to installing tensorflow with gpu disabled. I am a NN noob but still, it just shouldn’t be this hard.
Unfortunately that, whilst seeming simple-enough on the surface is one of those things that cuts across a whole number of layers, which makes it quite involved to solve any issues that arise.
Inside the source code files, they then use a decorator (@tf_export) to define where in the package/module hierarchy a function/class should get placed, see e.g.
For instance, the write() function in that module actually gets exposed as `tf.summary.write()` even though it's contained in the file tensorflow/python/ops/summary_ops_v2.py.
Needless to say, this even brings auto-completion of an IDE like IntelliJ to its knees at times.
To make things even worse, things like Tensorboard also make use of global state everywhere, see e.g. the function _should_record_summaries_internal() in the above file. But now the question is: Which module's global state does the function actually access if it gets "moved" to a different module by TF? At the end of the day, I gave up hunting down that bug (_should_record_summaries_internal() always returned False, no matter what) and resorted to mocking the function out whenever I used Tensorboard. To do that (i.e. to find its parent module in the first place), I in turn used the `inspect` package.
I ran into that issue a few weeks ago. I had a small project that was supposed to migrate rows from a MySQL server into a SQL Server instance. For some reason that I was too exhausted to figure out, the program would only work if I imported the mysql connector before the pydobc stuff. That was very frustrating to say the least.
I've seen this complaint for almost every language, and the few languages i don't see this complaint for, people end up complaining about something else. It's too late to change how imports work and it's not especially complicated so no use complaining.
I pretty much never have problems with imports in Ruby. It's not like it's an unsolvable problem. Nothing would ever get better if we just took the "no use complaining" approach.
Well, to be fair, Tensorflow is a Google library - pretty much any Google-made library will have five levels of indirection before it does anything, regardless of language. It's in their DNA.
> I ask because from what I understand from Google's monorepo, it should be easy for engineers to take direct dependencies on whatever they want.
Not correct. The dependency system supports visibility restrictions - you can take direct dependencies to anything only if you're in your own experimental tree. Otherwise package authors can control access.
I worked at Google for several years until 2015 - one of the running inside jokes was that increasing a border by one pixel is a full-quarter project with a PM, a tech lead, and multiple engineers.
It's not like most people would want more complexity (at least I hope not), but somehow it kept happening, and I feel at least some of it is cultural.
At the risk of over generalizing, I think 2 things are going on:
1) They hire a lot of smart but inexperienced people. I've noticed that those kinds of programmers often write clever and overengineered things, which means overly complex.
2) They get promoted for creating complicated things. They don't get promoted for writing simple things.
Well, to be clear if you accidentally downloaded a main.c file with a virus and then accidentally copied someone’s advice to compile an app using “cpp main.c” while being accidentally in your Downloads folder, you will also accidentally end up with a virus on your computer.
This article is too long for what it is trying to say. Although I will go the author a kudos for being able to write so well about nothing - it took me to scroll about a third of the article to realise that what they were saying was blatantly obvious.
PS: the only reason I clicked on it was because I thought python by default would import files with some “magic” names in the current directory.
The attack they're describing is even more subtle than that. The problem is that you could have zip.py in your downloads folder, and if pip itself has `import zip` then it will pick that up in preference to the standard library one.
It will also execute libc.so and libpcre.so from that folder. That's simply how libraries are loaded on both Linux and Windows, nothing specific to python.
It's really far fetched to go for python or jupyter when there are easier options like system DLL. The downloads directory is fundamentally unsafe, that we can agree on. Thankfully browsers don't let website push files unlike what the author may imply.
> It will also execute libc.so and libpcre.so from that folder. That's simply how libraries are loaded on both Linux and Windows
That's true on Windows but NOT on Linux (which I actually find a bit annoying for deploying applications to users). On Linux you could explicitly add "." to your LD_LIBRARY_PATH variable but even then I think it would look in the working directory rather than the executable directory.
> which I actually find a bit annoying for deploying applications to users
Linux ELF binaries can have a library search path embedded using the -rpath linker option. Inside that search path, '$ORIGIN' refers to the binary location so you can have the Windows behavior if you want to.
Amazing, thanks! I already knew about the embedded -rpath but hadn't heard about $ORIGIN before so I always dismissed it as not being much use. According to this StackOverflow answer [1] you need to use the -z origin switch but I only found that thanks to you pointing me in the right direction. I'll be adding this particular gotcha to my deployed executables immediately!
And since XP SP2 windows by default searches current directory as one of the last places that it looks into. It is not exactly perfect mitigation but probably works for most instances of this issue.
I think the main concern here is the directory containing the application (especially if it's in the downloads directory), which is always first in the search order on Windows [1]. There was an article in the Old New Thing blog about how this is because the application's directory is expected to be its own safe package, usually a subdirectory of Program Files.
You're talking about the current working directory, which can be different, and you're right it was moved later in the search order in XP SP2 (also mentioned in [1]). I had no idea it was ever searched at all, so thanks for that!
But maybe it's more analogous to ask about running ~/Downloads/legit_program (or maybe ...\Downloads\legit_installer.exe). As I said in my sibling comment, this is safe on Linux (if legit_program really is safe) but they're right that on Windows it's dangerous because the program will pick up zlib.dll etc. in the downloads directory. Maybe this is part of the reason for .msi installers: they end up running the MSI installer executable, which is located in some system directory instead.
Sure, running executables in random places may be risky. But with python it’s not just the location of the binary program that matters and running scripts or notebooks stored in random places may also be risky.
python -m pip is often advised over pip or /usr/bin/pip, because it uses the version of pip associated with that particular major+minor python install.
But in 99.9999999% of cases it will do what you expect (whereas "pip install x" will be easier to use the wrong one if you have multiple Python versions).
Even with the other form, if pip imports anything named like a pyhon module in the dl dir, it will be imported. That's because the current dir is automatically added to the PATH, using -m only add pip.py to the attack surface.
I do it on Windows using the python executor py. So I'll run something like py -2 or py -3 or even py -3.8 to start the correct version of python when I'm not inside a venv.
Also the same goes for modules. Since I've installed youtube_dl in my global python 3 I'll always run it like py -3 -m youtube_dl and update it using py -3 -m pip install youtube-dl -U. Or to create a new python 2 venv (for whatever reason) py -2 -m virtualenv venv.
for a very long time (I don't use python as a daily driver, so not sure if it's still the case) guidance was to use `python -m pip` and if you tried to run `pip ...` it would explicitlytell you to run `python -m pip`. I regularly install python on clean systems and all of my scripts run `python -m pip`. Should I be changing?
But that's not what they're talking about. The attack they're describing is like if you did `cpp main.c` on a genuinely legitimate file in your downloads directory, but it picked up virus from printf.c (but would silently fall back to the real standard library version if printf.c isn't there).
I disagree, people that download .py files also tend to use pip and REPL. I can easily imagine looking at some example code, download it to view in a text editor without paying too much attention to it and after a long while run 'python -m pip install...' Without paying attention to what directory I am running it from. It's a solid advice, learned something new. Much in the same vain, DLL/.so search order hijacking is a thing too so maybe I will avoid running executables from my Downloads folder as well.
Now I'm imagining some alternate world where compilers are sandboxed well-enough that the OS's search-indexer is willing to point compiler toolchains at source-code-looking files just to find out more about them. That'd be kind of neat.
I think OS X does it by default - it uses the file system to mark files as “recently downloaded from the web” and requires additional modal agreement to run the executable.
That's completely different and done by windows too (and IIRC by default files downloaded via browsers in linux will not have the executable permission).
The parent is talking about having such extremely well done sandboxing so that you can automatically run a compiler on source files (even when downloaded from untrustworthy sources) just to get more info about it.
Python seems to embrace more magic and other subtle visual cues than any other language. I am convinced it is only "beginner friendly" because of how much folks insist it is so.
And php, which is also a really beginner-friendly language. In fact, not needing the big toolchain at the beginning helps experimenting a lot and is probably one big factor of what made both of them take off so much in that niche.
As a long time user of both Python and Ruby, Ruby has plenty of ad hoc magic, at least as much of Python, though it's likely you won't see that as much in whatever language you are less familiar with.
You can go a long way with jq, pup and similar tools.
As an exercise, I used Bash to build scraping and download scripts for about a dozen websites. I found that I only needed to dip into Python when doing complicated operations on strings, or I needed access to data structures.
I actually liked the approach of using Bash first for scraping, because it gave me direct access to Bash's superior IPC and concurrency constructs. I could plug in a Python script into the pipeline and it wouldn't be any different than interacting with another Bash process.
I do most of my straightforward JSON download and processing in bash. Well, in fish.
As everyone else replying to you indicated, I use jq for it. But when I type `curl http://example.com | jq .`, I'm programming in fish.
Granted, when I want to do something more complex and durable, I'm likely to reach for, yes, python. But nearly as often, I build up a moderately complex jq command, and set an alias to it.
Yup. There are a lot of problems with Python, but you're not going to run into them for several years. If you're running into them right away then you're probably not a beginner.
I'm not so sure. A couple times I've shown WTFpython [0] to a co-worker, who after spending a bit of time skimming through it mentions that they finally understand a bug they had when they were learning python. And which entry it is is usually different between them.
Yeah there are some weird quirks relating to scope and mutable default arguments, but overall I think there are still way less WAT issues than with javascript.
Python has a bunch of magic but it's mostly helpful for beginners it’s only when you start to do anything beyond simple scripts that it gets in the way.
It normally doesn't matter that it's magic because most people don't need to interact with it. And it causes a very intuitive result in the part of the language that most people actually do interact with: that x = obj.foo; x() is the same as obj.foo() (not true in Javascript!). But that doesn't change the fact that method binding happens in a more complex way under the hood than you'd expect.
Self is a little beginner-unfriendly if they have exposure to other OO languages. In fact the OO model in Python, aside from inheritance/polymorphism, IMO provides a rather uncompelling story for why you should use it other than just naked data structures, because its implementation is too close to a dict, and it doesn't offer you good compile time/type checks (or if it does, it's not readily apparent how to take advantage of them) which will lead a beginner to lots of AttributeError
Rails pushes a lot of the "magic adjacent" features of Ruby very, very hard, which I'm not sure has entirely been to Ruby's benefit -- it's very often not just the first exposure someone has to Ruby, but the only exposure. It's not as bad as being introduced to PHP by WordPress, to be sure, but it's still going to color your opinion.
I think Ruby's actually a very nice general-purpose scripting language; as much as I like Python in principle, I find myself reaching for Ruby more often.
> It's not as bad as being introduced to PHP by WordPress
i had to learn PHP to do WordPress stuff (after programming in other langs for many years). what's so bad about their usage of PHP?
(off the top of my head: in WP's "everything happens in some hooked callback" model, debugging can be a pain, lots of messing with globals and gluing strings together)
Your "off the top of my head" would be the main ones I would have come up with, maybe not quite as fast as you did. :) WP is very much of its time in terms of PHP good (and bad) practice.
(Also, WP's style guidance is super weird compared to, well, just about any other PHP style that I've ever seen anywhere, or that of any other C-style language, for that matter. I know that's subjective, but it's Just. So. Weird.)
I've used Python for so long that it's hard for me to see its flaws; what (aside from the behavior of import which we're discussing here) seems most worryingly "magical" to you?
Even accepting your argument downthread that the absence of "super().__init__()" here is a bug, this would be a problem that comes from two little magic, not too much! Python requires you to be explicit about this aspect of multiple inheritance, and it sounds to me like you're saying you would rather it be implicit.
No I'm not saying I would rather it be implicit. That can have other adverse repercussions. My goal here was just to identify the problem precisely, because (as you can clearly tell from the thread) it is anything but obvious, even for people who've been using Python for years. I can imagine other ways to mitigate it that don't have dangerous side-effects (like warnings), but there are lots of approaches, and how to solve it is a bit more subjective and warrants a longer discussion.
The pitfall is only part of the problem. The problem is much deeper than that. In Python, the solution lies in the base class, which another author may have written in a totally different time/place. Yet the problem only manifests itself when you, another poor soul, try to derive from it later. This kind of spooky action at a distance pierces abstractions, which is just about the last thing you should want from a programming language... and in a sense it literally violates causality (for the lack of a better word). The poor soul that notices this in multiple-inheritance will have to go very much out of their way to work around it: after spending a while trying to track it down (and understanding what's even going on, which is not easy), they'll have to either monkey-patch the original class at runtime, or introduce superfluous wrappers. That is a very high price to pay, and that's on top of the silent failure that led your code to crash and burn.
But anyway, I was just illustrating Python can be quite complicated (even moreso than C++ in some ways) and has flaws despite its simple and straightforward appearance, and it can catch even experienced developers completely off-guard. Just answering the question you asked, basically.
The original code snippet had no bugs until you added more code, stop arguing in bad faith.
Also, the new snippet has a syntax error, the last version where this was valid reached end of life January 1st, and has been deprecated for 10 years prior to that. Not sure how much stock to put in your python opinions given that context.
> The original code snippet had no bugs until you added more code, stop arguing in bad faith.
Wow, that's a really low blow. I'm arguing in perfectly good faith. Failing to call the base class initializer introduces misbehavior in multiple inheritance and the way this happens in Python is completely unexpected. Not every language is like this.
> Also, the new snippet has a syntax error, the last version where this was valid reached end of life January 1st, and has been deprecated for 10 years prior to that. Not sure how much stock to put in your python opinions given that context.
> Wow, that's a really low blow. I'm arguing in perfectly good faith. Failing to call the base class initializer introduces misbehavior in multiple inheritance and the way this happens in Python is completely unexpected. Not every language is like this
Isn't this a bug in class C, not your original snippet?
I'm also not sure what you mean by "failing to call the base class initializer introduces misbehavior in multiple inheritance", this seems like an issue related to the MRO of the inherited classes.
If you write class C with class B inherited first, the code runs.
The answer is a qualified "yes". Because not only can it no longer participate in multiple inheritance, but it is very prone to misbehaving silently when used in such a manner, even for experienced Python users.
I say qualified because whether or not the bug warrants fixing is another matter. It's more warranted for public-facing APIs than internal code, since it's less practical for downstream users to modify your code. In your internal code, if your team knows about the issue or just avoids multiple inheritance altogether, or if you have some kind of static analysis to check class hierarchies for you, it might be safe to avoid. (Just listing some considerations that I can think of. There might be more.)
My overall point though was just to illustrate one particular example of a flaw that catches even experienced Python developers off-guard, let alone beginners.
That's an excellent question and this opens the door to another another interesting discussion. I don't have the energy for that unfortunately, but to reply to your comment: it's not really important that it succeeds per se. Drastically increasing the likelihood of an error is good enough. What's important is that it mitigates the chances of introducing a bug silently.
There’s no error in the original example; in Python, the ability to participate in inheritance diamonds with arbitrary other classes is a feature which must be explicitly documented and maintained. James Knight wrote a really good piece on this many years ago: https://fuhm.net/super-harmful/
The solution is just to avoid inheritance. It’s full of terrible pitfalls in most languages, but Python more than most.
What's the issue with the original code snippet? This seems to be demonstrating the pitfalls of multiple inheritance which is super clunky in any language.
It's missing a super().__init__() call. This code would've worked fine had that line been there.
You would not encounter this issue in e.g. C++. It's strictly due to Python's idiosyncratic linearization of the inheritance hierarchy, which can make your sibling class (which you have no knowledge of) your superclass. (!!)
"2 calls to Base" is a misleading way to put it. You in fact get exactly 1 call per every instance of Base in the hierarchy (and they act independently). Contrast that with Python where you can accidentally call the same constructor multiple times for the same instance of that class, and Python doesn't care to prevent you. (And notice how others' proposed solutions in another comment did exactly that, and they didn't realize it either.)
> There's virtual base class inheritance but then that's a different problem that's hardly any more straightforward than Python.
This is debatable (the Python behavior is quite unintuitive) but "straightforwardness" is actually besides my point. At least with virtual base classes, the solution (whether complicated or not) is in the same place as the problem: both are in the derived class. With Python, the solution is in the base class, which another author wrote a long time ago. Whereas the problem only occurs when you, another poor soul, tries to derive from it a long time later. This kind of spooky action at a distance makes it impractical to abstract things away... in a sense it literally violates causality (for the lack of a better word)! That's not something to take lightly, to say the least.
That's not at all what Python does. It linearizes the inheritance hierarchy for the purposes of superclass attribute lookup, in an attempt to solve some aspects of the diamond problem, or at least make them solvable. But it doesn't touch the types themselves, only the MRO used by the super builtin. That's why the super builtin type takes both the "current" class and the object instance itself, which was more visible before the whining of people who didn't understand it grew too loud to be borne by the maintainers and the project introduced the `super()` syntactic sugar. Not that I'm bitter about that or anything.
Anyway, your argument seems to be that you should be able to use naively written classes in a multiple inheritance hierarchy. I suppose that's a defensible opinion, but it's also not the case in any language I know of; C++ solves the problem of not calling the "correct" superclass by default by foisting the responsibility of choosing the correct subset onto the derived class author, but doesn't solve the "I called my superclass' method too many times" problem, only the "I have too many (n>1) copies of my superclass' data" problem (the solution to which is also opt-in). Python resolves all three of these, but in so doing requires careful construction of multiple inheritance hierarchies and use of the super builtin that is religious to the point of fanatical. That's a tradeoff made in respect to a feature that, IMO, is inherently sharp.
Aside: "idiosyncratic" is an interesting word to use given that the C3 MRO was originally intended for Dylan.
I know what's going on and my comment was sufficiently accurate to get the point across in 1 sentence. I wasn't exactly intending to get into a lecture on MRO here.
> C++ solves the problem of not calling the "correct" superclass by default by foisting the responsibility of choosing the correct subset onto the derived class author
Which is much better than leaving a ticking time bomb.
> doesn't solve the "I called my superclass' method too many times"
Er, yes it does. Every class instance in a hierarchy is initialized exactly once. Trying to call it twice will produce an error (e.g. "class has already been initialized").
> Python resolves all three of these
It does not. It's perfectly fine letting you call a class's __init__ multiple times. Unlike with C++.
> Aside: "idiosyncratic" is an interesting word to use given that the C3 MRO was originally intended for Dylan.
It is a fine word. "Idiosyncrasy" does not imply there exists only 1 person in the whole world engaging in the given practice:
"Her habit of using “like” in every sentence was just one of her idiosyncrasies."
No I don't (the code is as-is), but in fact if I did, those those would still suffer from basically the same problem even if I did (exercise for the reader). In modern Python you're supposed to use super().__init__() to let it handle this issue with multiple inheritance.
No no, I understand the problem you're talking about. It is a problem. I'm asking about the solution. Why wouldn't you handle it in the child instead of having siblings be superable?
Essentially, why not do it like this:
class A():
def __init__(self):
self.x = 1
class B():
def __init__(self):
self.y = 2
class C(A, B):
def __init__(self):
A.__init__(self)
B.__init__(self)
print(C().y)
Now your class C can't necessarily be used in a multiple inheritance hierarchy with some other class D, and a class E that inherits from C and D, as C hardcodes what its parents are. Python uses the C3 MRO algorithm to traverse all relevant classes in your hierarchy when you religiously use the super builtin.
But; multiple inheritance is a beast in any language due to its complexity and bringing it up in a discussion of "magic" that python exposes beginners to is specious. It's true that C++ doesn't have this problem, because it has no super keyword and you must explicitly delegate everything, but also introduces the problem that you can have multiple instances of a base class and so has added complexity, virtual inheritance, to solve THAT problem. And tons of languages have a problem where omitting a call to "super" is logically required is not statically diagnosable.
> It's true that C++ doesn't have this problem, because it has no super keyword and you must explicitly delegate everything
It's emphatically not "having a super keyword" or not having it. That's a red herring. Visual C++ has __super and yet it doesn't suffer from this: it errors with "ambiguous call to overloaded function" when there are multiple candidates.
There's a lot more I could say about this (frankly the issue I outlined just scratch the surface of a deeper problem in Python), but this is diverting the discussion. Nobody even said C++ is simple or somehow easier to learn than Python in the first place; I was just pointing out an insidious issue in Python, and frankly, it could've been done better without any need to emulate C++'s approach at all. (Exercise for the reader: suggest improvements.)
> It's emphatically not "having a super keyword" or not having it. That's a red herring. Visual C++ has __super and yet it doesn't suffer from this: it errors with "ambiguous call to overloaded function" when there are multiple candidates
What's your point about this nonstandard extension? My point is that C++ doesn't have the problem of classes not written to be part of a multiple inheritance hierarchy delegating somewhere unexpected because, by design, it doesn't attempt to solve the same problems that Python is. If anything that this nonstandard extension doesn't work in multiple inheritance is agreeing with the points I'm making.
> There's a lot more I could say about this (frankly the issue I outlined just scratch the surface of a deeper problem in Python), but this is diverting the discussion. Nobody even said C++ is simple or somehow easier to learn than Python in the first place; I was just pointing out an insidious issue in Python,
You were the one who brought up C++! (EDIT: this was in a cousin comment.) My point is that different languages navigate this thorny landscape in different ways, but the way in which Python does isn't as haphazard as you are suggesting.
> Exercise for the reader: suggest improvements
Perhaps being part of a multiple inheritance hierarchy is sharp and uncommon enough that it should be opt-in. What are yours?
> What's your point about this nonstandard extension?
My point was what I said: you can have your cake and eat it too. You claimed the issue was due to a lack of a super keyword and I showed you it would not occur even if C++ had super.
> You were the one who brought up C++!
I "brought it up" to illustrate it does one particular thing better than Python. Not to claim it does everything better than Python. If you want a simple, beginner-friendly language, avoid C++ like the plague.
> What are yours?
Idk, for starters maybe produce a warning when multiple inheritance occurs and one of the __init__s in the hierarchy doesn't have an obvious call super().__init__.
> My point was what I said: you can have your cake and eat it too. You claimed the issue was due to a lack of a super keyword and I showed you it would not occur even if C++ had super.
I made no such claim. I said (in my opinion) C++ doesn't have the issue you're raising with Python because it has a completely different design that doesn't attempt to solve diamond problems in superclass attribute lookup. Are we not in agreement?
> I "brought it up" to illustrate it does one particular thing better than Python. Not to claim it does everything better than Python. If you want a simple, beginner-friendly language, avoid C++ like the plague.
You can validly think that C++'s choice here is better. I don't think there's a counterargument to personal preference here. My point was that it does not solve all of the problems Python's super builtin is designed to. And that's fine! It avoids the issue in question! I actually don't understand what you are arguing with here. Personally I think the design goals here are just so wildly different that I don't have a preference, but I admit that's kind of a cop-out.
> Idk, for starters maybe produce a warning when multiple inheritance occurs and one of the __init__s in the hierarchy doesn't call super().__init__.
In general I think having more "warning labels" on multiple inheritance is the right idea, if a language is going to persist in offering it at all. I think this would be a good start, especially for a linter.
That, and: I forgot to mention that not using the super builtin can also lead to unintentionally calling a method twice or more for the same base class in a diamond situation (note that with new-style classes, all multiple inheritance has at least one "diamond" class, often object, but people normally do not delegate to object for obvious reasons; you can imagine a base class Z in this example that A and B inherit from instead, if you wish). So, not using the super builtin can result in both delegating to a superclass' implementation too much, and not at all! This is a problem with multiple inheritance that Python actually solves; virtual inheritance doesn't touch it.
Note that `object` being the root of the class hierarchy implies that you must have careful coordination between all classes in a multiple inheritance hierarchy to determine, for a particular method, when subclasses may/must delegate to super, what the arguments to the method are, and even what the argument names are in some cases! You can't just go glomming classes together and expect it to go well unless you know the classes either don't use the same method names or agree very carefully on what the delgatable interfaces are.
I'm not suggesting it's common; it really depends what the beginner is doing. If you're just learning Python because you're trying to use e.g. a web framework like Django (that uses mixin classes all over the place), then chances are you'll put __init__ somewhere and run into this. If you're doing something else, you might never see this in your life. My comment was just responding directly to the parent comment though (pointing out a Python flaw), not trying to suggest every Python developer will run into this.
God, when I was in the Ruby world it felt like other engineers would always say things like "Just type < insert magic line of text > and it'll just do this for you". I hated it. I hated it so much.
I have heard the opposite from my friends who interacted with people saying that the language is intimidating because its "a real programming language".
But, to be fair deployment and package management of python is a mess when you try to scale up.
With Perl we fixed that security risk earlier (3 years ago) and better. The path of the script is never added to the global script path anymore. This fixed all these attacks automatically. If the script was in /tmp, ~/Downloads or just ~/.
We don't need to warn our users to take care, they won't. Most of them don't even read such best practice warnings.
Similar to how Bash usually doesn't have current directory on the default PATH for executables, you must explicitly reference it ./local-executable for similar concerns.
csh (tcsh?) used to be my default shell, and the current directory was first in the path. That was really convenient. At the time, I didn't realise the risks.
I got burnt by an unfortunate combination of ActivePerl registering script execution as the default handler for .pl files, and some random file transfer application's "view file" dispatching to the Windows default file handler once. Wanted to view the source of a remote perl script; it ended up downloading and immediately executing the script locally with no warning. Boom!
The problem then is that more complex programs are harder break down, and people will tend to write their entire program in a single file. What does happen a lot with Perl.
Ah, that's also how you made a lot of CPAN simply not installable, prompting people who had actual jobs and deadlines just to drop in the variable USE_UNSAFE_INC=1 to get the old behavior back, and call it a day.
Hmm.. it seems that part of the problem is that Python interprets "empty entry in PYTHONPATH" as being the same as ".". That just seems wrong. If you want the current directory as one of your PYTHONPATH entries, then you should be required to insert ".".
Perhaps the sharp edge would be reduced if Python skipped empty entries in PYTHONPATH. This might require some people to explicitly add "." in their configurations, but they should have done that anyway, and it wouldn't require code to change. PYTHONPATH is also getting less used anyway, so this might be a relatively unnoticed change by most.
Yes, that is exactly the problem. People treat PYTHONPATH just like PATH, but the "easy scripting tricks" that are fine with PATH become subtle security vulnerabilities when you do the same thing with PYTHONPATH.
E.g., people will often add a directory to PATH by doing:
PATH="$PATH:my_new_dir"
This is practically always fine for PATH, since it's practically always set. If you use exactly the same syntax with PYTHONPATH, you have an instant security vulnerability. Now the first directory searched is the current directory, and if you ever wander into directories with untrusted fils (like "Downloads/" or "somebody_elses_container/") it could become serious fast. This is a sharp edge that was never intended.
I'm going to look into proposing this as a PEP. The change is trivial, but there may be surprising uses. A PEP would mean that it'd get more scrutiny (so the change will be less likely to be reverted later).
-I is hardly useful, because it also removes the user's site directory packages directory (typically ~/.local/lib/python3.X/site-packages/) from sys.path.
So packages installed with, say, "python3 -m pip install --user" wouldn't be importable.
Aside from the Python related stuff one thing that confuses me about this article is the claim than a browser can put files in your Downloads directory without you requesting that. Exactly what browser/HTTP feature allows that ?
I don't think I've never encountered this behavior and any time I've downloaded a file I've had the opportunity to see and change the name it is stored under.
You're right, if I just click on a link to a file that's not a PDF in chrome (in this case it was a .ps) it does immediately download the file, but it at least displays the filename in the downloads bar below the main window.
So as far as I can tell it would be hard for a site to cause a download to occur without the user noticing.
Safari will absolutely let you do it, though it might not be the default anymore (certainly used to be, and you can configure it in Preferences > Websites > Downloads).
Way more problematically, Safari will also open “safe” files (which includes zips and PDFs) by default after having downloaded them.
Reminds me of a Wordpress site that ended up having a bunch of files uploaded to the uploads directory with the extension .php -- and the webserver allowed them to be executed. Unfortunately things like this are often overlooked!
Some Windows programs has similar issue (DLL preloading attack). The vulnerability is common since around 10 years ago but the mitigation is not enabled by default so some apps/installers still vulnerable.
I wonder who and why decided that downloading files from the internets into a fixed location is a good idea? A downloaded file may be a PDF document, installer of a software, a song, a photo, CAD file, anything at all. Makes no sense to save everything into a same location.
Hierarchical file systems were invented for a reason. For the same reason, I want browsers and chat apps to ask me where to save each file I download. Fortunately, most of these apps have an option to enable such behavior.
As a nice side effects, this prevents security issues like the one described in the article.
Great observation about Python scripts in the downloads folder. Here's a way to make this vulnerability even easier to exploit: Place a `sitecustomize.py` file in there. That way, you don't even need the `-m pip`, since that file will be imported automatically on any Python run.
SELinux is made to prevent this kind of issues. Or rather: To encode the safety advice made by the author as a set of rules/constraints and ship it to everyone.
Now the question is: Is SELinux the sawstop the author asks for?
SELinux is bearable on a server, but I don't know if I could live with it on a dev workstation though (to be fair, as a mainly Windows dev, I never made the effort to "learn" SELinux properly).
Given that python 0.9.0 was released in 1991, it's kind of astonishing (in a good way) that issues like these take nearly 30 years to creep into our collective consciousness.
In my view, the focus on low friction and getting new users quickly productive with minimum fuss (shared by many scripting languages) sets the stage for these inevitable discoveries. And I'm not just hand-waving with the vague "these" characterization. Focus on ease-of-use creates roving best-practices factions that wax and wane over the years, with consequences perfectly captured in the famous xkcd "Python Environment" cartoon.[0] The fact that the empty string interpretation in PYTHONPATH hasn't been a bigger deal much earlier raises interesting questions. (Are bad guys not trying as hard as we imagine? Will newbie-friendly languages always end up with these kinds of exploitable surfaces? etc)
This isn't a solution, but I recommend always using /tmp as a Downloads folder (i.e. symlinking ~/Downloads to /tmp under macos so that it is at least purged every time you reboot.
Don't get me wrong, I LOVE Python! It's my go-to language for both small and big tasks. But sometimes I do wish Python had a function like PHP's require()… (Obviously for package-local imports, not for importing other packages.)