pmxbot IRC Log Viewer

[05:03:53] <dude-x> huh. installed pip 20.1 thought i'd try the new beta resolver to upgrade a few packages. nothing gets upgraded. :/

[18:26:52] <Raito_Bezarius> Hello, anyone knows why https://files.pythonhosted.org/packages/py2.py3/r/requests-oauthlib/requests_oauthlib-1.3.0-py2.py3-none-any.whl is a 404 though on PyPI, it seems like such a file should exist (?)

[18:31:25] <tos9> Raito_Bezarius: You almost certainly aren't really supposed to hotlink to random files on PyPI in that way -- is that what you're trying to do there?

[18:32:21] <ngoldbaum> that file is available at https://files.pythonhosted.org/packages/a3/12/b92740d845ab62ea4edf04d2f4164d82532b5a0b03836d4d4e71c6f3d379/requests_oauthlib-1.3.0-py2.py3-none-any.whl FWIW

[18:32:27] <tos9> Raito_Bezarius: That wheel does appear here: https://pypi.org/project/requests-oauthlib/#files, so if you're .. what ngoldbaum said :)

[18:32:41] <tos9> Basically pip will find it. Whether it has some hotlink-available version I doubt is guaranteed

[18:32:44] <ngoldbaum> i don't think there's a stable URL scheme for wheels

[18:32:48] <tos9> But if it is,... also what ngoldbaum said :D

[18:32:51] <Raito_Bezarius> tos9: I don't have the choice

[18:32:52] <ngoldbaum> there is one for the sdist though

[18:32:55] <tos9> Raito_Bezarius: Why not?

[18:33:08] <Raito_Bezarius> tos9: I'm statically fetching dependencies, that requires predictable URLs

[18:33:14] <tos9> Raito_Bezarius: No it doesn't :)

[18:33:23] <tos9> Raito_Bezarius: `pip` is the way to statically fetch dependencies as well.

[18:33:28] <tos9> And knows how to find your wheel.

[18:33:34] <ngoldbaum> pip is the way to install wheels

[18:33:38] <Raito_Bezarius> tos9: pip is not reproducible and at the moment has no proper dependency solver

[18:33:41] <tos9> `pip download requests-oauthlib==1.3.0` or whatever

[18:33:47] <ngoldbaum> if you can live without the wheels there is a static URL scheme for the sdist

[18:33:49] <Raito_Bezarius> Also, pip has no --offline flag

[18:34:08] <Raito_Bezarius> This is what I meant by statically fetching the dependencies

[18:34:10] <tos9> Raito_Bezarius: What does this have to do with dependency solving? Or reproducibility

[18:34:19] <tos9> Raito_Bezarius: pip supports offline installs.

[18:34:21] <Raito_Bezarius> tos9: I'm using Nix/NixOS to do it

[18:34:24] <ngoldbaum> https://pypi.io/packages/source/o/requests-oauthlib/requests-oauthlib-{{ version }}.tar.gz

[18:34:30] <Raito_Bezarius> And I would like to let Nix do the proper dependency solving

[18:34:33] <tos9> Raito_Bezarius: Sure, NixOS knows how to do all this (and uses pip for part of it I believe)

[18:34:43] <Raito_Bezarius> tos9: poetry2nix performs exactly what I've showed

[18:34:51] <ngoldbaum> if you're packaging it for nix then you should use the sdist i would think

[18:34:51] <Raito_Bezarius> it uses files.pythonhosted.org/<guess the URL>

[18:34:52] <ngoldbaum> not the wheel

[18:35:11] <Raito_Bezarius> sdist is not always available or easily available due to pyproject.toml stuff

[18:35:14] <Raito_Bezarius> take SciPy

[18:35:17] <ngoldbaum> huh?

[18:35:27] <ngoldbaum> scipy's sdist is available?

[18:35:42] <Raito_Bezarius> It is available, but the setup.py has assumptions on build dependencies

[18:35:43] <ngoldbaum> do you mean it's not easily installable, sure, but if you're nix then you have a package for scipy

[18:35:43] <tos9> (there are other things for which the sdist isn't available, you could have said tensorflow)

[18:35:50] <ngoldbaum> and all of scipy's dependencies

[18:35:51] <tos9> But that doesn't change this being correct or supported :)

[18:36:07] <Raito_Bezarius> ngoldbaum: nixpkgs has a certain package for SciPy

[18:36:14] <Raito_Bezarius> I want to generate arbitrary expressions for any type of package

[18:36:23] <ngoldbaum> ok, well if you want to install wheels

[18:36:24] <tos9> Raito_Bezarius: You cannot.

[18:36:30] <ngoldbaum> you should probably use pip to do it

[18:36:50] <Raito_Bezarius> https://github.com/nix-community/poetry2nix/blob/4ebbfdacd1b14a6713c634c2af6e3f880e416fd3/lib.nix#L95

[18:36:55] <Raito_Bezarius> tos9: Why?

[18:37:15] <ngoldbaum> it may be the case that URL scheme worked at one point

[18:37:19] <ngoldbaum> but i don't think it works now...

[18:37:19] <Raito_Bezarius> Is it a "You cannot" due to arbitrary limitations in PyPI/pip or "You cannot" due to "it's a bit complex"?

[18:37:34] <tos9> Raito_Bezarius: Nothing is ever arbitrary about limitations :)

[18:37:35] <ngoldbaum> e.g. pypi moved to the URLs with the hashes for improved security a while ago

[18:37:36] <Raito_Bezarius> ngoldbaum: Is there any reason for the URL scheme to require those xx/yy/zzz, are those computable?

[18:37:48] <ngoldbaum> not without having the file already i think

[18:37:50] <Raito_Bezarius> I'm okay with hashes in the URL, but are those hashes documented?

[18:38:41] <ngoldbaum> there's a pep where all this is documented, i don't have a reference for it off the top of my head

[18:39:01] <Raito_Bezarius> https://www.python.org/dev/peps/pep-0503/ this one?

[18:39:47] <Raito_Bezarius> ah maybe this one https://www.python.org/dev/peps/pep-0458/

[18:40:25] <ngoldbaum> i'm not sure, also keep in mind that there are peps that are in the database but where never implemented or have since been superceded

[18:40:39] <ngoldbaum> unfortunately keeping all this straight approaches full-time job :/

[18:41:12] <Raito_Bezarius> Makes sense

[18:41:35] <Raito_Bezarius> I'm just trying to see how feasible it is to get metadata on PyPI, because that would make Python packaging on Nix so simple

[18:41:57] <Raito_Bezarius> ngoldbaum: there must be some way for pip to find those URLs, right?

[18:42:29] <ngoldbaum> i don't know but also keep in mind that as tos9 told you already they don't want you to reimplement what pip does

[18:42:44] <ngoldbaum> it's probably a lot easier just to use pip for whatever you're trying to do

[18:44:00] <Raito_Bezarius> i'd love to, but pip is just not doing the job alas

[18:44:12] <Raito_Bezarius> maybe in the future, with the new dependency resolving & stuff

[18:44:22] <ngoldbaum> you can tell pip to not install dependencies of a package

[18:44:27] <tos9> Raito_Bezarius: You're making incorrect assumptions

[18:44:37] <Raito_Bezarius> tos9: please, elaborate

[18:44:37] <tos9> Whatever you're doing yourself, youc an use pip to do

[18:44:52] <tos9> Raito_Bezarius: if you don't want pip doing the install, don't use pip to do the install

[18:44:56] <Raito_Bezarius> AFAIK, I cannot ask pip to dump a lockfile right now

[18:45:02] <tos9> If you don't want pip doing the dependency resolution, don't use pip to do the dependency resolution

[18:45:11] <ngoldbaum> what does a lockfile have to do with this?

[18:45:12] <tos9> If all you want is a supported client for PyPI that knows where to find files, pip is that

[18:45:20] <tos9> Everything else, go wild, use what you want

[18:45:23] <Raito_Bezarius> ngoldbaum: I want to pin my dependencies

[18:45:29] <tos9> Raito_Bezarius: pip already supports that

[18:45:33] <tos9> if again, you don't want to use it, don't

[18:45:33] <ngoldbaum> you want to use pip as your widget to convert a package name to a url

[18:45:35] <tos9> still use pip to find the files

[18:45:37] <ngoldbaum> where does a lockfile come in?

[18:45:55] <Raito_Bezarius> Sorry, the discussion just got confusing, I'll summary what I'm trying to do

[18:46:04] <tos9> Raito_Bezarius: That's exactly what I was hoping you'd do!

[18:47:18] <Raito_Bezarius> Given a requirements.txt or pyproject.toml, I'd like to ensure that dependencies are pinned (e.g. I need a lockfile, right?) and I want to be able to install them as a derivation from Nix, I cannot just do `pip install XXX` because it'll use Internet to install dependencies and I don't want to use fixed-output derivation because they're cumbersome for development

[18:47:33] <Raito_Bezarius> Also, I would like to install wheels and not sdist

[18:47:36] <Raito_Bezarius> Because of tensorflow & friends

[18:48:20] <Raito_Bezarius> (let us ignore special cases such as local URLs in requirements.txt, editable and git URLs for example.)

[18:49:26] <Raito_Bezarius> So what I tried is to use poetry2nix which uses the lockfile from Poetry to download the dependencies (and bypass pip I guess), the thing is that as you said, I cannot rely on the URL scheme for security reasons

[18:49:34] <ngoldbaum> isn't a requirements.txt with pinned dependencies exactly what you want?

[18:49:43] <Raito_Bezarius> I'm pretty sure the data in the lockfile from Poetry includes enough information to rebuild the hashes that PyPI uses

[18:49:58] <Raito_Bezarius> ngoldbaum: Well, that was until I encountered dependencies which installed contradictory versions

[18:50:16] <Raito_Bezarius> I didn't investigate really, but I'm okay with requirements.txt if it's enough

[18:50:20] <ngoldbaum> then the requirements.txt didn't have all the packages in the virtualenv

[18:51:10] <Raito_Bezarius> That's possible, I preferred to be on the safe side with the poetry lock

[18:51:12] <ngoldbaum> you want to have a listing of all of your full python dependency tree along with the versions of the packages, pip freeze in a virtualenv will give you that

[18:51:22] <Raito_Bezarius> but pip freeze require to have it already installed

[18:51:26] <Raito_Bezarius> right?

[18:51:33] <ngoldbaum> to have pip installed?

[18:51:35] <ngoldbaum> yeah

[18:51:38] <Raito_Bezarius> to have the dependencies installed

[18:51:43] <ngoldbaum> sure

[18:51:51] <ngoldbaum> you'd be starting from a known good set, right?

[18:52:08] <Raito_Bezarius> Yes

[18:52:24] <Raito_Bezarius> Well, let's admit the bootstrap problem is solved

[18:52:26] <ngoldbaum> so then in that virtualenv where you have the known good set of dependencies installed, you run pip freeze

[18:52:27] <Raito_Bezarius> That's not really hard

[18:52:33] <ngoldbaum> that's the requirements.txt file you want

[18:52:44] <ngoldbaum> the poetry lock file would presumably be the same information?

[18:52:48] <Raito_Bezarius> Indeed

[18:53:03] <Raito_Bezarius> But `pip` does not interface very well under a very hardened sandbox for installation phase

[18:53:23] <ngoldbaum> so wherever you're doing this doesn't have access to the internet?

[18:53:38] <Raito_Bezarius> I can have Internet but I don't want to use Internet

[18:53:42] <Raito_Bezarius> The rationale is extra caching, etc.

[18:53:44] <ngoldbaum> e.g. you can do "pip install package==version --single-version-externally-managed"

[18:53:58] <ngoldbaum> i think that's the name of the option you want

[18:54:13] <ngoldbaum> to tell pip to only install that version and no dependencies, because you are managing the dependencies manually

[18:54:15] <Raito_Bezarius> I don't find it in the man pip-install

[18:54:29] <Raito_Bezarius> (pip 20.0.2)

[18:54:45] <ngoldbaum> it looks like conda-forge does "{{ PYTHON }} -m pip install . --no-deps -vv"

[18:54:54] <ngoldbaum> to install python packages into the conda environment

[18:55:01] <ngoldbaum> so maybe you just want --no-deps

[18:55:24] <ngoldbaum> but i guess in that context it's not fetching data from pypi at all

[18:55:30] <ngoldbaum> so it's not really what you want to do

[18:55:32] <Raito_Bezarius> Indeed

[18:56:00] <Raito_Bezarius> I want to do the download phase, hash check phase, caching phase, myself

[18:56:22] <ngoldbaum> but of wheels, which is a little weird

[18:56:23] <Raito_Bezarius> And for the installation phase, I'd prefer to delegate to pip or whatever which has implemented more of PEPXXX than me regarding "installation mechanisms in Python"

[18:56:34] <Raito_Bezarius> ngoldbaum: Why so?

[18:57:02] <Raito_Bezarius> (thank you for your input/help BTW, it's super helpful, tos9 & ngoldbaum)

[18:57:06] <ngoldbaum> because of exactly the difficulties you're running into

[18:57:12] <ngoldbaum> which are purposely there for security reasons

[18:57:40] <ngoldbaum> there is a stable URL scheme for the sdist

[18:57:50] <ngoldbaum> because downstream packaging schemes want to consume sdists

[18:57:58] <ngoldbaum> it sort of sounds like that's what you're buildinb

[18:58:02] <ngoldbaum> but you want to use wheels

[18:58:16] <ngoldbaum> from pip/pypi perspective, wheels are a binary artifact that are supposed to be consumed by pip

[18:58:17] <Raito_Bezarius> Because I don't want to recompile everything each time I install a package for example

[18:58:27] <Raito_Bezarius> wheels are exclusively made for pip?

[18:58:50] <ngoldbaum> it's a bit like trying to use an rpm in a package manager that you're maintaining yourself on a redhat system

[18:59:22] <Raito_Bezarius> I don't see how it's bad

[18:59:26] <Raito_Bezarius> I'm not trying to fully reuse pip

[18:59:32] <Raito_Bezarius> But I'm trying to get machine-reusable output from pip

[18:59:36] <Raito_Bezarius> To build automation on the top of pip

[18:59:42] <Raito_Bezarius> But I can understand that pip is not made for that

[18:59:52] <Raito_Bezarius> But surely, there must be lower levels tooling or documentation

[18:59:53] <ngoldbaum> there's likely an option you can use

[19:00:26] <Raito_Bezarius> I read multiple times the man but I don't see such an option

[19:01:06] <ngoldbaum> ok, well, you've already explained it and i still don't really understand what you're doing, and i'm no expert myself

[19:01:14] <ngoldbaum> so good luck

[19:01:17] <Raito_Bezarius> Thanks :)

[20:45:26] <Raito_Bezarius> FWIW, I found how those hashes are generated

[20:45:32] <Raito_Bezarius> https://github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L1340

[20:45:57] <Raito_Bezarius> The only question I have left, which I asked at #pypa-dev, is: why using blake2_256 when sha256 is used everywhere else (and only available to Simple API)

[20:46:20] <Raito_Bezarius> "curl https://pypi.org/pypi/requests-oauthlib/json" don't expose any blake2_256 hash

[21:18:56] <energizer> does pypi have any data about how often packages are installed using Poetry?

[21:31:21] <dstufft> energizer: does it have a unique user agent?

[21:38:21] <energizer> dstufft: i think no, it only sets the user agent when uploading. for downloading i think it wraps pip. does pip have a way to set the user agent?

[21:39:19] <energizer> oh i see, yes. PIP_USER_AGENT_USER_DATA

[21:40:09] <energizer> as of last week! :)

[21:40:16] <energizer> https://github.com/pypa/pip/pull/5550

[21:52:38] <dstufft> such a silly feature lol, but w/e - that doesnt really help if poetry is using pip until they populate that field and pypi is updated to understand it

[21:53:48] <dstufft> in anycase, i dont remembe what uas we support atm, but given a unique ua we can track metric against that

[21:54:31] <dstufft> https://packaging.python.org/guides/analyzing-pypi-package-downloads/

Log file Viewer

Help | Karma | Search:

#pypa logs for Wednesday the 29th of April, 2020