RHOAIENG-18714: chore(deps): keep requirements.txt in sync with Pipfile.lock #875
Conversation
There was a problem hiding this comment.
error: build error: committing container for step {Env:[SOURCE_CODE=cuda/ubi9-python-3.11 container=oci STI_SCRIPTS_URL=image:///usr/libexec/s2i STI_SCRIPTS_PATH=/usr/libexec/s2i APP_ROOT=/opt/app-root HOME=/opt/app-root/src PLATFORM=el9 NODEJS_VER=20 PYTHON_VERSION=3.11 PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/app-root/src/.local/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PYTHONUNBUFFERED=1 PYTHONIOENCODING=UTF-8 LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 CNB_STACK_ID=com.redhat.stacks.ubi9-python-311 CNB_USER_ID=1001 CNB_GROUP_ID=0 PIP_NO_CACHE_DIR=off SUMMARY=Platform for building and running Python 3.11 applications DESCRIPTION=Python 3.11 available as container is a base platform for building and running various Python 3.11 applications and frameworks. Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python's elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas on most platforms. BASH_ENV=/opt/app-root/bin/activate ENV=/opt/app-root/bin/activate PROMPT_COMMAND=. /opt/app-root/bin/activate BUILD_LOGLEVEL=0 OPENSHIFT_BUILD_NAME=cuda-ubi9-python-3.11-amd64 OPENSHIFT_BUILD_NAMESPACE=ci-op-t1277kqv NVARCH=x86_64 NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 NV_CUDA_CUDART_VERSION=12.4.127-1 CUDA_VERSION=12.4.1 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 NVIDIA_VISIBLE_DEVICES=all NVIDIA_DRIVER_CAPABILITIES=compute,utility NV_CUDA_LIB_VERSION=12.4.1-1 NV_NVTX_VERSION=12.4.127-1 NV_LIBNPP_VERSION=12.2.5.30-1 NV_LIBNPP_PACKAGE=libnpp-12-4-12.2.5.30-1 NV_LIBCUBLAS_VERSION=12.4.5.8-1 NV_LIBNCCL_PACKAGE_NAME=libnccl NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1 NV_LIBNCCL_VERSION=2.21.5 NCCL_VERSION=2.21.5 NV_LIBNCCL_PACKAGE=libnccl-2.21.5-1+cuda12.4 XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/local/cuda NV_NVPROF_VERSION=12.4.127-1 NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4-12.4.127-1 NV_CUDA_CUDART_DEV_VERSION=12.4.127-1 NV_NVML_DEV_VERSION=12.4.127-1 NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1 NV_LIBNPP_DEV_VERSION=12.2.5.30-1 NV_LIBNPP_DEV_PACKAGE=libnpp-devel-12-4-12.2.5.30-1 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-devel NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1 NV_LIBNCCL_DEV_PACKAGE=libnccl-devel-2.21.5-1+cuda12.4 NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1 NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4-12.4.1-1 LIBRARY_PATH=/usr/local/cuda/lib64/stubs NV_CUDNN_VERSION=9.1.0.70-1 NV_CUDNN_PACKAGE=libcudnn9-cuda-12-9.1.0.70-1 NV_CUDNN_PACKAGE_DEV=libcudnn9-devel-cuda-12-9.1.0.70-1] Command:label Args:[io.openshift.build.name cuda-ubi9-python-3.11-amd64 io.openshift.build.namespace ci-op-t1277kqv io.openshift.build.commit.author io.openshift.build.commit.date io.openshift.build.commit.id io.openshift.build.commit.message io.openshift.build.commit.ref io.openshift.build.name io.openshift.build.namespace io.openshift.build.source-context-dir io.openshift.build.source-location io.openshift.ci.from.base-ubi9-python-3.11 sha256:f232c94fd86721dabec7125eab470e77b34461aa7e8f93edd3f6654aedfb6de0 vcs-ref vcs-type vcs-url ] Flags:[] Attrs:map[] Message:LABEL "io.openshift.build.name" "cuda-ubi9-python-3.11-amd64" "io.openshift.build.namespace" "ci-op-t1277kqv" "io.openshift.build.commit.author" "" "io.openshift.build.commit.date" "" "io.openshift.build.commit.id" "" "io.openshift.build.commit.message" "" "io.openshift.build.commit.ref" "" "io.openshift.build.name" "" "io.openshift.build.namespace" "" "io.openshift.build.source-context-dir" "" "io.openshift.build.source-location" "" "io.openshift.ci.from.base-ubi9-python-3.11" "sha256:f232c94fd86721dabec7125eab470e77b34461aa7e8f93edd3f6654aedfb6de0" "vcs-ref" "" "vcs-type" "" "vcs-url" "" Heredocs:[] Original:LABEL "io.openshift.build.name"="cuda-ubi9-python-3.11-amd64" "io.openshift.build.namespace"="ci-op-t1277kqv" "io.openshift.build.commit.author"="" "io.openshift.build.commit.date"="" "io.openshift.build.commit.id"="" "io.openshift.build.commit.message"="" "io.openshift.build.commit.ref"="" "io.openshift.build.name"="" "io.openshift.build.namespace"="" "io.openshift.build.source-context-dir"="" "io.openshift.build.source-location"="" "io.openshift.ci.from.base-ubi9-python-3.11"="sha256:f232c94fd86721dabec7125eab470e77b34461aa7e8f93edd3f6654aedfb6de0" "vcs-ref"="" "vcs-type"="" "vcs-url"=""}: copying layers and metadata for container "dfb6e20365c3fbc81dcaabc62c510df1958018021c7862a33e9fa9c18eb91017": writing blob: error storing blob to image destination for cache "containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:mountopt=metacopy=on]temp.builder.openshift.io/ci-op-t1277kqv/cuda-ubi9-python-3.11-amd64:2bb0dd18": storing blob to file "/var/tmp/container_images_storage4267129774/1": write /var/cache/blobs/sha256:2304e935d0e39e01a82eaae63320bc7aaf5233f0e2a776e9ab7a9a19f6d30cec3463870028: no space left on device
|
/lgtm |
| micropipenv --version || pip install micropipenv | ||
| find . -name Pipfile.lock -execdir bash -c ' | ||
| echo "# Generated by /scripts/sync-requirements-txt.sh from Pipfile.lock" > requirements.txt && | ||
| echo >> requirements.txt && |
There was a problem hiding this comment.
There is also this command pipenv requirements --hash > requirements.txt that can generate the requirements.txt with hashes. Did you try it?
There was a problem hiding this comment.
Yes, I started with that, but changed later.
In our Dockerfiles we use micropipenv, so I think its a better choice. Also it really is micro, so it is smaller and faster. And in the requirements.txt it generates, it formats them better, with indentation. And adds hashes by default, pipenv requirements needs extra flag.
There was a problem hiding this comment.
When we're building Dockerfiles now, it prints things like
Collecting torch==2.4.1+rocm6.1 (from -r /tmp/requirements_micropipenv-w76_pwe3.txt (line 2))
Downloading https://download.pytorch.org/whl/rocm6.1/torch-2.4.1%2Brocm6.1-cp311-cp311-linux_x86_64.whl (2662.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 GB 272.0 MB/s eta 0:00:00
because micropipenv takes Pipfile.lock and first generates requirements.txt into a /tmp file and then runs pip to install from it.
done! |
|
/hold @atheo89 we need to do upstream release r.n. so we won't be merging this and do the release first, to get it done and avoid any possible problems that this PR may cause |
jiridanek
force-pushed
the
jd_sync_pipfiles
branch
from
0477adb to
37d2ba9
Compare
|
/unhold |
jiridanek
force-pushed
the
jd_sync_pipfiles
branch
from
37d2ba9 to
544a91d
Compare
jiridanek
added
the
tide/merge-method-rebase
|
/override ci/prow/codeserver-notebook-e2e-tests ci/prow/runtimes-ubi9-e2e-tests ci/prow/notebooks-ubi9-e2e-tests ci/prow/rstudio-notebook-e2e-tests ci/prow/rocm-notebooks-e2e-tests ci/prow/rocm-runtimes-ubi9-e2e-tests ci/prow/images ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror |
|
@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/codeserver-notebook-e2e-tests, ci/prow/images, ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror, ci/prow/notebooks-ubi9-e2e-tests, ci/prow/rocm-notebooks-e2e-tests, ci/prow/rocm-runtimes-ubi9-e2e-tests, ci/prow/rstudio-notebook-e2e-tests, ci/prow/runtimes-ubi9-e2e-tests In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
tried this in snyk and it does not help there, it does not detect the requirements.txt files any better than it detected Pipfile.lock files so far |
|
Maybe stupid question but why we backport this to downstream main and 2024b?? |
|
/lgtm |
|
I was trying to import the pr branches in snyk, to see what would happen. I don't intend to merge them and I will close them. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jiridanek The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
1 similar comment
|
/retest |
|
/override ci/prow/codeserver-notebook-e2e-tests ci/prow/images ci/prow/notebooks-ubi9-e2e-tests ci/prow/rocm-notebooks-e2e-tests ci/prow/rocm-notebooks-e2e-tests ci/prow/rocm-runtimes-ubi9-e2e-tests ci/prow/rstudio-notebook-e2e-tests ci/prow/runtimes-ubi9-e2e-tests |
|
@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/codeserver-notebook-e2e-tests, ci/prow/images, ci/prow/notebooks-ubi9-e2e-tests, ci/prow/rocm-notebooks-e2e-tests, ci/prow/rocm-runtimes-ubi9-e2e-tests, ci/prow/rstudio-notebook-e2e-tests, ci/prow/runtimes-ubi9-e2e-tests In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror |
|
@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror |
|
@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-merge-bot
bot
merged commit 37db918
into
opendatahub-io:main
https://issues.redhat.com/browse/RHOAIENG-18714
Description
Our build tooling depends on requirements.txt files with hashes. Namely, Konflux (https://konflux-ci.dev/), and Cachi2 (https://github.com/containerbuildsystem/cachi2). This way, we will have the requirements.txt files ready.
Even before migrating to Konflux, we can change the use of micropipenv in our Dockerfiles to just feeding the requirements.txt with hashes to pip directly.
How Has This Been Tested?
https://github.com/jiridanek/notebooks/actions/runs/13038088119/job/36373345053
example of failure: https://github.com/jiridanek/notebooks/actions/runs/13038060069/job/36373257460
piplock renewal example: https://github.com/jiridanek/notebooks/actions/runs/13055611878/job/36425787057
Merge criteria: