The problems start when we start using compiled Python extensions. In either case, pure Python will Just Work, because it’s interpreted at runtime: there’s no CPU-specific machine code, it’s just text that the Python interpreter knows how to run. On ARM64 machines like my Mac Mini, python:3.9-slim is the ARM64 image, which has a Python executable compiled for ARM64.On Intel or AMD CPUs, python:3.9-slim is the x86_64 aka amd64 image, which has a Python executable compiled for x86_64.If we’re using pure Python code, it’s almost invisible. How does Python usage interact with Docker images and CPU instrunction sets? Note: Docker also supports Windows-oriented images, but that’s out of scope for this article so I won’t mention it again. On a newer Mac using M1/M2/Silicon chips, docker pull will the pull the linux/arm64/v8 image. In short: on an Intel/AMD PC or Mac, docker pull will pull the linux/amd64 image. So whether you’re on a Mac, Linux, or Windows, typically you’ll be running linux Docker images. Since macOS is not Linux, on macOS this is done by running a virtual machine in the background, and then the Docker images run inside the virtual machine. In order to meet its build-once-run-everywhere promise, Docker typically runs on Linux. The relevant ones for our purposes are the linux/amd64 image, used on Intel or AMD CPUs, and the linux/arm64/v8 used on newer Macs.ĭepending what CPU you’re using, Docker will pull the respective image. If we look at the images for the python:3.9-slim tag, we can see there are actually multiple images available, covering multiple different architectures. ![]() Since executable code needs to match the CPU, you need different Docker images for different CPU instruction sets.Įven though on both machines the base image was python:3.9-slim, in practice different images were chosen. How Docker deals with CPU instruction sets (the simple version) Note: AMD64 and ARM64 are visually similar, but different instruction sets, so be careful when reading those names. That means binary code like an executable or Python extension compiled for x86_64 can’t run on a ARM64 CPU, and vice versa. Older Macs use this instruction set as well, but new Macs with M1 or M2 processors use the ARM64 instruction set, aka aarch64 Apple calls these “Apple Silicon” CPUs just to throw in a little meaningless terminology confusion.ĪRM64 and x86_64 instruction sets are different languages a CPU that speaks one can’t understand the other. My Linux computer is running an Intel chip, which uses the x86_64 instruction set, also known as AMD64 since it was first created by AMD.ĬPUs from Intel and AMD uses this instruction set. The CPU instruction set is the language the CPU speaks, the set of binary instructions it can interpret. It’s not the operating system it’s the result of a different CPU instruction set combined with lack of binary wheels. So why do the Linux machine and Mac have such different outcomes? The problem: different hardware and wheel availability Identifying the problem Symptom #1: You need a compilerĮRROR: Could not find a version that satisfies the requirement filprofiler=2022.05.0 (from versions: none)ĮRROR: No matching distribution found for filprofiler=2022.05.0 A takeaway for maintainers of open source Python packages.Solving problem with CPU emulation, some of the downsides of this solution, and future improvements to look forward to.Solving the problem by ensuring the code is installable or compilable.The cause of the problem: a different CPU instruction set.Common symptoms of the problem when using Python. ![]() The symptoms can be non-obvious, though, so in this article we’ll cover: The problem is that the promise of reproducibility relies on certain invariants that don’t apply on newer Macs. What used to work before-on an older Mac, or on a Linux machine-fails in completely unexpected ways. ![]() So it can be a little confusing when you try to build your Python-based Dockerfile on a new Mac, and everything starts failing. ![]() One of the promises of Docker is reproducibility: you can build an image on a different machine, and assuming you’ve done the appropriate setup, get the same result.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |