Relative paths in compose file not resolved correctly?

[teohhanhui]

Given the current working directory is
/src/api-platform/api-platform

when doing
docker buildx bake -f docker-compose-prod/docker-compose.build.yml

then the relative path ../admin in the build context of the admin service
(https://github.com/api-platform/docker-compose-prod/blob/3306ffa4723c2636677ff025e5d703a89347ddc1/docker-compose.build.yml#L52)

should be resolved to
/src/api-platform/api-platform/admin

(This works correctly with docker-compose build -f docker-compose-prod/docker-compose.build.yml)

Actual result

unable to prepare context: path "../admin" not found

[FernandoMiguel]

that is correct and expected.
the context is different.
from BuildX perspective the root is the context, not the service.

you can easily add an override yml

[teohhanhui]

It contradicts the Compose file documentation: https://docs.docker.com/compose/compose-file/#context

When the value supplied is a relative path, it is interpreted as relative to the location of the Compose file.

You risk breaking compatibility with many Compose files...

[FernandoMiguel]

there is no enforced compatibility with compose... only that BuildX is able to read them
the same way it parses HCL and JSON.

as i said, you can just as easily keep using your existing compose YAML, just add an override.
I've done this in the past.
super easy

[tonistiigi]

@tiborvass I remember we discussed how all of these paths should be resolved a while back. Do you have notes from that discussion?

[tiborvass]

There are valid arguments to the current path logic that bake uses, but I agree with @teohhanhui it is preferable to align ourselves with established docker-compose user expectations, so let's fix this.

For keeping a written record, below is a table with all possible cases and proposed UX summing up conversations with @tonistiigi. Basically, we would detect a ./ prefix and interpret it as relative to working directory.

With context path resolution:

• buildx bake -f filename

filename	context	resulting context path
foo/docker-compose.yml	bar/	./foo/bar
foo/docker-compose.yml	./bar/	./bar
foo/docker-compose.yml	git://remote/repo	/tmp/git/repo
http://remote/docker-compose.yml	bar/	./bar
http://remote/docker-compose.yml	./bar/	./bar
http://remote/docker-compose.yml	git://remote/repo	/tmp/git/repo

(filename cannot be git:// because a file, not a folder has to be specified)

Same with Dockerfile path resolution:

• buildx bake mytarget
  • bakefile: {"mytarget": {"context" : "git://remote/repo", "dockerfile": "foo.Dockerfile"}}
    • Resulting dockerfile path: /tmp/git/repo/foo.Dockerfile
  • bakefile: {"mytarget": {"context" : "git://remote/repo", "dockerfile": "./foo.Dockerfile"}}
    • Resulting dockerfile path: ./foo.Dockerfile
• buildx bake git://remote/bakery mytarget (would require detecting protocol prefix in target)
  • bakefile: {"mytarget": {"context" : "git://remote/repo", "dockerfile": "foo.Dockerfile"}}
    • Resulting dockerfile path: /tmp/git/repo/foo.Dockerfile
  • bakefile: {"mytarget": {"context" : "git://remote/repo", "dockerfile": "./foo.Dockerfile"}}
    • Resulting dockerfile path: /tmp/git/bakery/foo.Dockerfile

/cc @ulyssessouza for harmonizing this suggested solution with docker-compose.

[teohhanhui]

@tiborvass The different behaviour between bar/ and ./bar/ will really bite the users.

. means the current directory, so if we follow:

When the value supplied is a relative path, it is interpreted as relative to the location of the Compose file.

they should not be different.

If the intention is to allow anchoring to the current working directory, perhaps it should be something explicit like ${PWD}, instead of introducing subtle differences that will lead to bad DX.

[andoks]

EDIT: sorry for first deleting my comment, then recreating it. But I started thinking this issue was specifically for the "context" property, but I then changed my mind. My issue still relates to the way relative paths and paths in general are specified/interpreted in docker-compose.

I think I might have hit something that relates to this issue today when trying to mount files as volumes into a container using a docker-compose override file located in a subpath.

This subpath also contains all the files to be mounted into the various containers. In my instance compose complains about ERROR: Named volume "cert.pem:/etc/cert.pem:rw" is used in service "myservice" but no declaration was found in the volumes section. when using the suggested form from @tiborvass 's comment of starting paths that are to be considered relative to the compose-file with no prefix.

My expectation was that paths starting with ./ were to be considered relative to the docker-compose file, while the ${PWD} prefix specifies relative to current working directory.

My versions of docker:

Client:
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.2-0ubuntu1~20.10.1
 Built:             Tue Mar 23 23:21:32 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.2-0ubuntu1~20.10.1
  Built:            Tue Mar 16 19:16:09 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.2-0ubuntu1~20.10.2
  GitCommit:        
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.10.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

[nirradi]

When using more than one docker-compose file, then there is inconsistent behavior in this aspect. The first docker-compose yml determines the working directory for path resolution, so files that work on their own suddenly stop working.

example
path1/file1.yml:

version: "3"
services: 
   servicea: 
      env_file:
        - dev.env
     

path2/file2.yml

version: "3"
  

then this works

docker-compose -f path1/file1.yml up 
docker-compose -f path1/file1.yml -f path2/file2.yml up 

and this doesn't

docker-compose -f path2/file2.yml -f path1/file1.yml up 

even when the working directory is path1