AnsweredAssumed Answered

MapR NFS Backing Docker Registry V2

Question asked by mandoskippy on Aug 28, 2015
Latest reply on Aug 31, 2015 by mandoskippy
I am trying to run a Docker Registry on my network where MapR FS is backing the registry via NFS.  At this time, the registry is running in docker (Chicken... meet egg) on my MapR Node which has MFS self mounted. I am exposing this to the docker container via -v /var/lib/docker:/mapr/mycluster/appdata/registryv2  in my docker command. This "should" all work.  When I am trying to push images though, I get a registry unexpected error 500.  Digging into the logs, I see issues related to the filesystem, like it trying to delete a directory and getting a device/resource busy from the FS... is there something in how the registry is storying files that would make it incompatible with NFS via MapR? Are their conditions that MapR doesn't keep up with in these situations? This is MapR 5.0.0.  I put some of the registry logs below, note the device/resource busy errors. Note I didn't see any thing in MapR logs, although if one were to tell me to to look again at a specific place, I would be amiable.  Is docker registry just moving to fast in it's FS operations?

msg="error canceling upload after error: remove /var/lib/registry/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a/.nfs0000000040857a330000003a: device or resource busy"


time="2015-08-28T12:53:17.845468662Z" level=debug msg="filesystem.Delete(\"/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a\")" environment=development http.request.host="registryv2.marathon.mesos:31713" http.request.id=95945547-007a-4e77-b1fa-c292bb198c7e http.request.method=PUT http.request.remoteaddr="192.168.0.15:53081" http.request.uri="/v2/registryv2/blobs/uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a?_state=2Azc9NW2GPp94HpUbJkOUgrxdUbubGOZlpvgJpkhF3l7Ik5hbWUiOiJyZWdpc3RyeXYyIiwiVVVJRCI6IjUwYjM0ZTRhLWZmZDQtNGQ0ZS1hZTQ1LWMwNjUxMWY5MzExYSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTUtMDgtMjhUMTI6NTM6MTdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-32-generic os/linux arch/amd64" instance.id=3607c667-15ae-4416-ae40-184895f51acb service=registry trace.duration=9.793826ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete" trace.id=4541f3d7-a12a-4f4b-a7c8-a0a21c32b951 trace.line=149 vars.name=registryv2 vars.uuid=50b34e4a-ffd4-4d4e-ae45-c06511f9311a version=v2.1.1-39-gc2b2505
time="2015-08-28T12:53:17.845582985Z" level=error msg="unable to delete layer upload resources \"/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a\": remove /var/lib/registry/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a: directory not empty" environment=development http.request.host="registryv2.marathon.mesos:31713" http.request.id=95945547-007a-4e77-b1fa-c292bb198c7e http.request.method=PUT http.request.remoteaddr="192.168.0.15:53081" http.request.uri="/v2/registryv2/blobs/uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a?_state=2Azc9NW2GPp94HpUbJkOUgrxdUbubGOZlpvgJpkhF3l7Ik5hbWUiOiJyZWdpc3RyeXYyIiwiVVVJRCI6IjUwYjM0ZTRhLWZmZDQtNGQ0ZS1hZTQ1LWMwNjUxMWY5MzExYSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTUtMDgtMjhUMTI6NTM6MTdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-32-generic os/linux arch/amd64" instance.id=3607c667-15ae-4416-ae40-184895f51acb service=registry vars.name=registryv2 vars.uuid=50b34e4a-ffd4-4d4e-ae45-c06511f9311a version=v2.1.1-39-gc2b2505
time="2015-08-28T12:53:17.845672579Z" level=error msg="unknown error completing upload: &os.PathError{Op:\"remove\", Path:\"/var/lib/registry/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a\", Err:0x27}" environment=development http.request.host="registryv2.marathon.mesos:31713" http.request.id=95945547-007a-4e77-b1fa-c292bb198c7e http.request.method=PUT http.request.remoteaddr="192.168.0.15:53081" http.request.uri="/v2/registryv2/blobs/uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a?_state=2Azc9NW2GPp94HpUbJkOUgrxdUbubGOZlpvgJpkhF3l7Ik5hbWUiOiJyZWdpc3RyeXYyIiwiVVVJRCI6IjUwYjM0ZTRhLWZmZDQtNGQ0ZS1hZTQ1LWMwNjUxMWY5MzExYSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTUtMDgtMjhUMTI6NTM6MTdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-32-generic os/linux arch/amd64" instance.id=3607c667-15ae-4416-ae40-184895f51acb service=registry vars.name=registryv2 vars.uuid=50b34e4a-ffd4-4d4e-ae45-c06511f9311a version=v2.1.1-39-gc2b2505
time="2015-08-28T12:53:17.845743391Z" level=debug msg="(*blobWriter).Rollback" environment=development http.request.host="registryv2.marathon.mesos:31713" http.request.id=95945547-007a-4e77-b1fa-c292bb198c7e http.request.method=PUT http.request.remoteaddr="192.168.0.15:53081" http.request.uri="/v2/registryv2/blobs/uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a?_state=2Azc9NW2GPp94HpUbJkOUgrxdUbubGOZlpvgJpkhF3l7Ik5hbWUiOiJyZWdpc3RyeXYyIiwiVVVJRCI6IjUwYjM0ZTRhLWZmZDQtNGQ0ZS1hZTQ1LWMwNjUxMWY5MzExYSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTUtMDgtMjhUMTI6NTM6MTdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-32-generic os/linux arch/amd64" instance.id=3607c667-15ae-4416-ae40-184895f51acb service=registry vars.name=registryv2 vars.uuid=50b34e4a-ffd4-4d4e-ae45-c06511f9311a version=v2.1.1-39-gc2b2505
time="2015-08-28T12:53:17.849555281Z" level=debug msg="filesystem.Delete(\"/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a\")" environment=development http.request.host="registryv2.marathon.mesos:31713" http.request.id=95945547-007a-4e77-b1fa-c292bb198c7e http.request.method=PUT http.request.remoteaddr="192.168.0.15:53081" http.request.uri="/v2/registryv2/blobs/uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a?_state=2Azc9NW2GPp94HpUbJkOUgrxdUbubGOZlpvgJpkhF3l7Ik5hbWUiOiJyZWdpc3RyeXYyIiwiVVVJRCI6IjUwYjM0ZTRhLWZmZDQtNGQ0ZS1hZTQ1LWMwNjUxMWY5MzExYSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTUtMDgtMjhUMTI6NTM6MTdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-32-generic os/linux arch/amd64" instance.id=3607c667-15ae-4416-ae40-184895f51acb service=registry trace.duration=3.701933ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete" trace.id=4daf1bff-6c80-4bd5-9eac-54c05436f4a1 trace.line=149 vars.name=registryv2 vars.uuid=50b34e4a-ffd4-4d4e-ae45-c06511f9311a version=v2.1.1-39-gc2b2505
time="2015-08-28T12:53:17.849668898Z" level=error msg="unable to delete layer upload resources \"/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a\": remove /var/lib/registry/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a/.nfs0000000040857a330000003a: device or resource busy" environment=development http.request.host="registryv2.marathon.mesos:31713" http.request.id=95945547-007a-4e77-b1fa-c292bb198c7e http.request.method=PUT http.request.remoteaddr="192.168.0.15:53081" http.request.uri="/v2/registryv2/blobs/uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a?_state=2Azc9NW2GPp94HpUbJkOUgrxdUbubGOZlpvgJpkhF3l7Ik5hbWUiOiJyZWdpc3RyeXYyIiwiVVVJRCI6IjUwYjM0ZTRhLWZmZDQtNGQ0ZS1hZTQ1LWMwNjUxMWY5MzExYSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTUtMDgtMjhUMTI6NTM6MTdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-32-generic os/linux arch/amd64" instance.id=3607c667-15ae-4416-ae40-184895f51acb service=registry vars.name=registryv2 vars.uuid=50b34e4a-ffd4-4d4e-ae45-c06511f9311a version=v2.1.1-39-gc2b2505
time="2015-08-28T12:53:17.849747341Z" level=error msg="error canceling upload after error: remove /var/lib/registry/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a/.nfs0000000040857a330000003a: device or resource busy" environment=development http.request.host="registryv2.marathon.mesos:31713" http.request.id=95945547-007a-4e77-b1fa-c292bb198c7e http.request.method=PUT http.request.remoteaddr="192.168.0.15:53081" http.request.uri="/v2/registryv2/blobs/uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a?_state=2Azc9NW2GPp94HpUbJkOUgrxdUbubGOZlpvgJpkhF3l7Ik5hbWUiOiJyZWdpc3RyeXYyIiwiVVVJRCI6IjUwYjM0ZTRhLWZmZDQtNGQ0ZS1hZTQ1LWMwNjUxMWY5MzExYSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTUtMDgtMjhUMTI6NTM6MTdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-32-generic os/linux arch/amd64" instance.id=3607c667-15ae-4416-ae40-184895f51acb service=registry vars.name=registryv2 vars.uuid=50b34e4a-ffd4-4d4e-ae45-c06511f9311a version=v2.1.1-39-gc2b2505
time="2015-08-28T12:53:17.849906795Z" level=error msg="response completed with error" environment=development err.code=UNKNOWN err.detail="remove /var/lib/registry/docker/registry/v2/repositories/registryv2/_uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a: directory not empty" err.message="unknown error" http.request.host="registryv2.marathon.mesos:31713" http.request.id=95945547-007a-4e77-b1fa-c292bb198c7e http.request.method=PUT http.request.remoteaddr="192.168.0.15:53081" http.request.uri="/v2/registryv2/blobs/uploads/50b34e4a-ffd4-4d4e-ae45-c06511f9311a?_state=2Azc9NW2GPp94HpUbJkOUgrxdUbubGOZlpvgJpkhF3l7Ik5hbWUiOiJyZWdpc3RyeXYyIiwiVVVJRCI6IjUwYjM0ZTRhLWZmZDQtNGQ0ZS1hZTQ1LWMwNjUxMWY5MzExYSIsIk9mZnNldCI6MzIsIlN0YXJ0ZWRBdCI6IjIwMTUtMDgtMjhUMTI6NTM6MTdaIn0%3D&digest=sha256%3Aa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.8.1 go/go1.4.2 git-commit/d12ea79 kernel/3.13.0-32-generic os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=33.861421ms http.response.status=500 http.response.written=208 instance.id=3607c667-15ae-4416-ae40-184895f51acb service=registry vars.name=registryv2 vars.uuid=50b34e4a-ffd4-4d4e-ae45-c06511f9311a version=v2.1.1-39-gc2b2505




Outcomes