[gndrmnn]

This adds the ability to set the system scope for RoleAssignments. Currently only 'all' is a supported value as per OpenStack API. I also added a api validation test to ensure that project domain and system scope are mutually exclusive.

Closes #923

[gndrmnn]

After careful recreation of the e2e tests, I think we are running into a gophercloud bug here...

I verified that the system scope role assignment is successfully created. But the additional check in which k-orc tries to fetch the just created role assignment fails.

This is because we correctly create the ListAssignmentOpts, but gophercloud's ListAssignments creates the URL: https://<keystone_url>/v3/role_assignments?role.id=<role_id>&scope.system=all&user.id=<user_id>.
This URL appears to be invalid. There is no scope.system parameter check [1], which results in OpenStack returning an empty list and the check fails.

Instead, gophercloud should, in the scope.system case, use the system API [2] and the URLs https://<keystone_url>/v3/system/users/<user_id>/roles or https://<keystone_url>/v3/system/groups/<group_id>/roles to list the roles for a specific user or group.

Gophercloud already does this for the ListAssignmentsOnResource case [3]. We need this for the ListAssignments case [4].

This also implies a different issue that we need to address here: There is no API for /system/roles without specifying a user/group and my tests show that there is indeed no API endpoint answering to that call. That means a RoleAssignmentFilter which has only a system field set is not enough to query for role assignments in OpenStack and k-orc.

[1] https://docs.openstack.org/api-ref/identity/v3/index.html#list-role-assignments
[2] https://docs.openstack.org/api-ref/identity/v3/index.html#list-system-role-assignments-for-a-user https://docs.openstack.org/api-ref/identity/v3/index.html#list-system-role-assignments-for-a-group
[3] https://github.com/gophercloud/gophercloud/blob/d7f64b210958f4001f99782568cba8bcc5c6eb8a/openstack/identity/v3/roles/requests.go#L364-L377
[4] https://github.com/gophercloud/gophercloud/blob/d7f64b210958f4001f99782568cba8bcc5c6eb8a/openstack/identity/v3/roles/requests.go#L251-L263

[gndrmnn]

This appears to be a timing issue with system scoped role assignments. I can query the scope.system roles on the role_assignment endpoint using curl. It just happens to be that the OpenStack documentation is missing that parameter.

Anyway, it seems that scope.system roles simply do not immediately become available for whatever reason.

[gndrmnn]

Ok, so I can not curl role_assignments which contain all three user.id, role.id and scope.system. Very curious... Looking at upstream keystone code now

Update: It will become available eventually. It takes the better part of 20 minutes though. I have so far no idea why this happens. Let's see if the test in the CI pipeline is faster

[winiciusallan]

Update: It will become available eventually. It takes the better part of 20 minutes though. I have so far no idea why this happens. Let's see if the test in the CI pipeline is faster

If this is really what's happening, then I'll be very surprised. Our CI uses a single-node deployment, and operations like that should be atomic (I think), so they're written in the DB then in a N+1 operation you could see the results. Could be that some concurrency problem? 🤔

[gndrmnn]

If this is really what's happening, then I'll be very surprised. Our CI uses a single-node deployment, and operations like that should be atomic (I think), so they're written in the DB then in a N+1 operation you could see the results. Could be that some concurrency problem?

This issue is extremely weird and I have spent so much time already debugging it. The issue must be somewhere in keystone, because even when I curl keystone directly I run into this issue.

So basically it boils down to this:

• Create a system scoped role assignment

• ORC will report the role assignment as missing

• Observe that openstack role assignment list does list the system scoped assignment

• Observe that curl <keystone_url>/v3/role_assignments?role.id=<role_id>&scope.system=all&user.id=<user_id> does not list the system scoped assignment

• However, any permutation of the curl URL which does not contain all three parameters at once does list the assignment.

• After quite some time (about 20 minutes - I don't know if that is deterministic) the assignment suddenly shows up everywhere without changing anything

At this point I have already gone through the keystone code superficially but I could not find any place where this would be cached or something. It is an extremely weird problem.

Note that the CI pipeline in this PR failed before I replaced the fatal error for a retry-reconcile status. So whatever goes wrong for me locally, also affects the CI pipeline. Whatever that might be

[mandre]

If this is really what's happening, then I'll be very surprised. Our CI uses a single-node deployment, and operations like that should be atomic (I think), so they're written in the DB then in a N+1 operation you could see the results. Could be that some concurrency problem?

This issue is extremely weird and I have spent so much time already debugging it. The issue must be somewhere in keystone, because even when I curl keystone directly I run into this issue.

So basically it boils down to this:

* Create a system scoped role assignment

* ORC will report the role assignment as missing

* Observe that `openstack role assignment list` **does** list the system scoped assignment

* Observe that `curl <keystone_url>/v3/role_assignments?role.id=<role_id>&scope.system=all&user.id=<user_id>` **does not** list the system scoped assignment

* However, any permutation of the curl URL which does not contain all three parameters at once **does** list the assignment.

* After quite some time (about 20 minutes - I don't know if that is deterministic) the assignment suddenly shows up everywhere without changing anything

At this point I have already gone through the keystone code superficially but I could not find any place where this would be cached or something. It is an extremely weird problem.

Note that the CI pipeline in this PR failed before I replaced the fatal error for a retry-reconcile status. So whatever goes wrong for me locally, also affects the CI pipeline. Whatever that might be

Possibly fixed by https://review.opendev.org/c/openstack/keystone/+/997392 ? Could you check if you get the same behavior with hibiscus?

[gndrmnn]

Possibly fixed by https://review.opendev.org/c/openstack/keystone/+/997392 ? Could you check if you get the same behavior with hibiscus?

That does very much look like a fix for the issue I am experiencing! It will take a bit of time for me to set this up locally with a new OpenStack version.

[gndrmnn]

CI:

"e2e hibiscus - Successful in 29m"

"e2e gazpacho/flamingo - Started 36m ago"

[mandre]

CI:

"e2e hibiscus - Successful in 29m"

"e2e gazpacho/flamingo - Started 36m ago"

The time difference was actually in deploying devstack, for both gazpacho and flamingo it took longer than for hibiscus.

Running the tests took around 11 minutes for all 3 jobs, so we can't really draw a conclusion :-/

[mandre]

I'm happy with the state of this PR and think it's ready to merge. If we find that the possible keystone caching issue makes the CI flacky, we can always disable the impacted tests until we have a fix in the relevant openstack releases.

[gndrmnn]

I'm happy with the state of this PR and think it's ready to merge. If we find that the possible keystone caching issue makes the CI flacky, we can always disable the impacted tests until we have a fix in the relevant openstack releases.

Since replacing the fatal error with a retry, the CI has not failed yet. Worst case, I think, is that they e2e might time out, but it looks like we will not reach that threshold.

Maybe we should add a user facing comment somewhere that explains why system scoped roles might not become immediately available?

[winiciusallan]

Maybe we should add a user facing comment somewhere that explains why system scoped roles might not become immediately available?

I think it's worth mentioning, otherwise the change looks good.