feat: clear errors for evaluator permission denials and add scope changes

CHANGELOG.md

@@ -24,6 +24,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   sent.
 - `Evaluator.is_global` and `Evaluator.scope_projects` report the scope the API returned, after
   `create()`, `get()`, `list()` and `refresh()`.
+- `EvaluatorPermissionError`, a `ForbiddenError` subclass, replaces the generic 403 message when creating,
+  updating, deleting or re-scoping an evaluator is refused, and says what is needed. Create needs the owner or
+  editor role on every listed project. Update and delete need it on one of the evaluator's projects. A scope change
+  needs it on one of the current projects and on each project added or removed. Anything on a global evaluator,
+  including creating one, needs the org admin role. The server response body is kept in the message.
+- `Evaluator.set_scope(project_ids=[...])` and `set_scope(is_global=True)` replace an evaluator's scope through
+  `PUT /scorers/{scorer_id}/scope`. Promoting to global is org admin only.
 
 ### Changed
 
@@ -68,6 +75,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- `Evaluators.delete_evaluator()` and `Evaluator.refresh()` now say "not found, or not shared with your projects"
+  when the evaluator is missing, since the server hides evaluators outside the caller's projects. `Evaluator.get()`
+  still returns `None` in that case. Calls by id (`update()`, `delete()`, `set_scope()`) may raise `NotFoundError`
+  instead; `delete_evaluator(name=...)` and `Evaluator.delete_by_name()` raise `ValueError`.
+- `Evaluator.delete()` now deletes by id. It used to delete every evaluator that shared its name, which could also hit
+  one the caller had no permission to delete.
+- Creating an LLM or code evaluator no longer fails with an `AttributeError` when the scorer request is answered with
+  a validation error; the detail is raised as a `ValueError`, as it already was for decision evaluators.
 - The API client now knows the permission and enum values that current servers return, so reading projects and
   other resources no longer fails on them. For example, `Projects().get(id=...)` raised
   `ValueError: 'configure_rate_limit' is not a valid ControlResourceAction` (the project permission enum

README.md

@@ -627,6 +627,34 @@ run_experiment(
 )
 ```
 
+#### Evaluators
+
+New evaluators are scoped to the current project (decorator context, then `SPLUNK_AO_PROJECT_ID`, then
+`SPLUNK_AO_PROJECT`) by default. Pass `project_ids` to choose other projects, or `is_global=True` for a global
+evaluator. With no current project and no `project_ids`, the evaluator is created global, which only org admins can do.
+Updating or deleting an evaluator needs the owner or editor role on one of its projects (org admin for a global
+evaluator). Changing scope needs the owner or editor role on one of its current projects and on each project added or
+removed; making it global, or changing the scope of a global evaluator, needs the org admin role.
+
+```python
+from splunk_ao import EvaluatorPermissionError, LlmEvaluator
+
+try:
+    evaluator = LlmEvaluator(
+        name="politeness", prompt="Is the answer polite?", project_ids=["<project-id>"]
+    ).create()
+    # Replace the scope: scoped to exactly these projects (is_global=True is org admins only).
+    evaluator.set_scope(project_ids=["<project-id>", "<other-project-id>"])
+except EvaluatorPermissionError as error:
+    print(error)  # says which role is missing and how to fix it
+```
+
+`EvaluatorPermissionError` subclasses `ForbiddenError`. `Evaluator.get(name=...)` returns `None` when the evaluator
+does not exist or is not shared with your projects. The server answers 404 rather than 403 for an evaluator outside
+your projects on some paths. In that case `update`, `delete` and `set_scope` (which go by id) can raise `NotFoundError`,
+while `delete_evaluator(name=...)` and `Evaluator.delete_by_name` (which go by name) raise a `ValueError` saying the
+evaluator was not found, or not shared with your projects.
+
 ### Sessions
 
 Sessions group related traces under the same conversation. To explicitly create

src/splunk_ao/__init__.py

@@ -57,6 +57,7 @@
     AuthenticationError,
     BadRequestError,
     ConflictError,
+    EvaluatorPermissionError,
     ForbiddenError,
     NotFoundError,
     RateLimitError,
@@ -129,6 +130,7 @@
     "DecisionVersionOutcomeUnknownError",
     "Document",
     "Evaluator",
+    "EvaluatorPermissionError",
     "Experiment",
     "ForbiddenError",
     "Integration",

src/splunk_ao/evaluator.py

@@ -20,18 +20,24 @@
 from splunk_ao.config import SplunkAOConfig
 from splunk_ao.configuration import Configuration
 from splunk_ao.evaluators import (
+    _SCOPE_CHANGE_DENIED_HINT,
     DecisionAnswerOptions,
     DecisionVersionOutcomeUnknownError,
     Evaluators,
+    _create_denied_hint,
+    _edit_denied_hint,
     _resolve_create_scope,
+    _translate_forbidden,
     normalize_decision_answer_options,
     resolve_decision_output_type,
     validate_decision_request,
 )
 from splunk_ao.resources.api.data import (
     create_code_scorer_version_scorers_scorer_id_version_code_post,
     create_scorers_post,
+    delete_scorer_scorers_scorer_id_delete,
     get_validate_code_scorer_task_result_scorers_code_validate_task_id_get,
+    set_scorer_scope_scorers_scorer_id_scope_put,
     update_scorers_scorer_id_patch,
     validate_code_scorer_scorers_code_validate_post,
 )
@@ -49,6 +55,7 @@
     ScorerTypes,
     TaskResultStatus,
     UpdateScorerRequest,
+    UpdateScorerScopeRequest,
 )
 from splunk_ao.resources.models.invalid_result import InvalidResult
 from splunk_ao.resources.types import UNSET, File, Unset
@@ -285,7 +292,7 @@ def get(cls, *, id: str | None = None, name: str | None = None) -> Evaluator | N
         Returns
         -------
             Optional[Evaluator]: The metric if found (SplunkAOEvaluator, LlmEvaluator, CodeEvaluator, or DecisionEvaluator),
-                None otherwise.
+                None otherwise. ``None`` can also mean the evaluator exists but isn't shared with the caller's projects.
 
         Raises
         ------
@@ -384,7 +391,9 @@ def delete_by_name(cls, name: str) -> None:
 
         Raises
         ------
-            ValueError: If no metric with the given name exists.
+            ValueError: If no metric with the given name exists, or it isn't shared with your projects.
+            EvaluatorPermissionError: If the API answers 403 (owner or editor role needed on one of the
+                evaluator's projects; org admin for a global evaluator).
 
         Examples
         --------
@@ -514,6 +523,8 @@ def update(self, **kwargs: Any) -> Evaluator:
                 or the metric is in FAILED_SYNC state.
             ValueError: If any unsupported fields are passed.
             APIError: If the API returns a validation error or empty response.
+            EvaluatorPermissionError: If the API answers 403: updating needs the owner or editor role on one of the
+                evaluator's projects (org admin for a global evaluator). State is set to FAILED_SYNC.
             Exception: If the API call fails (state is set to FAILED_SYNC).
 
         Examples
@@ -546,7 +557,8 @@ def update(self, **kwargs: Any) -> Evaluator:
         logger.info(f"Evaluator.update: id='{self.id}' name='{self.name}' - started")
         try:
             config = SplunkAOConfig.get()
-            response = update_scorers_scorer_id_patch.sync(scorer_id=self.id, client=config.api_client, body=body)
+            with _translate_forbidden(_edit_denied_hint("Updating")):
+                response = update_scorers_scorer_id_patch.sync(scorer_id=self.id, client=config.api_client, body=body)
         except Exception as e:
             self._set_state(SyncState.FAILED_SYNC, error=e)
             logger.error(f"Evaluator.update: id='{self.id}' - failed: {e}")
@@ -562,16 +574,106 @@ def update(self, **kwargs: Any) -> Evaluator:
         logger.info(f"Evaluator.update: id='{self.id}' - completed")
         return self
 
+    def set_scope(self, *, project_ids: builtins.list[str] | None = None, is_global: bool = False) -> Evaluator:
+        """
+        Replace this evaluator's scope on the API.
+
+        This is a full replace: the evaluator ends up scoped to exactly ``project_ids``, or global. It never
+        falls back to the current project. On success the instance is updated with the API response and
+        returned in SYNCED state.
+
+        Parameters
+        ----------
+        project_ids : list[str], optional
+            Projects to scope the evaluator to. Required (and non-empty) unless ``is_global=True``.
+        is_global : bool
+            Make the evaluator global. Cannot be combined with ``project_ids``. Org admins only.
+
+        Returns
+        -------
+            Evaluator: This evaluator instance with updated attributes from the API.
+
+        Raises
+        ------
+            ValidationError: If this is a local evaluator, or the arguments are invalid (``is_global=True`` with
+                ``project_ids``, ``project_ids`` missing, empty or a single string).
+            ValueError: If the evaluator ID is not set, the evaluator is deleted, or the evaluator is in FAILED_SYNC state.
+            APIError: If the API returns a validation error or empty response.
+            EvaluatorPermissionError: If the API answers 403: changing an evaluator's scope needs the owner or
+                editor role on one of its current projects and on each project added or removed. Making it global,
+                or changing the scope of a global evaluator, needs the org admin role.
+            NotFoundError: If the API answers 404. The server also answers 404 for an evaluator that exists but
+                isn't shared with your projects, or the server is too old to support evaluator scopes.
+            Exception: If the API call fails (state is set to FAILED_SYNC).
+
+        Examples
+        --------
+            evaluator = Evaluator.get(name="factuality-checker")
+            evaluator.set_scope(project_ids=["<project-id>"])
+            evaluator.set_scope(is_global=True)  # org admins only
+        """
+        if isinstance(self, LocalEvaluator):
+            raise ValidationError("Local evaluators don't exist on the server and can't change scope.")
+        if self.id is None:
+            raise ValueError("Evaluator ID is not set. Cannot change the scope of a local-only evaluator.")
+        if self.sync_state == SyncState.DELETED:
+            raise ValueError("Cannot change the scope of a deleted evaluator.")
+        if self.sync_state == SyncState.FAILED_SYNC:
+            raise ValueError(
+                "Cannot change the scope of an evaluator in FAILED_SYNC state. "
+                "Call refresh() to re-sync from the API, then retry your changes."
+            )
+
+        if is_global:
+            if project_ids:
+                raise ValidationError("An evaluator cannot be both global and scoped: pass is_global=True or project_ids.")
+            body = UpdateScorerScopeRequest(is_global=True)
+        else:
+            # Typed as object so the str guard is not flagged unreachable for a list[str] parameter.
+            requested: object = project_ids
+            if isinstance(requested, str):
+                raise ValidationError("project_ids must be a list of project ids, not a single string.")
+            if not project_ids:
+                raise ValidationError("project_ids must not be empty. Pass is_global=True to make the evaluator global.")
+            body = UpdateScorerScopeRequest(is_global=False, project_ids=list(dict.fromkeys(project_ids)))
+
+        logger.info(f"Evaluator.set_scope: id='{self.id}' name='{self.name}' - started")
+        try:
+            config = SplunkAOConfig.get()
+            with _translate_forbidden(_SCOPE_CHANGE_DENIED_HINT):
+                response = set_scorer_scope_scorers_scorer_id_scope_put.sync(
+                    scorer_id=self.id, client=config.api_client, body=body
+                )
+        except Exception as e:
+            self._set_state(SyncState.FAILED_SYNC, error=e)
+            logger.error(f"Evaluator.set_scope: id='{self.id}' - failed: {e}")
+            raise
+
+        if isinstance(response, HTTPValidationError):
+            raise APIError(f"Evaluator scope validation error: {response.detail}")
+        if response is None:
+            raise APIError(f"Evaluator scope update returned empty response for id '{self.id}'")
+
+        self._populate_from_scorer_response(response)
+        self._set_state(SyncState.SYNCED)
+        logger.info(f"Evaluator.set_scope: id='{self.id}' - completed")
+        return self
+
     def delete(self) -> None:
         """
         Delete this metric.
 
-        Only works for server-side metrics. Local metrics don't need deletion.
+        Only works for server-side metrics. Local metrics don't need deletion. Deletes by id, so a different
+        evaluator that shares this one's name is not touched.
 
         Raises
         ------
             ValidationError: If this is a local metric.
-            ValueError: If the metric is not synced.
+            ValueError: If the metric ID is not set, or the API answers with a validation error or empty response.
+            EvaluatorPermissionError: If the API answers 403: deleting needs the owner or editor role on one of the
+                evaluator's projects (org admin for a global evaluator).
+            NotFoundError: If the API answers 404. The server also answers 404 for an evaluator that exists but
+                isn't shared with your projects.
 
         Examples
         --------
@@ -586,8 +688,13 @@ def delete(self) -> None:
 
         try:
             logger.info(f"Evaluator.delete: id='{self.id}' name='{self.name}' - started")
-            metrics_service = Evaluators()
-            metrics_service.delete_evaluator(name=self.name)
+            config = SplunkAOConfig.get()
+            with _translate_forbidden(_edit_denied_hint("Deleting")):
+                response = delete_scorer_scorers_scorer_id_delete.sync(scorer_id=self.id, client=config.api_client)
+            if isinstance(response, HTTPValidationError):
+                raise ValueError(response.detail)
+            if response is None:
+                raise ValueError("Failed to delete metric.")
             self._set_state(SyncState.DELETED)
             logger.info(f"Evaluator.delete: id='{self.id}' - completed")
         except Exception as e:
@@ -605,7 +712,8 @@ def refresh(self) -> None:
         ------
             ValidationError: If this is a local metric.
             ValueError: If the metric is not synced.
-            Exception: If the API call fails or the metric no longer exists.
+            ValueError: If the metric is not found, or not shared with your projects.
+            Exception: If the API call fails.
 
         Examples
         --------
@@ -625,7 +733,7 @@ def refresh(self) -> None:
             retrieved_scorer = next((s for s in scorers if s.id == self.id), None)
 
             if retrieved_scorer is None:
-                raise ValueError(f"Evaluator with id '{self.id}' no longer exists")
+                raise ValueError(f"Evaluator with id '{self.id}' not found, or not shared with your projects.")
 
             self._populate_from_scorer_response(retrieved_scorer)
             self._set_state(SyncState.SYNCED)
@@ -847,6 +955,8 @@ def create(self) -> LlmEvaluator:
             ValidationError: If configuration is invalid, or the scope is invalid (``is_global=True`` with
                 ``project_ids``, an empty ``project_ids``, or ``is_global=False`` with no project). Scope errors
                 are ``ValidationError``, not ``ValueError``.
+            EvaluatorPermissionError: If the API answers 403: org admins only for a global evaluator, owner or
+                editor role on the projects for a scoped one.
             NotFoundError: If the current project is set but cannot be found.
             Exception: If the API call fails.
 
@@ -1020,6 +1130,9 @@ def create(self) -> DecisionEvaluator:
             ValidationError: If the scope is invalid (``is_global=True`` with ``project_ids``, an empty
                 ``project_ids``, or ``is_global=False`` with no project). Scope errors are ``ValidationError``,
                 not ``ValueError``.
+            EvaluatorPermissionError: If the API answers 403: org admins only for a global evaluator, owner or
+                editor role on the projects for a scoped one. Decision evaluators may also not be enabled for the
+                organization. A 403 on the version request deletes the new scorer before the error is raised.
             NotFoundError: If the current project is set but cannot be found.
             ValueError: If an argument is invalid or the API answers with a validation error.
             SplunkAOAPIError: For HTTP errors. A scorer created before a clear rejection is deleted.
@@ -1361,6 +1474,8 @@ def create(self) -> CodeEvaluator:
             ValidationError: If code is not set, validation fails, or configuration is invalid, or the scope is
                 invalid (``is_global=True`` with ``project_ids``, an empty ``project_ids``, or ``is_global=False``
                 with no project). Scope errors are ``ValidationError``, not ``ValueError``.
+            EvaluatorPermissionError: If the API answers 403: org admins only for a global evaluator, owner or
+                editor role on the projects for a scoped one.
             NotFoundError: If the current project is set but cannot be found.
             Exception: If the API call fails.
 
@@ -1416,8 +1531,11 @@ def create(self) -> CodeEvaluator:
                 project_ids=scope_project_ids,
             )
 
-            scorer_response = create_scorers_post.sync(client=config.api_client, body=scorer_request)
+            with _translate_forbidden(_create_denied_hint(scope_is_global)):
+                scorer_response = create_scorers_post.sync(client=config.api_client, body=scorer_request)
 
+            if isinstance(scorer_response, HTTPValidationError):
+                raise ValueError(scorer_response.detail)
             if scorer_response is None:
                 logger.debug("CodeEvaluator.create: No response from create_scorers_post")
                 raise ValueError("Failed to create code-based metric: No response from API")