Schedule Sortinghat jobs using GrimoireLab core scheduler#122
Merged
Conversation
Contributor
Author
|
Jobs can still be scheduled using the GraphQL API, but there won’t be any worker to process them. We may need a way to disable this in the settings. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
jjmerchante
changed the title
sduenas
requested changes
Jan 23, 2026
Member
sduenas
left a comment
sduenas
left a comment
There was a problem hiding this comment.
Please check my comments. Also, update the commit message. It says the api calls are /api/v1/<str:task_type>/ but they should be `/api/v1/tasks/str:task_type/'.
Comment on lines
+25
to
+36
| from django.db.models import CharField, TextChoices | ||
| from sortinghat.core.importer.backend import find_import_identities_backends | ||
| from sortinghat.core.jobs import ( | ||
| recommend_affiliations, | ||
| recommend_matches, | ||
| recommend_gender, | ||
| affiliate, | ||
| unify, | ||
| genderize, | ||
| import_identities, | ||
| ) | ||
| from sortinghat.core.context import SortingHatContext |
Member
There was a problem hiding this comment.
Suggested change
| from django.db.models import CharField, TextChoices | |
| from sortinghat.core.importer.backend import find_import_identities_backends | |
| from sortinghat.core.jobs import ( | |
| recommend_affiliations, | |
| recommend_matches, | |
| recommend_gender, | |
| affiliate, | |
| unify, | |
| genderize, | |
| import_identities, | |
| ) | |
| from sortinghat.core.context import SortingHatContext | |
| from django.db.models import CharField, TextChoices | |
| from sortinghat.core.context import SortingHatContext | |
| from sortinghat.core.importer.backend import find_import_identities_backends | |
| from sortinghat.core.jobs import ( | |
| recommend_affiliations, | |
| recommend_matches, | |
| recommend_gender, | |
| affiliate, | |
| unify, | |
| genderize, | |
| import_identities, | |
| ) | |
Comment on lines
+191
to
+201
| class SortingHatTask(Task): | ||
| """Base class for SortingHat tasks.""" | ||
|
|
||
| class JobType(TextChoices): | ||
| AFFILIATE = "affiliate", "Affiliate" | ||
| UNIFY = "unify", "Unify" | ||
| GENDERIZE = "genderize", "Genderize" | ||
| RECOMMEND_AFFILIATIONS = "recommend_affiliations", "Recommend affiliations" | ||
| RECOMMEND_MATCHES = "recommend_matches", "Recommend matches" | ||
| RECOMMEND_GENDER = "recommend_gender", "Recommend gender" | ||
| IMPORT_IDENTITIES = "import_identities", "Import identities" |
Member
There was a problem hiding this comment.
Why did you decide to have just one type of task and several types of jobs? Wouldn't be better to have several tasks? I think it's easy to handle parameters if the task types are different. They can inherit from SortingHatTask (or IdentitiesTask) if needed.
| task_args, job_interval, job_max_retries, burst=burst, *args, **kwargs | ||
| ) | ||
| try: | ||
| job_type = kwargs["job_type"] |
Member
There was a problem hiding this comment.
You can pass the type of job directly in create_task as a parameter, so you don't have to get it from kwargs.
Comment on lines
+49
to
+54
| """ | ||
| Serializer mixin for Task models to be used for get and create views of tasks. | ||
|
|
||
| Subclasses should define the `model` in Meta and can extend the `fields` list | ||
| and create_scheduler_task_args() method as needed. | ||
| """ |
Member
There was a problem hiding this comment.
Suggested change
| """ | |
| Serializer mixin for Task models to be used for get and create views of tasks. | |
| Subclasses should define the `model` in Meta and can extend the `fields` list | |
| and create_scheduler_task_args() method as needed. | |
| """ | |
| """Serializer mixin for Task models to be used for get and create views of tasks. | |
| Subclasses should define the `model` in Meta and can extend the `fields` list | |
| and create_scheduler_task_args() method as needed. | |
| """ |
Member
There was a problem hiding this comment.
Do you mind to add some basic documentation to the classes of this file?
| self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST) | ||
| self.assertIn("Unknown task type", str(response.data)) | ||
|
|
||
| @patch("grimoirelab.core.scheduler.api.schedule_task") |
Member
There was a problem hiding this comment.
Is this really needed? The mock object is never used.
| user = get_user_model().objects.create(username="test", is_superuser=True) | ||
| self.client.force_authenticate(user=user) | ||
|
|
||
| @patch("grimoirelab.core.scheduler.api.reschedule_task") |
| user = get_user_model().objects.create(username="test", is_superuser=True) | ||
| self.client.force_authenticate(user=user) | ||
|
|
||
| @patch("grimoirelab.core.scheduler.api.cancel_task") |
jjmerchante
force-pushed
the
sortinghat-jobs
branch
2 times, most recently
from
6841da7 to
03f1143
Compare
If a task fails to create a job for any reason, it will not be executed. Ensure that every task has a related job. Signed-off-by: Jose Javier Merchante <jjmerchante@bitergia.com>
This commit includes SortingHat tasks to GrimoireLab scheduler. This allows to run SortingHat jobs from GrimoireLab, such as affiliate, unify, genderize, import identities, and generate recommendations. Include new generic API endpoints to list, create, retrieve, reschedule, and cancel tasks: Include the following API endpoints: ``` /api/v1/tasks/ /api/v1/tasks/<str:task_type>/ /api/v1/tasks/<str:task_type>/<str:uuid>/ /api/v1/tasks/<str:task_type>/<str:uuid>/reschedule/ /api/v1/tasks/<str:task_type>/<str:uuid>/cancel/ /api/v1/tasks/<str:task_type>/<str:task_id>/jobs/ /api/v1/tasks/<str:task_type>/<str:task_id>/jobs/<str:uuid>/ /api/v1/tasks/<str:task_type>/<str:task_id>/jobs/<str:uuid>/logs ``` Currently, <task_type> could be 'eventizer', 'recommend_affiliations', 'affiliate', 'recommend_matches', 'unify', 'recommend_gender', 'genderize', or 'import_identities'. Signed-off-by: Jose Javier Merchante <jjmerchante@bitergia.com> # Conflicts: # poetry.lock
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR includes SortingHat tasks using GrimoireLab scheduler. This allows to run SortingHat jobs from GrimoireLab, such as affiliate, unify, genderize, import identities, and generate recommendations.
Include new generic API endpoints to list, create, retrieve, reschedule, and cancel tasks:
Include the following API endpoints:
Currently, <task_type> could be 'eventizer', 'recommend_affiliations', 'affiliate', 'recommend_matches', 'unify', 'recommend_gender', 'genderize', or 'import_identities'.
Fixes chaoss/grimoirelab#798