Conversation
Import web-bugs GitHub issues so that we can collect statistics on them and use them with autowebcompat. Because there are are large number of issues backfill is expected to be slow, so there's support for interruptable import of the initial issues. This is expected to be run before deploying the ETL.
| if backfill_file and repo.repo == "webcompat/web-bugs": | ||
| # Milestone 8 is invalid issues, skip these in the backfill because the | ||
| # web-bugs data files overwrite these with web-bugs-private data | ||
| milestone = 8 |
There was a problem hiding this comment.
So I realized this would also skip issues in invalid milestone that are triaged by Softvision, there are about 20k:
https://github.com/webcompat/web-bugs/issues?q=is%3Aissue%20state%3Aclosed%20milestone%3Ainvalid%20-label%3Abugbug-probability-high
We replace the content of a public issue based on its title on the bugbug side
if item["title"] == "Issue closed.":, so we can do the same to match bugbug? These are the issues that are automatically closed by ml bot.
|
|
||
| # Otherwise try both parsers and see which returns the most data. | ||
| webcompat_parse = self._parse_issue_body_webcompat(number, body) | ||
| webbugs_parse = self._parse_issue_body_webcompat(number, body) |
There was a problem hiding this comment.
| webbugs_parse = self._parse_issue_body_webcompat(number, body) | |
| webbugs_parse = self._parse_issue_body_webbugs(number, body) |
| context.bq_client, | ||
| [WebBugsRepo(gh_client, context.project["web_bugs"]["web_bugs"].table())], | ||
| backfill=context.args.web_bugs_backfill, | ||
| backfill_files=parse_backfill_paths(context.args.web_bugs_backfill_file), |
There was a problem hiding this comment.
This seems to be failing with ERROR:root:'NoneType' object is not iterable when trying to run the default job without supplying backfill as parse_backfill_paths expects type Iterable
| ) | ||
| path_obj = Path(path) | ||
| if not path_obj.exists(): | ||
| ValueError(f"Invalid --web-bugs-backfill-file {item}; path does not exist") |
There was a problem hiding this comment.
should this raise? and the one above
2 participants
Checklist for reviewer:
Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
Scan the PR and verify that no changes (particularly to
.circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logsEnsure the container image will be using permissions granted to telemetry-airflow responsibly.