GitLab

On every push and pull request that affects relevant files, check the protocol buffer files for code formatting
inconsistency.

The repository contains multiple Go modules. Previously, the CI workflow that lints and checks the formatting of the
repository's Go code was only checking the primary module in the root of the repo.

This workflow is now expanded to cover all Go code through the use of a dedicated matrix job for each of the modules.

Arbitrary Go module paths can be specified to the tasks by defining the `GO_MODULE_PATH` environment variable. If this
variable is not defined, the default root module path is used as default, preserving the previous task behavior.

[skip changelog] Revert "Add skip changelog prefix to Dependabot workflow update commits (#1381)" (#1389)

This reverts commit 4fbf7b7b.

It turns out there is an undocumented maximum length of 15 characters for this prefix:

https://github.com/arduino/arduino-cli/network/updates

```
Dependabot encountered the following error when parsing your .github/dependabot.yml:

The property '#/updates/0/commit-message/prefix' was not of a maximum string length of 15
Please update the config file to conform with Dependabot's specification.
```
Unfortunately, even though it is possible to reduce the prefix length by 1 with the removal of the trailing space, we
still end up with 16 characters.

The code tried to log the wrong error, which was nil, hiding the real
error.

This originally occurred for example when a version of the core was already
installed manually in the sketchbook, and you would try to install
another version using arduino-cli, though that has since been fixed
(arduino-cli no longer tries to uninstall cores from the sketchbook).

To still reproduce this problem, you can e.g. break the permissions of
an installed core:

    $ arduino-cli core install arduino:avr@1.8.1
    (...)
    $ chmod a-w ~/.arduino15/packages/arduino/hardware/avr/1.8.1
    $ arduino-cli core install arduino:avr@1.8.2
    Tool arduino:avr-gcc@7.3.0-atmel3.6.1-arduino5 already installed
    Tool arduino:avrdude@6.3.0-arduino17 already installed
    Tool arduino:arduinoOTA@1.3.0 already installed
    Downloading packages...
    arduino:avr@1.8.2 already downloaded
    Upgrading arduino:avr@1.8.1 with arduino:avr@1.8.2...
    Error upgrading platform: %!s(<nil>)...
    Error during install: upgrading platform: removing platform files: unlinkat /home/matthijs/.arduino15/packages/arduino/hardware/avr/1.8.1/firmwares: permission denied

With this fix applied, this produces a proper error message:

    $ arduino-cli core install arduino:avr@1.8.2
    Tool arduino:avr-gcc@7.3.0-atmel3.6.1-arduino5 already installed
    Tool arduino:avrdude@6.3.0-arduino17 already installed
    Tool arduino:arduinoOTA@1.3.0 already installed
    Downloading packages...
    arduino:avr@1.8.2 already downloaded
    Upgrading arduino:avr@1.8.1 with arduino:avr@1.8.2...
    Error upgrading platform: removing platform files: unlinkat /home/matthijs/.arduino15/packages/arduino/hardware/avr/1.8.1/firmwares: permission denied...
    Error during install: upgrading platform: removing platform files: unlinkat /home/matthijs/.arduino15/packages/arduino/hardware/avr/1.8.1/firmwares: permission denied

Bumps [pyyaml](https://github.com/yaml/pyyaml) from 5.3 to 5.4.
- [Release notes](https://github.com/yaml/pyyaml/releases)
- [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES)
- [Commits](https://github.com/yaml/pyyaml/compare/5.3...5.4)

---
updated-dependencies:
- dependency-name: pyyaml
  dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

The "gRPC reference" section of the documentation website is automatically generated from the repository's protocol
buffer files. In order to make this generation process easy for developers to run, the necessary commands are defined in
a task.

For some reason, there were two tasks for this purpose, with different names and descriptions, but identical commands.
This makes the taskfile more difficult to understand and more difficult to maintain.

One of the tasks, `protoc:docs` is called by the `docs:generate` task. The other, `docs:gen:protobuf`, is not used or
referenced anywhere in the repository. So I have removed `docs:gen:protobuf` from the taskfile.

* [skip changelog] Sync testing infrastructure with "template" assets

We have assembled a collection of reusable project assets:
https://github.com/arduino/tooling-project-assets
These will be used in the repositories of all Arduino tooling projects.

Some minor improvements and standardizations have been made in the upstream "template" assets, and those are hereby
introduced to this repository.

Notable:

- Configure paths filters to avoid unnecessary workflow runs
- Increased parallelism
- Improved maintainability

* Remove obsolete testing tasks

The sync with the template testing assets has resulted in new standardized names for some of the test runner tasks:

- `test-unit` -> `go:test`
- `test-integration` -> `go:test-integration`

* [skip changelog] Update package names in imports

As of Go 1.7 the `golang.org/x/net/context` package is available in the standard library under the name `context`.

* [skip changelog] Tidy root module dependencies

Result of running `go mod tidy` with Go 1.16.6.

* [skip changelog] Tidy `github.com/arduino/arduino-cli/arduino/discovery/discovery_client` module's dependencies

Result of running `go mod tidy` with Go 1.16.6.

* [skip changelog] Tidy `github.com/arduino/arduino-cli/term_example` module's dependencies

Result of running `go mod tidy` with Go 1.16.6.

* [skip changelog] Tidy `github.com/arduino/arduino-cli/docsgen` module's dependencies

Result of running `go mod tidy` with Go 1.16.6.

* [skip changelog] Fix broken workflow link

The GitHub Actions workflow used to deploy the documentation website was renamed to a filename more suitable for
existence in a collection of reusable workflows that have multiple variations according to project requirements.

The link to the file was not updated accordingly, which resulted it breaking.

* [skip changelog] Move website versioning information out of CONTRIBUTING.md

This content is for maintainers, and is not of interest to the average contributor. It makes an already overly long
contributor guide even more overwhelming, and thus likely to be skipped.

The documentation versioning and publishing system is now maintained in a centralized repository hosting a collection of
reusable project assets. The documentation content previously located in CONTRIBUTING.md has all been migrated there and
expanded.

Since this is not high level contributor guidelines, it will be better to maintain it in a single place, linking to that
documentation as needed.

* [skip changelog] Sync documentation assets with templates

We have assembled a collection of reusable project assets:
https://github.com/arduino/tooling-project-assets
These assets will be used in the repositories of all Arduino tooling projects.

Some significant improvements and standardizations have been made in the upstream "template" assets, and those are
introduced to this repository here.

Notable:

- Use markdownlint to check for common problems and formatting.
- Make link check task Windows compatible
- Streamline website versioning system
- Use modern MkDocs versioning features
- Make command reference generation task Windows compatible
- Consolidate all Python dependency management to Poetry
- Standardize documentation-related task names and deprecate old task names

* Remove obsolete documentation tasks

The sync with the template documentation assets has resulted in new standardized names for some of the documentation
management tasks:

- `docs:gen:commands` -> `go:cli-docs`
- `docs:gen` -> `docs:generate`
- `docs:build` -> `docs:check`
- `docs:serve` -> `website:serve`
- `docs:check-links` -> `markdown:check-links`

* [skip changelog] Use standard list prefix in Markdown

This is the standardized ordered list prefix for use in all Arduino Tooling Markdown files.

The Markdown renderer will automatically handle numbering. Manual numbering means that either all following list items
need to be renumbered every time a list item is added or removed, resulting in more work for contributors and reviewers.

* [skip changelog] Use standardized Markdown code block style

"Code fencing" is superior to the indentation code block syntax of Markdown because it permits syntax highlighting. Even
in cases where syntax highlighting is not applicable, a consistent style should be used throughout all the Markdown files
of Arduino Tooling projects.

* Update github.com/miekg/dns lib to v1.1.43

* fix go.sum in docsgen folder

* fix go.sum in client_example folder

* [skip changelog] Sync general formatting check workflow with template

We have assembled a collection of reusable GitHub Actions workflows:
https://github.com/arduino/tooling-project-assets
These workflows will be used in the repositories of all Arduino tooling projects.

The Prettier formatting tool is used in the tasks for formatting and checking the formatting of the repository's
Markdown and YAML files. Prettier supports a long list of other languages. There is no reason not to take advantage of
those capabilities as well, so the new workflow applies it to all supported files.

Several of the tasks have been deprecated in favor of the new `general:format-prettier` task:

- `docs:check`
- `docs:format`
- `config:check`
- `config:format`

These still work as before, but they now display a deprecation warning.

* Remove superfluous formatting tasks

These are replaced by the `general:format-prettier` task. Even though that doesn't provide a formatting compliance check,
the developer doesn't have any need for such a thing, since they will be better off to simply format, and the CI now
takes the approach of formatting and then failing if it produced a diff.

* [skip changelog] Bring all files into Prettier formatting style compliance

Previously, Prettier was only run on the Markdown and YAML files. With the new expanded scope of Prettier coverage, the
JSON files are now required to be compliant.

Dependabot is configured to submit pull requests for updates to the GitHub Actions actions used in the repository's
workflows whenever a new version is available. These commits will not result in any user facing changes and so should not
be mentioned in the release changelog.

The changelog generation system automatically excludes any commit that has the `[skip changelog]` prefix on its message
title. Adding this changelog to the commits generated by Dependabot will save the release manager time when cleaning up
the raw changelog.

We have assembled a collection of reusable GitHub Actions workflows:
https://github.com/arduino/tooling-project-assets
These workflows will be used in the repositories of all Arduino tooling projects.

Some minor improvements and standardizations have been made in the upstream "template" workflow, and those are introduced to this repository via this pull request.
Notable:

- Trigger workflow run on modification to facilitate testing.
- `repository_dispatch` event trigger to allow for automated triggering across many repositories via the GitHub API following a relevant external change.
- Change Slack webhook repository secret name.
- Use major version ref of `rtCamp/action-slack-notify` so that the latest release of the action is used up to the next major bump.

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1 to 2.0.2.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1...v2.0.2)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/github-script](https://github.com/actions/github-script) from 3 to 4.0.2.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v3...v4.0.2)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-type: direct:production
  update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [rtCamp/action-slack-notify](https://github.com/rtCamp/action-slack-notify) from 2.1.0 to 2.2.0.
- [Release notes](https://github.com/rtCamp/action-slack-notify/releases)
- [Commits](https://github.com/rtCamp/action-slack-notify/compare/v2.1.0...v2.2.0)

---
updated-dependencies:
- dependency-name: rtCamp/action-slack-notify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 1 to 2.2.2.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v1...v2.2.2)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bring Go code formatting into compliance

* Expand i18n coverage to all relevant strings

This provides internationalization of the strings of the Arduino CLI code base. It will now be possible to completely
translate the primary Arduino CLI user interface.

We decided that the log messages would be excluded from the internationalization scope for now. The reason is that the
addition of these strings would significantly increase the workload on the community translators, while generally being
of less visibility and/or importance to the users. Even though on an individual basis there are surely specific log
messages of higher importance than specific non-log strings that were internationalized, the log/non-log dichotomy
provided an objective criterion for determining which strings were in scope for this initial effort. Perhaps we will
expand the i18n coverage at some time in the future after there has been good progress on the initial translation effort.

Some techniques were employed to facilitate translation:

- Code references were moved out of strings in cases where a translator might mistake them for prose and translate them.
- Indices were added to printf verbs when multiple were present in a string. This allows the translator to easily insert
  them at the appropriate location in the sentence structure without being required to understand the Go printf syntax
  and without being restricted to their relative order in the English language source string.
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

Dependabot will periodically check the versions of all actions used in the repository's workflows. If any are found to
be outdated, it will submit a pull request to update them.

NOTE: Dependabot's PRs will occasionally propose to pin to the patch version of the action (e.g., updating
`uses: foo/bar@v1` to `uses: foo/bar@v2.3.4`). When the action author has provided a major version ref, use that instead
(e.g., `uses: foo/bar@v2`). Dependabot will automatically close its PR once the workflow has been updated.

More information:
https://docs.github.com/en/github/administering-a-repository/keeping-your-actions-up-to-date-with-dependabot

Dependabot Preview shut down 2021-08-03. It is replaced by GitHub-native Dependabot, which uses the
`.github/dependabot.yml` configuration file, with a different data format.

From an investigation into how the obsolete configuration could be migrated to the new format, I determined that the
specific "security updates only" configuration used by this repository is not provided by the new configuration file,
which is for configuring updates of any outdated dependencies. The "security updates" feature is configured via the
repository settings (Settings > Security & analysis).

Since the old file is no longer used, and might cause confusion, it is best to remove it from the repository.

The user can configure their locale preference via the `locale` configuration key. This information is given priority by
the `github.com/arduino/arduino-cli/configuration` package over the automatically detected locale.

Previously, the `i18n` package got the configuration setting from the `github.com/arduino/arduino-cli/configuration`
package, but this will result in an import cycle when the `i18n` package is used to enable translation of the output
strings of the `configuration` package.

To avoid this, the caller now reads the configuration and passes the locale code to the `i18n` package via its `Init`
function:

i18n.Init("it")

The argument can be omitted if only automated locale detection is needed:

i18n.Init()

* Cache board identification properties

* Board details now shows pluggable discovery identification props

* Use new methods in go-properties-orderedmap to calculate subsets

* Updated i18n

* Added migration docs

* Improved tests

* Update arduino/cores/board.go
Co-authored-by: per1234 <accounts@perglass.com>

* Update docs/UPGRADING.md
Co-authored-by: per1234 <accounts@perglass.com>

For the sake of portability of tasks, Task uses `mvdan.cc/sh` to provide an command interpreter that is independent of
the environment. While one of the great strengths of Task, this produces unexpected results in some rare cases. This is
one of those cases. There is some unusual interaction between `mvdan.cc/sh`, Windows, and running the `rice embed-go`
command from the subshell created by the parentheses in this command:

```
$ task i18n:update
←[32mtask: [i18n:update] go run ./i18n/cmd/main.go catalog generate . > ./i18n/data/en.po
←[0m←[32mtask: [i18n:generate] git add -N ./i18n/data
←[0m←[32mtask: [i18n:generate] git diff --exit-code ./i18n/data &> /dev/null || (cd ./i18n && rice embed-go)
←[0merror reading package: go/build: go list github.com/arduino/arduino-cli/i18n: exec: "go": executable file not found in %PATH%
```

It is my belief that the sole purpose of these parentheses was to group the commands and that there is no need to run
these commands in a subshell. If so, the change to using braces to group the commands without the creation of a subshell
will have no functional effect on the commands, but will allow the task to run on Windows.

Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>

* Refactor codebase to use a single Sketch struct

* Add back removed legacy test

* Changed Sketch struct to not use pointers

* Fix small comment

* Fix checkSketchCasing to work with .pde files too

* Add tests loading Sketch with symlinks

* Update UPGRADING.md

* Clarify a code comment

* Add some more tests for Sketches with symlinks

* Fix Sketch symlinks tests for Windows

* Implemented dry-run for upload and burn-bootloader

* Reduced Reset timeout in dry-run mode

* Fix discovery client

* Fix client_example

* Fixed BoardWatch message loop

* Upgraded go-properties-orderedmap to v1.5.0

* Added a couple of utility functions

* Update arduino/cores/packagemanager/package_manager.go
Co-authored-by: per1234 <accounts@perglass.com>

* Make linter happy
Co-authored-by: per1234 <accounts@perglass.com>

Also fixed a bug that prevented tools-reload after auto install in some
rare circumstances (almost impossible to trigger actually).

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

In the early days of Arduino CLI's gRPC interface, the protocol buffer compilation was done via `go generate`, configured
by a directive comment. The `protoc:compile` task is now used instead, but the old directive was left in place, where it
became outdated through changes to the repository structure.

This directive no longer functions or serves a purpose and might cause confusion to contributors. So it should be removed.

The key names used in Arduino CLI's JSON output were made consistent and standardized. One of the helper functions for
the integration tests was not updated accordingly. This was missed because it is only used when running the tests locally
due to requiring a board to be connected to the computer.

* added '.tpp' file ext to known list

* Add support for .ipp Sketch files extension

* Add tests for new .tpp and .ipp extensions suppport
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>

* Implemented pluggable discovery board matching

Added also a compatibility layer for existing platforms.

* Fixed docs and linter warnings

* Better comments

* Added tests and fixed a bug