GitLab

* Run integration tests concurrently

This enhancement reduces the amount of time needed to execute integration tests, since each test package no longer has to wait for the previous ones to be completed. In order to do this, a regex is used
to match each test file. It is also specified the path to the directory that contains the file.

* Merge all test files that share a package into one

Having different test files that shared the same package was inefficient, because the whole package test was run one time for each file. This enhancement avoids repeating the same tests more than once.

* Added in_development flag in Library gRPC message

* Small cosmetic fix in docs

* Implemented 'in_development' flag in libraries loader

* Added unit-test

Arduino IDE allows opening searches in Library Manager and Boards Manager by clicking a link with a special format in a
comment of the sketch.

During the initial implementation in Arduino IDE 1.x, support for setting the "Type" filter via the URI in addition to
specifying search keywords was attempted and claimed to work, but never did work. For this reason, it was not previously
mentioned in the sketch specification.

During the reimplementation of the Library/Boards Manager link feature for Arduino IDE 2.x, the "Type" filter setting
capability was finally truly implemented, and expanded to also support setting the "Topic" filter in Library Manager via
the same mechanism.

Now that support for the filter setting capability has been released in Arduino IDE 2.0.1, it should be documented in
the sketch specification.

* Allow 'locale'key to be used in 'config ...' commands

* Fix documentation

* Update docs/configuration.md
Co-authored-by: per1234 <accounts@perglass.com>

* Update docs/configuration.md
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

The `lib search` command accepts search keywords as an argument. All libraries with metadata within the search scope
matching those keywords will be returned.

Previously, the error message used the singular form "library". Since the command is as likely to be used to search for
libraries, using the plural form "libraries" is more appropriate.

Users opened a task inside Transifex requesting this change.
Co-authored-by: per1234 <accounts@perglass.com>

Add task to collect proto files and add steps to release them with tester, nightly and stable builds (#1931)

* Add task to zip all .proto files for release

* Add proto files in nightly released files

* Add proto files in tester released files

* Add proto files in stable released files

* Fix tester build workflow

* Fix checksums.txt creation in tester and stable build releases

* Change build job of tester build workflow to be generic

* Fix Publish Tester Build job dependencies

Bumps [carlosperate/download-file-action](https://github.com/carlosperate/download-file-action) from 1 to 2.
- [Release notes](https://github.com/carlosperate/download-file-action/releases)
- [Commits](https://github.com/carlosperate/download-file-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: carlosperate/download-file-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [xt0rted/markdownlint-problem-matcher](https://github.com/xt0rted/markdownlint-problem-matcher) from 1 to 2.
- [Release notes](https://github.com/xt0rted/markdownlint-problem-matcher/releases)
- [Changelog](https://github.com/xt0rted/markdownlint-problem-matcher/blob/main/CHANGELOG.md)
- [Commits](https://github.com/xt0rted/markdownlint-problem-matcher/compare/v1...v2)

---
updated-dependencies:
- dependency-name: xt0rted/markdownlint-problem-matcher
  dependency-type: direct:production
  update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Copy package to destination if move failed (should fix sporadic `Access is denied` errors on Windows) (#1938)

GitHub Actions provides the capability for workflow authors to use the capabilities of the GitHub Actions ToolKit
package directly in the `run` keys of workflows via "workflow commands". One such command is `set-output`, which allows
data to be passed out of a workflow step as an output.

It has been determined that this command has potential to be a security risk in some applications. For this reason,
GitHub has deprecated the command and a warning of this is shown in the workflow run summary page of any workflow using
it:

The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more
information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

The identical capability is now provided in a safer form via the GitHub Actions "environment files" system. Migrating
the use of the deprecated workflow commands to use the `GITHUB_OUTPUT` environment file instead fixes any potential
vulnerabilities in the workflows, resolves the warnings, and avoids the eventual complete breakage of the workflows that
would result from GitHub's planned removal of the `set-output` workflow command 2023-05-31.

* Removed redundant and useless commands from gRPC interface

* Moved some flags in the command creation

This will turn out useful in the following commits

* Made some cli procedures available for other packages

Unfortunately we cannot limit the availability of the cli.core.* package
to the cli.* packages, but they are now available as public API.

* Moved 'lib list' flags into command creation

* Moved 'core list' flags into command creation

* Made 'lib list' procedure public

* Made 'core list' procedure public

* instance.Init no longer return errors array.

They are printed directly inside the function through the feedback
package.

* Ultra-simplified 'outdated' 'update' and 'upgrade'

* Moved 'update' flags in command creation

* Fixed integration tests

* Updated docs

* Fixed integration tests

* Migrate TestUpdate from test_update.py to update_test.go

* Migrate TestUpdateShowingOutdated from test_update.py to update_test.go

* Migrate TestUpdateWithUrlNotFound from test_update.py to update_test.go

* Migrate TestUpdateWithUrlInternalServerError from test_update.py to update_test.go

* Migrate TestUpdateShowingOutdatedUsingLibraryWithInvalidVersion to update_test.go and delete test_update.py

* Change lib deps CLI command output to sorted alphabetically

* Fix plain text output not sorted correctly

* Simplify deps sorting

* Reworked test to prevent future breakages

As a newcomer, this made me assume ARMv6 is not supported so I searched the issues, found https://github.com/arduino/arduino-cli/issues/340, which suggested building from source so I did, only to discover an ARMv6 binary in the release files.

As for the table layout, it's not ideal. It would also make sense to have this in another column.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Added test for os-specific config options

* Build board config options structures only once and cache them

* Board's build options properties are now calculated only once and cached

* Added tests for config options ordering

It required insertion of test data with the properties.Set method to
preserve ordering.

* Renamed some variables to improve code readability

* Added board config identification subroutines

* Added board config detection in 'commands.identify' function

* Updated docs

* Apply suggestions from code review
Co-authored-by: per1234 <accounts@perglass.com>

* Fixed comment
Co-authored-by: per1234 <accounts@perglass.com>

* add step to generate a windows msi installer of the Arduino CLI

* remove warning:

`warning : Solution properties are only available during IDE builds or when building the solution file from the command line. To turn off this warning set <DefineSolutionProperties>false</DefineSolutionProperties> in your .wixproj`

* fix error caused by wix peculiar version constraints:

`error CNDL0108: The Product/@Version attribute's value, '0.0.0-test', is not a valid version.  Legal version values should look like 'x.x.x.x' where x is an integer from 0 to 65534.`

* fix `error LGHT0103: The system cannot find the file 'arduino-cli.exe'`

* final adjustments in wix config

* exclude prettier from checking the installer config files, prettier does not support xml out of the box

* add download links

* temporarily disable s3 push and homebrew update for testing

* Upload nightly artifacts for testing

* add windows check certs

* bump to latest windows-sdk and document better

specify timestamp algorithm. Apparently this is required in the latest version of the sdk

* remove useless job id not used anymore (followup of #1883)

* fixed comments

* It's useless to use the output from setupmsbuild

* use YAML-style syntax

* try to uniform the usage of version/tag across the workflow

* Revert "Upload nightly artifacts for testing" && "temporarily disable s3 push and homebrew update for testing"

This reverts commit 3123dfd8ea0b2fa4d2fd33aa4ccd5696f34355d2.
This reverts commit 24fa25ca65be2d8f149729f3c169479e1848fa51.

* Apply suggestions from code review
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

* Update publish-go-tester-task

* Update publish-go-nightly-task

* Update release-go-task

* Migrate TestCompileSketchWithPdeExtension from test_compile_part_3.py to compile_part_3_test.go

* Migrate TestCompileSketchWithMultipleMainFiles from test_compile_part_3.py to compile_part_3_test.go

* Migrate TestCompileSketchCaseMismatchFails from test_compile_part_3.py to compile_part_3_test.go

A new function that sets the working directory to a specific path has been added in order to run all the tests as intended.

* Migrate TestCompileWithOnlyCompilationDatabaseFlag from test_compile_part_3.py to compile_part_3_test.go

* Migrate TestCompileUsingPlatformLocalTxt from test_compile_part_3.py to compile_part_3_test.go

* Migrate TestCompileUsingBoardsLocalTxt from test_compile_part_3.py to compile_part_3_test.go

* Migrate TestCompileManuallyInstalledPlatform from test_compile_part_3.py to compile_part_3_test.go

* Migrate TestCompileManuallyInstalledPlatformUsingPlatformLocalTxt from test_compile_part_3.py to compile_part_3_test.go

* Migrate TestCompileWithFullyPrecompiledLibrary to compile_part_3_test.go and delete test_compile_part_3.py

* Rearrange tests to have them share the same environment

Bumps [geekyeggo/delete-artifact](https://github.com/geekyeggo/delete-artifact) from 1 to 2.
- [Release notes](https://github.com/geekyeggo/delete-artifact/releases)
- [Commits](https://github.com/geekyeggo/delete-artifact/compare/v1...v2)

---
updated-dependencies:
- dependency-name: geekyeggo/delete-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

The original copy sketch function written in conftest.py copies sketches into the testsuite's working directory. Changing the copy sketch function present in the golang testsuite to match the original one,
should prevent errors that could occur while migrating tests.

* Move upload of coverage data to dedicated job in test workflow

The code coverage data generated by running the unit tests in the "Test Go" GitHub Actions workflow is uploaded to
Codecov.

This upload is prone to occasional transient failures. With the previous workflow configuration this caused a failure of
the Linux test run job. Contributors would need to review the logs to understand that these failures were not caused by
a test failing. The failure also causes the ongoing jobs that run the tests on other operating systems to be immediately
canceled, meaning their full test results are not available. The entire test suite must be reran just to attempt the
upload again.

Moving the upload to a dedicated workflow job makes it faster and easier for contributors to interpret the cause of a
test failure. It also allows the test suite to complete for all operating systems, making their results immediately
available to the contributor even when the coverage data upload fails. The coverage upload job can be reran, making
recovery from a transient failure more efficient.

* Use a single step to upload coverage data in "Test Go" workflow

The code coverage data generated by running the unit tests in the "Test Go" GitHub Actions workflow is uploaded to
Codecov. Arduino CLI's unit tests are split into two collections: those covering the modernized part of the codebase,
and those covering the "legacy" code inherited from arduino-builder. A separate code coverage data file is produced for
each of these collections.

The `codecov/codecov-action` GitHub Actions action is used to perform this upload. Previously, a separate step was used
for the upload of each of the code coverage files. The action provides the capability to specify multiple files for
upload in a comma-separated list. This approach will make the workflow more efficient and reliable.

* Migrated TestBoardList from test_board.py to board_test.go

* Migrated TestBoardListWithInvalidDiscovery from test_board.py to board_test.go

* Migrated TestBoardListall from test_board.py to board_test.go

* Migrated TestBoardListallWithManuallyInstalledPlatform from test_board.py to board_test.go

* Migrated TestBoardDetails from test_board.py to board_test.go

* Migrated TestBoardDetailsNoFlags from test_board.py to board_test.go

* Migrated TestBoardDetailsListProgrammersWithoutFlags from test_board.py to board_test.go

* Migrated TestBoardDetailsListProgrammersFlags from test_board.py to board_test.go

* Migrated TestBoardSearch from test_board.py to board_test.go

* Migrated TestBoardAttachWithoutSketchJson from test_board.py to board_test.go

* Migrated TestBoardSearchWithOutdatedCore to board_test.go and deleted test_board.py

To ensure functionality on all platforms, the "Test Go" GitHub Actions workflow runs the tests on Windows, Linux, and
macOS. This is done via a "job matrix", which runs the three jobs in parallel.

By default GitHub Actions uses a "fail fast" behavior for job matrixes, where all in progress or pending jobs of the
matrix are immediately canceled upon failure of any job.

The benefit of this "fail fast" behavior is efficiency in the case where a failure is universal and those other jobs
were certain to eventually fail. However, in other cases a failure is either specific to the configuration of a single
job (e.g., a Windows-specific bug), or a spurious result of a transient network outage. In the latter case, canceling
the other jobs is very harmful. Running the jobs for the other operating systems would highlight the nature of an
OS-specific failure for the contributor. Canceling other jobs due to a transient failure means all those jobs must be
reran instead of only the specific job that suffered the failure.

The workflow is hereby configured to disable the "fail fast" behavior. All the matrix jobs will now run even if one of
them has failed.

* Factor all compile_part_1 and 2 test setup

* Made an object to simplify running of subtests

* Moved all subtests in the same file

Just moved functions around, no code change.

* Reuse installed platform to speedup debug integration tests

* Integrated 'go.bug.st/testsuite'

It doesn't make much sense to keep it separate in a different library

* Do not use shared download dir for 'cache clean' command tests

* Added method to get cli download dir

* Create infra for sharing directories between tests

* Slighlty improved test output

* Non-parallel testing is no longer required

* Add more time to complete tests

* Update internal/integrationtest/cache/cache_test.go
Co-authored-by: MatteoPologruto <109663225+MatteoPologruto@users.noreply.github.com>
Co-authored-by: MatteoPologruto <109663225+MatteoPologruto@users.noreply.github.com>

* Migrate TestInit from test_config.py to config_test.go

* Migrate TestInitWithExistingCustomConfig from test_config.py to config_test.go

* Migrate TestInitOverwriteExistingCustomFile from test_config.py to config_test.go

* Migrate TestInitDestAbsolutePath from test_config.py to config_test.go

* Migrate TestInitDestRelativePath from test_config.py to config_test.go

* Migrate TestInitDestFlagWithOverwriteFlag from test_config.py to config_test.go

* Migrate TestInitDestAndConfigFileFlags from test_config.py to config_test.go

* Migrate TestInitConfigFileFlagAbsolutePath from test_config.py to config_test.go

* Migrate TestInitConfigFileFlagWithOverwriteFlag from test_config.py to config_test.go

* Migrate TestDump from test_config.py to config_test.go

* Migrate TestDumpWithConfigFileFlag from test_config.py to config_test.go

* Migrate TestInitConfigFileFlagRelativePath from test_config.py to config_test.go

* Migrate TestAddRemoveSetDeleteOnUnexistingKey from test_config.py to config_test.go

* Migrate TestAddSingleArgument from test_config.py to config_test.go

* Migrate TestAddMultipleArguments from test_config.py to config_test.go

* Migrate TestAddOnUnsupportedKey from test_config.py to config_test.go

* Migrate TestRemoveSingleArgument from test_config.py to config_test.go

* Migrate TestRemoveMultipleArguments from test_config.py to config_test.go

* Migrate TestRemoveOnUnsupportedKey from test_config.py to config_test.go

* Migrate TestSetSliceWithSingleArgument from test_config.py to config_test.go

* Migrate TestSetSliceWithMultipleArguments from test_config.py to config_test.go

* Migrate TestSetStringWithSingleArgument from test_config.py to config_test.go

* Migrate TestSetStringWithMultipleArguments from test_config.py to config_test.go

* Migrate TestSetBoolWithSingleArgument from test_config.py to config_test.go

* Migrate TestSetBoolWithMultipleArguments from test_config.py to config_test.go

* Migrate TestDelete to config_test.go and delete config_test.py

* Migrate TestCompileWithOutputDirFlag from test_compile_part_2.py to compile_part_2_test.go

* Migrate TestCompileWithExportBinariesFlag from test_compile_part_2.py to compile_part_2_test.go

* Migrate TestCompileWithCustomBuildPath from test_compile_part_2.py to compile_part_2_test.go

* Migrate TestCompileWithInvalidUrl from test_compile_part_2.py to compile_part_2_test.go

* Migrate TestCompileWithExportBinariesEnvVar from test_compile_part_2.py to compile_part_2_test.go

* Migrate TestCompileWithExportBinariesConfig from test_compile_part_2.py
to compile_part_2_test.go

* Migrate TestCompileWithCustomLibraries from test_compile_part_2.py to compile_part_2_test.go

* Migrate TestCompileWithArchivesAndLongPaths from test_compile_part_2.py to compile_part_2_test.go

* Migrate TestCompileWithPrecompileLibrary to compile_part_2_test.go and delete test_compile_part_2.py

* Increase lapse before timeout by 5 minutes

* Download library index as a combined archive+signature

* Fixed integration tests

* uniquify configuration arrays in `config add`

* Add tests for unique board manager URLs

* fix linting

* lint: update function name

* update config set to uniquify slices

* Refactored DownloadProgress protocol

* Updated integration tests

* Do not create overly verbose errors

* Updated docs

* Update rpc/cc/arduino/cli/commands/v1/commands.proto
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

* Removed real_name field from gRPC Library message

* Update docs/UPGRADING.md
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

* Factored function to get search terms

* Made 'core search' logic identical to Arduino IDE 1.8.x search dialog

* Drastically simplified search algorithm for libs

* Fixed SearchTermsFromQueryString function

* Moving SearchTermsFromQueryString into proper place

Each Arduino library has a name, which is used as its sole identifier by the user in `arduino-cli lib` commands, and by
Arduino CLI when referring to the library in messages displayed to the user.

The name is defined by:

- "1.5 format" libraries: `name` field in the library.properties metadata file
- "1.0 format" (AKA "legacy") libraries: installation folder name

The name is resolved when loading the library and stored in the `Name` field of the
`github.com/arduino/arduino-cli/arduino/libraries.Library` struct.

The name of the library's installation folder is used by Arduino CLI in several other ways, most notably for determining
"folder name priority" for use in library dependency resolution. For this reason, the folder name is also stored in the
struct when loading the library.

Arduino CLI and arduino-builder have been plagued by problems caused by the inappropriate use of this folder name as the
identifier for the library instead of the sole correct identifier (which is only the folder name in the case of
"1.0 format" libraries. The design of the `github.com/arduino/arduino-cli/arduino/libraries.Library` struct may have
been a contributing factor in those bugs, since at the time of their occurrence the folder name was stored in the `Name`
field, the metadata-defined name in a `RealName`. In addition to the fact that no one field could be used as a source of
the name in all cases, I suspect the ambiguous field names themselves caused confusion to developers.

This situation was improved by providing the library identifier via a single field for all library formats. The name
provided by this field is the "canonical" name of the library. Inexplicably, at that time the field containing the
folder name was renamed "CanonicalName". The string contained by this field is in no way a "canonical" name for the
library, so the field name is bound to cause more of the same bugs and confusion the redesign of the struct was intended
to prevent.

The inappropriately named `github.com/arduino/arduino-cli/arduino/libraries.Library.CanonicalName` field is hereby
renamed to the accurate `DirName`.