GitLab

* Simplified gRPC streams helpers

* Moved FeedStreamTo and ConsumeStreamFrom into deamon package and mde them private

[breaking] Remove auto detection of Arduino IDE built-in libraries and tools / Allow gRPC install of built-in libraries (#1817)

* Removed IDE-bundle autodetection

* Do not return bundled tools folder with HardwarePaths

* Created configuration functions to obtain data dir and downloads dir

* Bundled libraries configuration is now considere a single directory

* Allow library installation on bundled libs dir via gRPC

* Added integration test for library install in bundled directory

* Fixed nil pointer exception in build options extraction

* Updated docs

* Removed useless custom env bindings

* Applied code review changes
Co-authored-by: per1234 <accounts@perglass.com>

* Set builtin libraries dir by default when running as a daemon

* Added test for regression

See https://github.com/arduino/arduino-cli/pull/1817#pullrequestreview-1092021620

* Fixed regression: 'lib uninstall' must operate only on user dir
Co-authored-by: per1234 <accounts@perglass.com>

* Migrated TestCompletionNoArgs from test_completion.py to completion_test.go

* Migrated TestCompletionBash from test_completion.py to completion_test.go

* Migrated TestCompletionZsh from test_completion.py to completion_test.go

* Migrated TestCompletionFish from test_completion.py to completion_test.go

* Migrated TestCompletionPowershell from test_completion.py to completion_test.go

* Migrated TestCompletionBashNoDesc from test_completion.py to completion_test.go

* Migrated TestCompletionZshNoDesc from test_completion.py to completion_test.go

* Migrated TestCompletionFishNoDesc from test_completion.py to completion_test.go

* Migrated TestCompletionPowershellNoDesc from test_completion.py to completion_test.go

* Migrated TestStaticCompletions from test_completion.py to completion_test.go

* Migrated TestConfigCompletion from test_completion.py to completion_test.go

* Migrated TestLibCompletion from test_completion.py to completion_test.go

* Migrated TestCoreCompletion to completion_test.go and deleted test_completion.py

* Added Category field to Platform gRPC struct

* Renamed gRPC response from Category string to Type []string

* testsuite: added functions to setup arduino-cli integration tests

* Moved 'requiredjson' library in his own package

* Modified tests in cache_test.go to use the new arduino-cli setup

* Modified tests in main_test.go to use the new arduino-cli setup
Co-authored-by: Cristian Maglie <c.maglie@arduino.cc>

* legacy: Removed ToolsLoader step

It has been splitted and merged into HardwareLoader and TargetBoardResolver
that are more appropriate.

* Fixed some function comments

* Thread-safe protect access to instances map

* Removed state-altering methods from PackageManager

They have been moved into a Builder object that has the ability to build
a new PackageManager. This allows to clearly separate subrotuines that
actually change the status of the PackageManager from subroutines that
just need to query it.

* Created packagemanager.Explorer to query PackageManager data

The Explorer object can be see as a read-only "view" to the underlying
PackageManager: we may ask the PackageManager to create an Explorer on
itself. The returned explorer will held a read-lock on the
PackageManager until it's disposed.

This architecture should prevent unwanted changes on the PackageManager
while it's being used, and viceversa, when the PackageManager is updated
it should be guaranteed that no Explorers are reading it.

* PlatformInstall/Uninstall must release PackageManager.Explorer before calling commands.Init

Otherwise, since Init will try to take a write-lock, it will block
indefinitely.

* Moved commands.InstanceContainer -> rpc.InstanceCommand

* Created a coreInstancesContainer

This container will handle all the atomic access to the instances map.

* Made CoreInstance.PackageManager field private

* Moved the reminder of PackageManager functions to Explorer or Builder

* Now GetPackageManager accepts an rpc.InstanceCommand

It has also been deprecated in favor of GetPackageManagerExplorer.

* Now GetLibraryManager accepts an rpc.InstanceCommand

* Refactored automatic builtin-tool installation

* Added gRPC LibraryUpgrade call and fixed 'lib upgrade' command

* Explorer and Builder should not extend PackageManager

Previuosly the methods PackageManager.NewBuilder and
PackageManager.NewExplorer were available also on Builder and Explorer.

Now Builder and Explorer does not inherith these methods anymore,
avoiding trivial errors like the one fixed in this commit in the
builder_utils package.

* Updated documentation

* Apply suggestions from code review
Co-authored-by: per1234 <accounts@perglass.com>

The `--dump-profile` flag causes Arduino CLI to print the board and sketch dependencies of that compilation as a YAML
document. This could be copied into a "sketch project file" for later use as a "build profile" specified via the
`--profile` flag.

A typo in the `profiles` key name caused the generated "build profile" entry to not be recognized when added to a
"sketch project file".

The key name is hereby corrected.

Arduino CLI has been translated to 12 languages.

The localization process follows the following steps:

1. An English language source string is defined in the Arduino CLI codebase
2. The source string is pushed to Transifex
3. Community translators localize the string
4. The localization data is pulled into the Arduino CLI repository
5. The localization data is incorporated into the Arduino CLI distribution

Experience with maintenance of the Arduino IDE 1.x project indicates that the data files generated at step (4) can
appear to be the appropriate place to make edits for casual contributors not familiar with the project's sophisticated
internationalization infrastructure.

Since those files are generated by automated systems, any edits made there would only be overwritten, so it is important
to clearly communicate the correct way to make enhancements or corrections to these strings. This is accomplished by a
local readme file most likely to be seen by those working in the folder containing these files, which supplements the
existing information about translation in the project's CONTRIBUTING.md.

* Migrated TestHelp from test_main.py to main_test.go

* Migrated TestVersion from test_main.py to main_test.go

* Migrated TestInventoryCreation from test_main.py to main_test.go

* Migrated TestLogOptions to main_test.go and deleted test_main.py

* [skip changelog] Remove unused link reference definitions in docs

Parts of the documentation use reference links, where the inline markup uses a label, which matches to a link reference
definition elsewhere in the documentation.

The major disadvantage of this approach is that unused link reference definitions are often left behind after the
referencing text is removed, cluttering up the documentation and increasing the maintenance burden. That was the case
here.

The unused link reference definitions are hereby removed.

* [skip changelog] Fix broken links in documentation

Some of the documentation links no longer reached their intended target.

These were not caught by the Markdown link check for one of two reasons:

- Redirects to a useless generic page instead of the moved specific target page
- Links in non-Markdown files

* [skip changelog] Update documentation links that depend on redirects

Some of the links in the documentation used URLs that were redirecting to the new location of the target page.

Even though these links still took the reader to the intended place eventually, it is safest to not rely on them
continuing to work for the long term, so is best to update to point directly to the target.

* Improved streaming of pluggable-discoveries events (WIP)

Now the DiscoveryManager is able to start the discoveries and add/remove
them in a thread-safe way. Also the watchers may connect and disconnect
seamlessly at any time, the incoming events from the discovery are
broadcasted correctly to each active watcher.

This refactoring dramatically simplifies the DiscoveryManager design.

* Added discovery id in discovery.Event struct

* Cache active ports and transmit them when a new watcher connects

* Correctly handle discovery cleanup

* Fixed wrong test

* Correctly handle discovery cleanup and re-add

* Added some doc comments in the source code

* Move Unlock under defer

* Factored subrotuine into a function

it will be useful in the next commits.

* Do not cache ports in the DiscoveryClient

there is already a cache in the DiscoveryManager there is no need to
duplicate it.

* Discovery: eventChan must be protected by mutex when doing START_SYNC

otherwise the discovery may send some events before the eventChan is
setup (and those events will be lost)

* Increased error level for logging watchers that lags

* Updated discvoery_client to the latest API

* Report discovery start errors

* Update arduino/discovery/discovery_client/main.go
Co-authored-by: Umberto Baldi <34278123+umbynos@users.noreply.github.com>
Co-authored-by: Umberto Baldi <34278123+umbynos@users.noreply.github.com>

Signed-off-by: Abirdcfly <fp544037857@gmail.com>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Added test for issue #1823

* Do not assume that version in platforms.txt is correct semver

* When parsing version in platform.txt perform a variable replace

* Update internal/integrationtest/core/testdata/issue_1823/DxCore-dev/megaavr/README.md

* Update internal/integrationtest/core/core_list_test.go
Co-authored-by: per1234 <accounts@perglass.com>

* testsuite: Added helper functions to handle test envs

* testsuite: Added helper functions to run arduino-cli

* testsuite: Added colored output to arduino-cli Run helper

* testsuite: Pass cli config through env variables

* testsuite: added env commands to download and extract files

* testsuite: added commands to start cli daemon and run some gRPC calls

* testsuite: moved test harness inside 'internal' package

* testsuite: added first daemon test for gRPC board watch

* testsuite: added http server helper

* testsuite: added JSON helpers

* testsuite: Added possibility to use shared download staging folder

* testsuite: Converted a core_test.py test (WIP)

* REMOVEME: Deactivate daemon integration test for now

* testsuite: force colored output

* testsuite: moved all generic subroutines into their own library

* testsuite: moved daemon test in his own dir

* fixed typo

* testsuite: added some helpers to improve daemon testing

* testsuite: added test for #1614

* Removed converted test

* testsuite: perform build before test

Otherwise people running the task locally will have a bad time when the test
either fails due to the missing executable, or far worse when they get
incorrect test results because they don't realize it is using whichever random
executable happened to be sitting around from the last time they did a build.

* Added missing comment

* Renamed test file

* Skip failing tests

* Updated licensed cache

* re-enable test for fixed bug

* testsuite: disable postinstall

Otherwise Windows CI will get stuck.

* Removed useless startDaemon

* Renamed inst -> grpcInst

* Close daemon gRPC connection on test cleanup

* Added comment

* Apply suggestions from code review
Co-authored-by: per1234 <accounts@perglass.com>

* Update internal/integrationtest/arduino-cli.go
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

`github.com/arduino/arduino-cli/i18n.Tr` provides a `fmt.Printf`-style interface, where the first argument may serve as
a template filled by subsequent arguments according to its format verbs.

This call contains a subsequent argument, but no verb in the format string to use it. This generates some cryptic
content in the error message.

For example:

Error during install: Error downloading tool arduino:bossac@1.7.0-arduino3: no versions available for the current OS%!(EXTRA *cores.ToolRelease=arduino:bossac@1.7.0-arduino3)

The tool name is already provided in the correctly configured `Message` field of the error, so there is no need to
include it in the `Cause` field as well, so the spurious argument can be removed entirely rather than the alternative
fix of adjusting the format string argument. I suspect the spurious argument was simply the result of a copy/paste error.

* Made PackageManager.TempDir private

* Merged InstallToolRelease function into the proper packagemanager method

* Moved uninstallToolRelease into the proper packagamanger method

* Moved uninstallPlatformRelease into the proper packagamanger method

* Moved downloadToolRelease into the proper packagamanger method

* Merged downloadTool with the proper packagamanger method

* Merged downloadPlatform with the proper packagamanger method

* Moved installPlatform into a packagamanger method

* Moved upgradePlatform into a packagamanger method

* Made PackageManager.Log field private

* Removed the massive code duplication in the 'upgrade' command

* Removed the massive code duplication in the 'outdated' command

* Updated docs

* Update arduino/cores/packagemanager/install_uninstall.go
Co-authored-by: per1234 <accounts@perglass.com>

* Update arduino/cores/packagemanager/install_uninstall.go
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

* Bugfix: do not overwrite board properies during compile

Fix #1614

* Removed AddBuildPropertyIfMissinc 'legacy' (inlined in other functions)

* Increase code coverage

Versions of the `codecov/codecov-action` GitHub Actions action prior to 1.0.6 required the use of a token provided by
Codecov in order to upload data to Codecov. This token was stored in secret in the Arduino CLI repository and used in
the test workflow.

For security reasons, secrets are not accessible when a workflow is triggered by an event generated by a fork of the
repository. This meant that it was impossible to upload coverage data for the test runs triggered by PRs from forks. A
conditional was added to the upload step of the workflow to cause it to only run on `push` event triggers, which
effectively prevented its failure for runs on PRs from forks.

The token requirement was removed in the 1.0.6 release of `codecov/codecov-action`, but the now pointless conditional
was never removed from the workflow. This prevented PRs from forks from receiving the automated code coverage report
comments that would otherwise encourage those contributors to resolve coverage deficiencies and facilitate the review
process.

The harmful conditional is hereby removed from the coverage data upload steps of the workflow and PRs from forks will
now receive coverage report comments, just as PRs from branches do already.

* Added tests for #1669

* Fixed regession in discoveries startup

Fix #1669

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This is done by providing the desired revision in the fragment, e.g.
`…/Library.git#0.1.0`. When set, this disables the clone depth limit so
all remote references will be available.

* packagemanager: Show size difference between package_index and downloaded file

This will help identify the problem:
- if the file is corrupted
- if the package_index is wrong
- if there is a bug in the CLI (more rare... but actually possible :-)

* Load package indexes after installed packages

The installed packages keeps an extract of the original
package_index.json, at the moment of the installation, inside
a file named installed.json. This extract may turn out useful
if the original package_XXX_index.json is lost or removed to keep the
platform functional. On the other hand, if the original
package_XXX_index.json is modified the information kept in the
installed.json may be outdated and should be replaced by the upstream
index: this is the reason why it's loaded after the hardware platforms.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Fix path in error messages on sketch loading

* Added integration tests

* Updated unit-test

* Updated integration test

* Even better error messages

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Added skip-libraries-discovery flag in Compile

* Always send compile response even in case of error

* Renamed variable

* Moved 'alread-installed' check in the proper place

* Added --no-overwrite flag in core install

* Fixed message

* Added tests

* Update cli/core/install.go
Co-authored-by: per1234 <accounts@perglass.com>

* Update rpc/cc/arduino/cli/commands/v1/core.proto
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

* Added no_overwrite field in LibraryInstall

* Added tests

* Check if libraries needs to be installed prior to download/install

* Update rpc/cc/arduino/cli/commands/v1/lib.proto
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

* Provide a way to define extra UserAgent fields via env vars

* Add secret token to User Agent

This will allow to possibly bypass spam-checks on the server side
reducing the jobs failures due to rate limiting.

[breaking] Fix gRPC streaming synchronization on close (should fix missing lines at the end of compile) (#1787)

* Correctly synchronize streams (Fix: some lines cut-off at the end of the output)

* Updated docs

* Fixed go-docs

* Use pipes with channel buffering to smooth out communications

* Update docs/UPGRADING.md
Co-authored-by: per1234 <accounts@perglass.com>
Co-authored-by: per1234 <accounts@perglass.com>

Arduino libraries may specify dependencies on other libraries in their metadata. The Arduino Library Manager offers the
option to install these dependencies when the user installs the library.

By default, the latest version of the dependency is installed. However, libraries may be compatible or tested working
only with specific versions of a dependency. In this case, the library developer may wish to specify a particular
version or version range of the dependency.

Arduino CLI and the Arduino Library index system has had support for such dependency version constraints for the last
2.5 years, but it was never documented so has only been used by the few library developers who discovered it via other
means.

The dependencies version constraint system is hereby fully documented in the Arduino library specification. This will
allow meaningful benefit to finally come from the work done to add this valuable capability.

* [skip changelog] Revert "codecov: Disable pull request comments"

This reverts commit 5c9eee34.

Codecov will now automatically comment on pull requests to provide a report of the code coverage impact that would
result from the proposed changes.

These comments are valuable for several reasons:

- Brings test deficiencies to the contributor's attention
- Encourages contributors to improve PRs that originally had insufficient test coverage for the added code
- Facilitates the review process for the maintainers
- Increases awareness of code coverage trends in the project (the Codecov website is great for this, but likely rarely visited)

* [skip changelog] Add support to "Test Go" workflow for alternative Codecov configuration paths

For the sake of efficiency, the "Test Go" GitHub Actions workflow is configured to run only when relevant files are
modified. Since the workflow uploads code coverage data to Codecov, the Codecov configuration file is one of these files.

The standard filename for the Codecov configuration file is codecov.yml, and the workflow's path filter was configured
for that filename. It turns out an alternative filename is also recognized: .codecov.yml, which is used by this project.
Two subfolders are also supported in addition to the root of the repository as locations for the configuration file.

The workflow's paths filter was not configured for the alternative filename and locations, meaning the workflow would
not be triggered on change to the Codecov configuration in projects that use the alternative configuration file name or
locations.

The workflow's paths filter is hereby configured to recognize changes to any valid Codecov configuration file.

During a refactoring of the "Publish Nightly Build" workflow, globbing of the macOS build artifact filename in the
repackaging step was accidentally lost, causing the glob pattern to be treated instead as a string and the updated
package saved to a file named by that string instead of to the original filename.

This resulted in the workflow failing with errors like:

Artifact path is not valid: /arduino-cli_nightly-*macOS_64bit.tar.gz. Contains the following character: Asterisk *

The previous `basename` command was providing the globbing. After the refactoring, `basename` is no longer needed, so an
alternative way to achieve globbing is needed. It seems the preferred approach is use of an array. Since the globbing
will only expand to a single filename, the array can be referenced as before by the rest of the step, since it will
resolve to the first element in the array (equivalent to `$PACKAGE_FILENAME[0]`).