GitLab

* Move Preprocess and Build in arduino/builder

* Move the detector inside arduino/builder

* remove legacy targets from TaskFile

* inline some variable declaration

* remove no longer used builder getters

* make only necessary function public in the new builder

* fix CR: avoid mutating internal state

* fix CR: only print normaloutput when is not verbose

* ported SketchThatIncludesArduinoH from legacy into integration test

* ported SketchWithStaticAsserts from legacy into integration test

* ported SketchWithEnumClass from legacy into integration test

* ported SketchWithExternC from legacy into integration test

* ported testBuilderSketchWithMultilinePrototypes from legacy into integration test

* ported SketchWithExternCMultiline from legacy into integration test

* ported SketchWithMultilineTemplate from legacy into integration test

* ported SketchWithFakeFunctionPointer from legacy into integration test

* fix wrong templating

* ported SketchWithMinMaxDefinitions from legacy into integration test

* ported SketchWithFastledsLibrary from legacy into integration test

* ported sketch with merged sketch and bootloader from legacy into integration test

* temporary disable legacy test, before porting them to the new infra

* update .gitignore

* ported CheckBuildOptionsFile from legacy into integration test

* ported SketchClassFunction from legacy into integration test

* ported SketchThatChecksIfSPIHasTransactionsAndIncludesMissingEthernet from legacy into integration test

* remove legacy builder test folder

* remove unsued fields from context

* move customBuildProperties to arduino/builder

* remove BuildProperties from context

* remove BuildPath from context

* remove sketch,libraries,core build path from Context

* remove buildPath parameter to PrepareSketchBuildPath func

* Make CoreBuilder a method recevier of arduino/builder

* Add BuilderLogger in arduino/builder

* Remove BuilderLogger from CoreBuild parameter

* Make getCachedCoreArchiveDirName unexported

* heavily refactored the ContainerBuildOptions

* remove fqbn from Context

* remove clean from Context

* remove unsued properties in Context

* remove sourceOverrides from Context

* make SketchBuilder a method recevier of arduino/builder

* make BuildLibraries a method recevier of arduino/builder

* make RunRecipe a method recevier of arduino/builder

* make RemoveUnusedCompiledLibraries a method recevier of arduino/builder

* make MergeSketchWithBootloader a method recevier of arduino/builder

* make WarnAboutArchIncompatibleLibraries a method recevier of arduino/builder

* make PrintUsedLibraries a method recevier of arduino/builder

* make ExportCmake and PreprocessorSketch a method recevier of arduino/builder

* remove legacy/constans pkg

* make Linker a method recevier of arduino/builder

* make Size a method recevier of arduino/builder

* remove onlyUpdateCompilationDatabase from Context

* remove Progress from Context

* remove ExecutableSectionSize from Context

* remove CompilationDatabase from Context

* remove LineOffset from Context

* introduce BuilderArtifacts to better isolate write operations

* remove ActualPlatform and TargetPlatform from Context

* directly pass the remaining properties of Context in the builder constructor

The remaning fields is used only by legacy tests

* polish legacy test

Logging with `info` generates too much noise.

Closes arduino/arduino-cli#2313
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>

Related to #2301 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* move CoreBuilder in arduino/builder and move CompilationDatabase in compilation pkg

* rename CompilationDatabse in Database

* move libraries phases in arduino/builder

* move linker phases in arduino/builder

* move sketch_builder phases in arduino/builder

* move sizer phases in arduino/builder/sizer

* create bulder logger

* use builder logger in the sketLibrariesDetector

* replace context Info logger with BuilderLogger

* replace context Warn logger with BuilderLogger

* replace context WriteStdout logger with BuilderLogger

* replace context WriteStderr logger with BuilderLogger

* directly pass the BuilderLogger to RecipeByPrefixSuffixRunner func

* directly pass the BuilderLogger to part2

* directly pass the BuilderLogger to Linker func

* directly pass the BuilderLogger to sizer.Size func

* remove Stdout, Stderr and stdlock from Context

* appease golint

* add missinig license headers

* remove WarningsLevel from context

* remove Verbose from context

* Fixed weird error message in 'core install' if invalid platform is specified

* Fixed integration test

* refactor RecipeRunner in a function

* refactor CreateBuildOptionsMap in a function

* refactor LoadPreviousBuildOptionsMap in a function

* refactor StoreBuildOptionsMap in a function

* refactor WipeoutBuildPathIfBuildOptionsChanged in a function

* refactor ContainerBuildOptions in a function

* refactor MergeSketchWithBootloader in a function

* refactor PrintUsedLibrariesIfVerbose in a function

* refactor UnusedCompiledLibrariesRemover in a function

* refactor WarnAboutArchIncompatibleLibraries in a function

* refactor CreateMakeRule in a function

* move include finder with regex tests under detector

* move jobs properites in arduino/builder component

* remove sketch property from context

* apply CR suggestion

* Slightly de-entangled makeSourceFile

* Moved some initializations directly into NewBuilder

* Pruned constants

* Pruned more constants

* Fixed lint warnings

* remove unused LibraryDir from legacy context

* remove unused WatchedLocation from legacy context

* remove unused IgnoreSketchFolderNameErrors from legacy context

* remove CanUseCachedTools from legacy context

* remove UseArduinoPreprocessor from legacy context

* make the CoreBuilder command a function

* remove the use of context from builder_utils

* mvoe types.ProgressStruct in a dedicated pkg

* move ExecCommand under arduino/utils

* move LogIfVerbose from utils to legacy builder

* move some legacy constans in builder package

* move builder_utils under arduino/builder/utils pkg

* appease golint

* move coreBuildCachePath in the arduino Builder

* refactor Linker command in a function

* refactor SketchBuilder in a function

* refactor LibrariesBuilder in a function

* refactor Sizer in a function

* remove empty file

* remove unused struct FailIfBuildPathEqualsSketchPath

* Ported TestCoreCaching to current integration test infra

* Removed useless TestBuilderEmptySketch

The same features are already tested in a number of other integration tests.

* Ported TestBuilderWithBuildPathInSketchDir to current integration test infra

* Ported TestLoadHardware to current integration test infra

* Moved user_hardware testdata resources

* Ported TestLoadHardwareMixingUserHardwareFolder to current integration test infra

* Removed redundant legacy tests

* Removed useless 'sleep' helper function

* Removed unused LoadAndInterpolate function

* Removed useless NoError helper function

* Grouped tests inside the same correct sub-group

* Ported TestFailIfBuildPathEqualsSketchPath and TestFailIfBuildPathEqualsSketchPathSketchPathDiffers to current integration test infra

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.32 to 3.1.34.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.32...3.1.34)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: direct:development
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Simplified compileFiles

* Starting new Builder (part 1/n)

* builder: renamed variable and moved dir creation up

* builder: made a sketch-prepress function part of Builder

* Removed ctx dependency in PreprocessSketchWithArduinoPreprocessor

* uniform parameters between preprocesors

* Moved PreprocessSketchWithArduinoPreprocessor into proper place

* Inlined function

* Converted sketchCopyAdditionalFiles into a Builder method

* Made SetupBuildProperties a method of the new Builder

* Refactor AddAdditionalEntriesToContext

https://github.com/cmaglie/arduino-cli/pull/32

* refactor AddAdditionalEntriesToContext in a function

* use the new function in all the tests

* Move the assignaton of LibrariesResolutionResults inside the ResolveLibrary func

The ResolveLibrary func is only called by the ContainerFindIncludes

* rename bPath to buildPath

* cleanup usless tests

* remove shadowed variable

* Refactor legacy LibrariesLoader command

* move LibrariesLoader under arduino/builder

* remove usless nil check

* remove AddAdditionalEntries func, in favour of initializing them in the compile command

* move a check directly in the compile command

* create the SketchLibrariesDetector struct

* move all the logic of ContainerSetupHardwareToolsLibsSketchAndProps in the compile command

* remove container_setup and adjust relative tests

* remove LibraryResolver property from context

* remove UseCachedLibrariesResolution for context

* remove ImportedLibraries from context

* remove LibrariesResolutionResults from context

* remove LibrariesManager from context

* fix regression exceeding 100% status bar

* refactor find_includes

* refactoring the cmd.Exec in favour of executils

* use detector FindIncludes in tests

* add comments and make some plubic methods private

---------
Co-authored-by: Alessio Perugini <alessioper.98@gmail.com>

* Change GetPort's returned type to rpc.Port

* Use LoadSketch in runUploadCommand

* Removed useless tty abstraction

* Print 'Connected to...' message immediatly after connection

* Added terminal raw mode

* Make local the argument/flags of 'monitor' command

* Applied code suggestion

* Ported legacy TestIncludesToIncludeFolders test

* Ported legacy TestIncludesToIncludeFoldersANewLibrary test

* Ported legacy TestIncludesToIncludeFoldersDuplicateLibs* test

Also improved build options in test

Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Simplified table.Render method

* Removed right padding from tables

* Moved out from legacy some TryBuild* integration tests

* Ported sketch_with_config integration tests out of legacy

* Ported sketch_no_functions integration tests out of legacy

* Ported sketch_with_backup_files integration tests out of legacy

* Ported sketch_with_old_lib integration tests out of legacy

* Ported sketch_with_subfolders integration tests out of legacy

* Ported sketch_with_class integration tests out of legacy

* Simplified integrationtest of preprocessor

* Ported sketch_with_typename integration tests out of legacy

* Ported sketch_with_namespace integration tests out of legacy

* Ported sketch_with_default_args integration tests out of legacy

* Ported sketch_with_inline_function_args integration tests out of legacy

* Ported sketch_with_function_signature_inside_ifdef integration tests out of legacy

* Ported sketch_with_usbcon integration tests out of legacy

* Ported sketch_with_const integration tests out of legacy

* Ported sketch_with_templates_and_shift integration tests out of legacy

* Ported sketch eol_processing integration tests out of legacy

* Ported SketchWithIfdef integration tests out of legacy

* Ported sketch_with_ifdef integration tests out of legacy

* Ported Bridge integration tests out of legacy

* Ported more Bridge integration tests out of legacy

* Ported even more Bridge integration tests out of legacy

* Better subtesting categorization

* Ported yet another Bridge integration tests out of legacy

* Ported Balanduino integration tests out of legacy

* Ported CharWithEscapedDoubleQuote integration tests out of legacy

* Ported IncludeBetweenMultilineComment integration tests out of legacy

* Ported LineContinuations integration tests out of legacy

* Ported StringWithComment integration tests out of legacy

* Ported SketchWithStruct integration tests out of legacy

* Ported SketchNoFunctionsTwoFiles integration tests out of legacy

* Ported SketchWithClassAndMethodSubstring integration tests out of legacy

* Ported SketchThatChecksIfSPIHasTransactions integration tests out of legacy

* Ported sketch_with_dependend_libraries integration tests out of legacy

* Ported sketch_with_function_pointer integration tests out of legacy

* Ported sketch_usbhost integration tests out of legacy

* Removed sketch1 from try_build* test

It's already compiled many times in other tests.

* Ported sketch9 integration tests out of legacy

* Removing no more used functions

* Ported USBHost test

* Removed no more useful test from legacy

* Removed no more needed tests

* Replaced copyRecursive with go-paths library calls

* Simplified download functions

* Removed no more needed functions

* Replaced some calls with go-paths library

* Replaced some calls with go-paths library

* Isolated some function into the (soon-to-be-removed) CMake generator

This change will not break the CMake generator but will remove the
CopyDir implementation from the public API space.

* Removed now dead code

* Moved md5 function to proper place

* Refactor findAllFilesInFolder to use more go-paths function

* Refactor findAllFilesInFolder to use more go-paths function

* Made FilesAreOlderThan function to rationalize code

* Yet another code-factorization of findAllFilesInFolder

* Further simplified version of FilesAreOlderThan

* Inlined CoreOrReferencedCoreHasChanged function

* Inlined TXTBuildRulesHaveChanged function

I see an increase in trainsient CI failures for this test. Let's try
to increase the timeout to see if solves the problem.

* Transformed FailIfImportedLibraryIsWrong into a functon

There was no need to have it encapsulated in a Command

* Move variable near the place it belongs

* Dramatically simplified SourceFile object

Removed dependency from types.Context

* Added comments about utility folder role in library discovery

* Simplified if construct

* Made RelativePath private

* Skip includes detection of precompiled libraries early

Instead of skipping include detection later, avoid to add the sources in
the queue right from the beginning.

* Keep extra-include dirs due to "utility" folder in SourceFile object

Also remove the reference to the original Library object because it's no
more needed.

* Add support for default profile to compile command

* Add support for default profiles to upload command

* Add TestCompileWithDefaultProfile to integration tests

* Get the profile's FQBN if it's not already specified in the request

* Update documentation regarding sketch projects

* Added integration tests for all default_profile cases

* Reverted old sketch_with_profile test

---------
Co-authored-by: Cristian Maglie <c.maglie@arduino.cc>

* Allow .cxx and .cc extensions

* Added integration test

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Add informations regarding profiles to LoadSketchResponse

* Load profiles when LoadSketch is called

* Add test to verify that LoadSketchResponse contains the profiles

* UploadResponse now has 'oneof' clause for better API design

* Added scaffolding to return updated-port after upload

* Upload port change detection (first draft)

* Simplified port detection using a Future-style abstraction

* Perform watcher-flush higher in the call tree

* Do not infer upload port if 'upload.wait_for_upload_port' is false

* Further simplified port detection subroutine structure

* fixed linter issue

* Always return an updatedUploadPort.

Arduino CLI should always return the port after an upload, even in the case
where no port change is expected. The consumer shouldn't be required to
implement "if not updated_upload_port, use original port" logic.

The whole point is that all the logic for determining which port should be
selected after an upload should be implemented in Arduino CLI. The consumer
should be able to simply select the port Arduino CLI tells it to select in
all cases.

* Updated docs

* Perform a deep-copy of upload ports where needed.

Previously only the pointer was copied, thus making changes in
`actualPort` to be reflected also to `port`. This lead to some weird
result in the `updatedUploadPort` result:

{
  "stdout": "Verify 11344 bytes of flash with checksum.\nVerify successful\ndone in 0.010 seconds\nCPU reset.\n",
  "stderr": "",
  "updated_upload_port": {
    "address": "/dev/tty.usbmodem14101",     <------- this address...
    "label": "/dev/cu.usbmodem14101",        <------- ...is different from the label
    "protocol": "serial",
    "protocol_label": "Serial Port (USB)",
    "properties": {
      "pid": "0x804E",
      "serialNumber": "94A3397C5150435437202020FF150838",
      "vid": "0x2341"
    },
    "hardware_id": "94A3397C5150435437202020FF150838"
  }
}

* When updating `actualPort` address, update also the address label.

* Fixed some potential nil pointer exceptions

* Further simplified board watcher

We must acesss the gRPC API only until we cross the `command` package
border. Once we are inside the `command` package we should use the
internal API only.

* Before returning from upload, check if the port is still alive

Now the upload detects cases when the upload port is "unstable", i.e.
the port changes even if it shouldn't (because the wait_for_upload_port
property in boards.txt is set to false).

This change should make the upload process more resilient.

* Apply suggestions from code review
Co-authored-by: per1234 <accounts@perglass.com>

* Fixed nil exception

* Improved tracking algorithm for upload-port reconnection

The new algorithm takes into account the case where a single board may
expose multiple ports, in this case the selection will increase priority
to ports that:

  1. have the same HW id as the user specified port for upload
  2. have the same protocol as the user specified port for upload
  3. have the same address as the user specified port for upload

---------
Co-authored-by: per1234 <accounts@perglass.com>

* regression: hide include-not-found errors during library discovery

This regression was made during a refactoring of the Arduino preprocessor.
In particular the wrong change was part of this commit:
https://github.com/arduino/arduino-cli/commit/0585435f8b1d05e0117606f69bea42d3f3dfec79#diff-65ff16cbee816c0f443157444d99bcc144beee06c3329aec891894c8aeda7b27L372-R378

-                       preproc_stderr, preproc_err = GCCPreprocRunner(ctx, sourcePath, targetFilePath, includes)
+                       var preproc_stdout []byte
+                       preproc_stdout, preproc_stderr, preproc_err = preprocessor.GCC(sourcePath, targetFilePath, includes, ctx.BuildProperties)
+                       if ctx.Verbose {
+                               ctx.WriteStdout(preproc_stdout)
+                               ctx.WriteStdout(preproc_stderr)
+                       }

Previously GCCPreprocRunner, in verbose modem will show ONLY the stdout of
the process, instead the "refactored" code wrongly added also stderr to the
output.

For reference this is the old GCCPreprocRunner implementation:
https://github.com/arduino/arduino-cli/commit/0585435f8b1d05e0117606f69bea42d3f3dfec79#diff-371f93465ca5a66f01cbe876348f67990750091d27a827781c8633456b93ef3bL36

-func GCCPreprocRunner(ctx *types.Context, sourceFilePath *paths.Path, targetFilePath *paths.Path, includes paths.PathList) ([]byte, error) {
-       cmd, err := prepareGCCPreprocRecipeProperties(ctx, sourceFilePath, targetFilePath, includes)
-       if err != nil {
-               return nil, err
-       }
-       _, stderr, err := utils.ExecCommand(ctx, cmd, utils.ShowIfVerbose /* stdout */, utils.Capture /* stderr */)
-       return stderr, err
-}

This commit fixes the regression.

* Added integration test

Previously the undefined template placeholders in "recipe.preproc.macros"
were silently replaced the empty string. This changed after a
refactoring in 0585435f.

Previously it was:

https://github.com/arduino/arduino-cli/commit/0585435f8b1d05e0117606f69bea42d3f3dfec79#diff-371f93465ca5a66f01cbe876348f67990750091d27a827781c8633456b93ef3bL62
-       cmd, err := builder_utils.PrepareCommandForRecipe(buildProperties, "recipe.preproc.macros", true, ctx.PackageManager.GetEnvVarsForSpawnedProcess())

The `true` parameter in the call to `builder_utils.PrepareCommandForRecipe` is the parameter `removeUnsetProperties`.

This behaviour has not been ported over after the "refactoring":
https://github.com/arduino/arduino-cli/commit/0585435f8b1d05e0117606f69bea42d3f3dfec79#diff-733dda6759fe968eb8a8d7305c37c7a320a8df52764ca0cba8e88a6f1d077eb5R44-R65

+       const gccPreprocRecipeProperty = "recipe.preproc.macros"
+       if gccBuildProperties.Get(gccPreprocRecipeProperty) == "" {
+               // autogenerate preprocess macros recipe from compile recipe
+               preprocPattern := gccBuildProperties.Get("recipe.cpp.o.pattern")
+               // add {preproc.macros.flags} to {compiler.cpp.flags}
+               preprocPattern = strings.Replace(preprocPattern, "{compiler.cpp.flags}", "{compiler.cpp.flags} {preproc.macros.flags}", 1)
+               // replace "{object_file}" with "{preprocessed_file_path}"
+               preprocPattern = strings.Replace(preprocPattern, "{object_file}", "{preprocessed_file_path}", 1)
+
+               gccBuildProperties.Set(gccPreprocRecipeProperty, preprocPattern)
+       }
+
+       pattern := gccBuildProperties.Get(gccPreprocRecipeProperty)
+       if pattern == "" {
+               return nil, nil, errors.Errorf(tr("%s pattern is missing"), gccPreprocRecipeProperty)
+       }
+
+       commandLine := gccBuildProperties.ExpandPropsInString(pattern)
+       args, err := properties.SplitQuotedString(commandLine, `"'`, false)
+       if err != nil {
+               return nil, nil, err
+       }

that is missing the call to `properties.DeleteUnexpandedPropsFromString`.

* Fixed loading of Library given a non-absolute path

Givin a relative path is not ideal, but at least allows the CLI to
compile the sketch correctly.

* Added test

* docs: Clearifiesd how tools flavours are selected

* Update docs/package_index_json-specification.md
Co-authored-by: Alessio Perugini <alessioper.98@gmail.com>

---------
Co-authored-by: Alessio Perugini <alessioper.98@gmail.com>

Bumps [arduino/setup-protoc](https://github.com/arduino/setup-protoc) from 1 to 2.
- [Release notes](https://github.com/arduino/setup-protoc/releases)
- [Commits](https://github.com/arduino/setup-protoc/compare/v1.0.0...v2)

---
updated-dependencies:
- dependency-name: arduino/setup-protoc
  dependency-type: direct:production
  update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.32.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.32)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: direct:development
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Added FAILED_INSTANCE_INIT_REASON_INDEX_DOWNLOAD_ERROR in gRPC Init errors

* Improved error reporting during Init and first-index-update