- ApiResponse: add readToBuffer option to defer/stream body instead of eagerly buffering
- TableDataProvider: implement HTML table parser with per-column provider support
- StealthConfig: add 10s page load timeout and copyCookiesFrom parameter for cookie sharing
- StealthUnitDownloader: catch WebDriverTimeoutException on navigation, log warning instead of throwing
- Bump version to 2.9.0
Added JetBrains Rider IDE configuration files and a backup for Beam.Api.csproj. Updated aeqw89.Beam project version to 2.7.0 and package references, including Selenium.WebDriver and System.IO.Hashing. Enhanced RelationalDataProvider to support NextSibling and PreviousSibling relations and configurable traversal distance.
Introduces a FollowRedirects property to UnitDownloaderOptions and its builder, allowing control over HTTP redirect behavior. Updates UnitDownloader to use this option, following redirects when enabled and reporting progress accordingly.
Introduces the Addon record to represent browser addons and updates StealthConfig to support loading multiple utility addons per browser. The Firefox driver now installs specified addons from the UtilityAddons array, improving extensibility for browser automation.
Upgrades Selenium.WebDriver to 4.38.0 and adds logic to automatically install the uBlock extension for FirefoxDriver instances. The uBlock extension file is now included in the project and set to copy to output. Warnings are logged if the extension fails to load.
Replaces generic RawType with ByteDocument in downloaders and context classes, simplifying type usage. Adds builder classes for FailurePredicateOptions, FragmentOptions, SkipPredicateOptions, and UnitDownloaderOptions to improve configuration flexibility. Introduces DownloadTarget enum and SkipPredicate delegate for more granular download control. Refactors Fluent API interfaces and implementations to remove RawType generics and streamline usage. Adds Playwright and Stealth download strategies for extensibility.
- Added `AnchorDataProvider`, `AnchorCollectionDataProvider`, `ContentsDataProvider`, `ContentsArrayDataProvider`, `DropDownDataProvider`, `ListContentDataProvider`, and `ParagraphedContentDataProvider` for enhanced data extraction flexibility.
- Updated project version to 2.5.0.
- Added `RemoteAddress` property to `StealthConfig` for remote WebDriver support.
- Refactored browser driver creation logic with `DriverDefinition` for enhanced consistency.
- Improved error handling in browser fallback mechanism.
- Incremented project version to 2.4.6.
- Removed obsolete data providers: `AnchorCollectionDataProvider`, `ContentsDataProvider`, and others, consolidating logic into new composable providers.
- Added `ComposeDataProviders`, `SelectDataProvider`, and `RelationalDataProvider` for improved flexibility and reusability.
- Introduced `IManySelectionComposableDataProvider` interface to support multiple-node selection.
- Enhanced `UnitDownloader` with more robust progress tracking.
- Updated package references and project dependencies for consistency.
- Improved error handling in `StealthConfig` initialization for better fallback on browser drivers.
- Incremented project version to 2.4.5.
Replaces specialized binary and HTML downloaders with a generic, options-driven UnitDownloader and UnitFragmentDownloader pattern. Introduces UnitDownloaderOptions and builder classes for flexible configuration, updates interfaces and method signatures to support progress reporting, and removes redundant binary-specific classes. Updates Playwright and Stealth downloaders to use the new generic base, and adds improved error handling and reporting. Also updates dependency versions and makes minor API consistency improvements across the Fluent and Models layers.
Marked UrlLocation properties as required in ResourceDefinition for improved null safety. Changed OrderedLinkGenerator to use the more general IStateChangeBehaviour instead of NumberedStateChanger, increasing flexibility.
- Bumped Microsoft.Extensions.Logging packages to version 9.0.9 across all projects.
- Updated aeqw89.Beam project version to 2.1.4.
- Added new transitive package references, including Microsoft.Recognizers.Text.Number, Microsoft.Playwright, EntityFramework, and others.
- Commented out or removed Beam.Temporary.Cli references.
- Enhanced package structure by rearranging content includes and cleaning up redundant package references.
- Introduced modularity by splitting Beam into new projects: Beam.Abstractions, Beam.Models, and Beam.Downloaders.
- Refactored existing classes into appropriate namespaces and projects.
- Replaced specific implementations with abstractions (e.g., SourceLinkBuilder to LinkBuilder, State to IState, etc.).
- Updated interfaces: added ITemplate, IArticleData, IDownloadReport, and others for improved extensibility.
- Removed deprecated classes like SourceLinkBuilder and StateChangerFactory.
- Enhanced link handling in downloaders by refactoring to use `string` over `SourceLink`.
- Consolidated shared logic under Beam.Abstractions.
Added new Beam.Fluent and Beam.Models projects with staged download builder and data context models. Refactored and moved model classes from Beam.Temporary.Cli to Beam.Models. Added new data providers and extended DataBindings in Beam.Dynamic. Renamed Beam.Puppeteer to Beam.Playwright and updated related classes. Updated project references and package versions. Removed obsolete and unused files from Beam.Temporary.Cli.
- Removed BindingType enum and all related logic from Binding.
- Made Binding implement new IBinding and IKeyed interfaces.
- Moved node selection logic to IBinding.Select; removed Resolve* methods from Binding.
- Added new IBinding interface for XPath/CssPath selection.
- Refactored IDataProvider to generic IDataProvider<T>; removed GetNode.
- Updated ListContentDataProvider and ParagraphedContentDataProvider to use IBinding.
- Added new ContentsDataProvider, ContentsArrayDataProvider, and DropDownDataProvider for flexible data extraction.
- Updated DataBindings to use IDataProvider<T> properties instead of Binding.
- Updated all usages to new interfaces and patterns.
The DownloadEnumerable class has been refactored to accept
an IAsyncEnumerable<Ordered<T>> instead of an IAsyncEnumerator<Ordered<T>>.
This change simplifies the class and improves its usability.
This allows for better integration with asynchronous streaming
of data, enhancing performance and flexibility.
Added a static `Empty` method to `DownloadEnumerable` for
creating an empty instance. Updated link handling in
`SequentialDownloader` to use `AbsoluteUri` instead of
`ToString()`, ensuring correct link representation.
These changes improve usability and ensure consistency in
link formatting.
feat: add PuppetConfig and integrate with CLI
Introduced a new PuppetConfig class in the Puppeteer
namespace to manage Puppeteer configurations. Updated
the CLI project to reference the Puppeteer project and
added a new method in DownloadBuilder for using a
Puppet manipulator.
This change enables better configuration management
for Puppeteer within the CLI.
```
This introduces a new Puppeteer-based mechanism for downloading
web content. It provides a flexible way to manipulate pages
during downloads, enhancing the ability to handle dynamic
content and improve the overall download process.
Refactor the transformers in the downloader classes to use
ByteDocument instead of byte arrays. This change improves type
safety and clarity in handling document content during
downloads, ensuring that the transformations are more
consistent and maintainable.
Updated project files for `Beam.Dynamic`, `Beam.Exports`, `Beam.Puppeteer`, `Beam.Temporary.Cli`, and `Beam` to include new XML headers, reorganize property groups, and add project references.
Modified `PuppetedUnitDownloader` to support additional parameters for async transformers. Changed return types in `CommonTransformers` to `AsyncTransformer` for asynchronous processing.
Significant refactoring in `DownloadBuilder`, `DownloadContext`, and `DownloadContextBuilder` to introduce generic parameters and improve context management. Updated `SequentialDownloader`, `SequentialFragmentDownloader`, and `UnitDownloader` to accommodate new async transformer types.
Introduced `TypeExtensions` for unique type name generation and added `UnitFragmentDownloaderBinary` for handling binary downloads. Updated solution file to include the new `aeqw89.Beam` project, ensuring proper references across the solution.
These changes enhance the asynchronous capabilities of the Beam library, improve type safety, and streamline the downloading process.
Updated project files for `Beam.Dynamic`, `Beam.Exports`, `Beam.Temporary.Cli`, and `Beam` to include additional metadata and specific package versions. Refactored `DataBindings` and `ResolvedBindings` to records, added a new `Text` property in `Binding.cs`, and introduced `ParseNumbers` in `OnlineCleaner`. New classes `PuppetContext` and `PuppetUnitDownloader` added for Playwright integration. Introduced `ImmutableState` struct and `UnitDownloaderBinary` class for improved download management. Updated tests in `UnitTest1.cs` for number localization. Added `Beam.Puppeteer` project to the solution.
qwsdcvghyu89
riley