Compare commits

...

36 Commits

Author SHA1 Message Date
74cbfd945b Merge pull request #8494 from wallabag/release/2.6.14
Prepare 2.6.14
2025-10-07 10:05:47 +02:00
de107deed2 Prepare 2.6.14 2025-10-07 09:49:06 +02:00
7108eea2da Merge pull request #8488 from wallabag/fix/doc-urls-api 2025-10-06 06:05:45 +02:00
7aadb8eaa8 Merge pull request #8332 from andreadecorte/fix-reading-time-computation 2025-10-06 06:05:31 +02:00
f2a3968a2f Merge pull request #8489 from wallabag/fix/bump-deps 2025-10-06 06:04:56 +02:00
19c0b9c800 Bump deps (mostly for siteconfig) 2025-10-02 12:19:53 +02:00
1a69855090 Fix urls parameter when sending many urls to be stored using the API 2025-10-02 10:23:14 +02:00
797d48905c Extract DEFAULT_WORDS_PER_MINUTE in Utils
This is done In order to avoid having magic numbers in the code
2025-09-07 22:32:55 +02:00
d24b703315 Extract user reading time logic in a single place
Extract user reading time computation in a single place under Entry to avoid duplication.

This will also fix reading time computation for short entries.

RuleTagger logic is slightly changed as we will round when filtering, which will be
consistent with frontend display.
2025-09-07 22:27:08 +02:00
1eca3cf327 Merge pull request #8440 from wallabag/fix-docker-base-image
Fix docker base image
2025-08-25 08:55:19 +02:00
91384c531d Fix docker base image 2025-08-24 23:47:18 +02:00
a86b16d679 Merge pull request #8435 from wallabag/update-dependencies
Update dependencies
2025-08-20 15:11:25 +02:00
2bfc3fd852 Update dependencies 2025-08-20 13:04:24 +02:00
653b198f8e Merge pull request #8346 from skn/2.6
Add annotations filter to entries API endpoint
2025-07-11 15:08:08 +02:00
5c5b20c83b Add annotations filter to entries API endpoint
Implement a new filter parameter 'annotations' for the GET /api/entries endpoint
that allows filtering entries based on whether they have annotations. When
annotations=1, only entries with one or more annotations are returned. When
annotations=0, only entries without annotations are returned. This feature
enables users to easily find annotated content through the API.
2025-07-02 22:24:29 +04:00
d00fe83366 Merge pull request #8267 from wallabag/fix/2.6-deprecation
Fix deprecation
2025-06-05 16:32:29 +02:00
f0d3db70c0 Fix deprecation
See https://php.watch/versions/8.2/$%7Bvar%7D-string-interpolation-deprecated
2025-06-05 15:54:51 +02:00
4f34cfa6fc Merge pull request #8251 from wallabag/change-version
Change version in wallabag.yml
2025-06-04 18:48:59 +02:00
8e90c0f320 Change version in wallabag.yml 2025-06-04 17:41:09 +02:00
6ffd7382c7 Merge pull request #8229 from wallabag/prepare-2613
Prepare 2.6.13 release
2025-06-04 17:29:19 +02:00
46d6e4d923 Update CHANGELOG 2025-06-04 13:59:00 +02:00
09af127446 Unlock php-readability 2025-06-04 13:59:00 +02:00
4c23196304 Prepare 2.6.13 release 2025-06-04 13:59:00 +02:00
42746f418e Merge pull request #8250 from wallabag/update-site-config
Update joker/graby-site-config
2025-06-04 13:54:23 +02:00
35c4feedd8 Update joker/graby-site-config 2025-06-04 13:42:08 +02:00
c451cc96e5 Merge pull request #8240 from wallabag/import/pocket-csv
Add support of Pocket CSV import
2025-06-03 13:32:51 +02:00
c4240c866b PocketCsvImport: rework test to ensure we import all entries
This also adds a test for #8217

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2025-06-03 13:24:27 +02:00
27d66d9e1d Add 'application/vnd.ms-excel' to allowed MIME types config
Backport #8036
2025-06-03 13:24:27 +02:00
c1397f43ac Add Pocket CSV import 2025-06-03 13:24:27 +02:00
52a16bb75f Merge pull request #8193 from wallabag/add-pocket-shaarli-import
Backport Pocket and Shaarli imports from master
2025-06-02 14:23:33 +02:00
f82c87b520 import: fix some tests
Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2025-06-02 13:42:44 +02:00
772a802596 Merge pull request #8194 from wallabag/fix-8158
Add j0k3r/php-readability:1.2.10 as a dependency
2025-06-01 16:51:35 +02:00
29162bde9d Pin j0k3r/php-readability to 1.2.10
We pin php-readability to 1.2.10 because of a regression in 1.2.12

Workaround for #8151

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2025-06-01 16:37:22 +02:00
b1614e9267 Add Pocket and Shaarli imports 2025-05-24 16:25:48 +02:00
70999075a6 Merge pull request #8139 from wallabag/fix/otp-wrongly-enabled
Avoid non-validated OTP to be enabled
2025-04-14 09:46:17 +02:00
262f674245 Avoid non-validated OTP to be enabled
The OTP code must be required when enabling OTP. If the provided code is wrong, disable OTP, redirect and notice the user.
2025-04-14 09:18:37 +02:00
50 changed files with 3085 additions and 418 deletions

View File

@ -1,12 +1,48 @@
# Changelog
## [2.6.14](https://github.com/wallabag/wallabag/tree/2.6.14)
[Full Changelog](https://github.com/wallabag/wallabag/compare/2.6.13...2.6.14)
### Improvements
* Add annotations filter to entries API endpoint by @skn in https://github.com/wallabag/wallabag/pull/8346
### Fixes
* Fix reading time computation for short entries by @andreadecorte in https://github.com/wallabag/wallabag/pull/8332
* Fix `urls` parameter when sending many urls to be stored using the API by @j0k3r in https://github.com/wallabag/wallabag/pull/8488
* Fix docker base image by @yguedidi in https://github.com/wallabag/wallabag/pull/8440
### Technical stuff
* Change version in wallabag.yml by @nicosomb in https://github.com/wallabag/wallabag/pull/8251
* Fix deprecation by @j0k3r in https://github.com/wallabag/wallabag/pull/8267
* Update dependencies by @yguedidi in https://github.com/wallabag/wallabag/pull/8435
* Bump deps (mostly for siteconfig) by @j0k3r in https://github.com/wallabag/wallabag/pull/8489
## [2.6.13](https://github.com/wallabag/wallabag/tree/2.6.13)
[Full Changelog](https://github.com/wallabag/wallabag/compare/2.6.12...2.6.13)
### Improvements
* Add support of Pocket CSV import by @kdecherf and @nicosomb in https://github.com/wallabag/wallabag/pull/8240
* Backport Pocket and Shaarli HTML imports from master by @nicosomb in https://github.com/wallabag/wallabag/pull/8193
### Fixes
* Avoid non-validated OTP to be enabled #8139 by @j0k3r in https://github.com/wallabag/wallabag/pull/8139
### Technical stuff
* Update j0k3r/php-readability:1.2.13 to fix regression (about latin1 instead of UTF-8 used for entries) by @nicosomb https://github.com/wallabag/wallabag/pull/8194
## [2.6.12](https://github.com/wallabag/wallabag/tree/2.6.12)
[Full Changelog](https://github.com/wallabag/wallabag/compare/2.6.11...2.6.12)
### Technical stuff
* Fix changelog by @yguedidi in [https://github.com/wallabag/wallabag/pull/8135](https://github.com/wallabag/wallabag/pull/8135)
* Update dependencies by @yguedidi in [https://github.com/wallabag/wallabag/pull/8136](https://github.com/wallabag/wallabag/pull/8136)
* Fix changelog by @yguedidi in https://github.com/wallabag/wallabag/pull/8135
* Update dependencies by @yguedidi in https://github.com/wallabag/wallabag/pull/8136
## [2.6.11](https://github.com/wallabag/wallabag/tree/2.6.11)
[Full Changelog](https://github.com/wallabag/wallabag/compare/2.6.10...2.6.11)
@ -16,14 +52,14 @@
### Fixes
* Fix redirection after action in search results by @nicosomb in [https://github.com/wallabag/wallabag/pull/7827](https://github.com/wallabag/wallabag/pull/7827)
* Fix title tag filter by @nicosomb in [https://github.com/wallabag/wallabag/pull/7846](https://github.com/wallabag/wallabag/pull/7846)
* Change NB_ELEMENTS in pocket importer to 30 by @j0k3r in [https://github.com/wallabag/wallabag/pull/7993](https://github.com/wallabag/wallabag/pull/7993)
* Fix entries counter for annotated entries in the menu by @j0k3r in [https://github.com/wallabag/wallabag/pull/7999](https://github.com/wallabag/wallabag/pull/7999)
* Fix redirection after action in search results by @nicosomb in https://github.com/wallabag/wallabag/pull/7827
* Fix title tag filter by @nicosomb in https://github.com/wallabag/wallabag/pull/7846
* Change NB_ELEMENTS in pocket importer to 30 by @j0k3r in https://github.com/wallabag/wallabag/pull/7993
* Fix entries counter for annotated entries in the menu by @j0k3r in https://github.com/wallabag/wallabag/pull/7999
### Technical stuff
* Prepare 2.6.11 release by @yguedidi in [https://github.com/wallabag/wallabag/pull/8133](https://github.com/wallabag/wallabag/pull/8133)
* Prepare 2.6.11 release by @yguedidi in https://github.com/wallabag/wallabag/pull/8133
## [2.6.10](https://github.com/wallabag/wallabag/tree/2.6.10)
[Full Changelog](https://github.com/wallabag/wallabag/compare/2.6.9...2.6.10)

View File

@ -27,12 +27,12 @@ class Version20170511211659 extends WallabagMigration
$this->addSql(<<<EOD
CREATE TEMPORARY TABLE __temp__wallabag_annotation AS
SELECT id, user_id, entry_id, text, created_at, updated_at, quote, ranges
FROM ${annotationTableName}
FROM {$annotationTableName}
EOD
);
$this->addSql('DROP TABLE ' . $annotationTableName);
$this->addSql(<<<EOD
CREATE TABLE ${annotationTableName}
CREATE TABLE {$annotationTableName}
(
id INTEGER PRIMARY KEY NOT NULL,
user_id INTEGER DEFAULT NULL,
@ -42,16 +42,16 @@ CREATE TABLE ${annotationTableName}
updated_at DATETIME NOT NULL,
quote CLOB NOT NULL,
ranges CLOB NOT NULL,
CONSTRAINT FK_A7AED006A76ED395 FOREIGN KEY (user_id) REFERENCES ${userTableName} (id),
CONSTRAINT FK_A7AED006BA364942 FOREIGN KEY (entry_id) REFERENCES ${entryTableName} (id) ON DELETE CASCADE
CONSTRAINT FK_A7AED006A76ED395 FOREIGN KEY (user_id) REFERENCES {$userTableName} (id),
CONSTRAINT FK_A7AED006BA364942 FOREIGN KEY (entry_id) REFERENCES {$entryTableName} (id) ON DELETE CASCADE
);
CREATE INDEX IDX_A7AED006A76ED395 ON ${annotationTableName} (user_id);
CREATE INDEX IDX_A7AED006BA364942 ON ${annotationTableName} (entry_id);
CREATE INDEX IDX_A7AED006A76ED395 ON {$annotationTableName} (user_id);
CREATE INDEX IDX_A7AED006BA364942 ON {$annotationTableName} (entry_id);
EOD
);
$this->addSql(<<<EOD
INSERT INTO ${annotationTableName} (id, user_id, entry_id, text, created_at, updated_at, quote, ranges)
INSERT INTO {$annotationTableName} (id, user_id, entry_id, text, created_at, updated_at, quote, ranges)
SELECT id, user_id, entry_id, text, created_at, updated_at, quote, ranges
FROM __temp__wallabag_annotation;
EOD

View File

@ -283,6 +283,21 @@ old_sound_rabbit_mq:
exchange_options:
name: 'wallabag.import.chrome'
type: topic
import_shaarli:
connection: default
exchange_options:
name: 'wallabag.import.shaarli'
type: topic
import_pocket_html:
connection: default
exchange_options:
name: 'wallabag.import.pocket_html'
type: topic
import_pocket_csv:
connection: default
exchange_options:
name: 'wallabag.import.pocket_csv'
type: topic
consumers:
import_pocket:
connection: default
@ -383,6 +398,33 @@ old_sound_rabbit_mq:
name: 'wallabag.import.chrome'
callback: wallabag_import.consumer.amqp.chrome
qos_options: {prefetch_count: "%rabbitmq_prefetch_count%"}
import_shaarli:
connection: default
exchange_options:
name: 'wallabag.import.shaarli'
type: topic
queue_options:
name: 'wallabag.import.shaarli'
callback: wallabag_import.consumer.amqp.shaarli
qos_options: {prefetch_count: "%rabbitmq_prefetch_count%"}
import_pocket_html:
connection: default
exchange_options:
name: 'wallabag.import.pocket_html'
type: topic
queue_options:
name: 'wallabag.import.pocket_html'
callback: wallabag_import.consumer.amqp.pocket_html
qos_options: {prefetch_count: "%rabbitmq_prefetch_count%"}
import_pocket_csv:
connection: default
exchange_options:
name: 'wallabag.import.pocket_csv'
type: topic
queue_options:
name: 'wallabag.import.pocket_csv'
callback: wallabag_import.consumer.amqp.pocket_csv
qos_options: {prefetch_count: "%rabbitmq_prefetch_count%"}
fos_js_routing:
routes_to_expose:

View File

@ -121,6 +121,21 @@ services:
$rabbitMqProducer: '@old_sound_rabbit_mq.import_wallabag_v2_producer'
$redisProducer: '@wallabag_import.producer.redis.wallabag_v2'
Wallabag\ImportBundle\Controller\ShaarliController:
arguments:
$rabbitMqProducer: '@old_sound_rabbit_mq.import_shaarli_producer'
$redisProducer: '@wallabag_import.producer.redis.shaarli'
Wallabag\ImportBundle\Controller\PocketHtmlController:
arguments:
$rabbitMqProducer: '@old_sound_rabbit_mq.import_pocket_html_producer'
$redisProducer: '@wallabag_import.producer.redis.pocket_html'
Wallabag\ImportBundle\Controller\PocketCsvController:
arguments:
$rabbitMqProducer: '@old_sound_rabbit_mq.import_pocket_csv_producer'
$redisProducer: '@wallabag_import.producer.redis.pocket_csv'
Wallabag\ImportBundle\:
resource: '../../src/Wallabag/ImportBundle/*'
exclude: '../../src/Wallabag/ImportBundle/{Consumer,Controller,Redis}'
@ -394,6 +409,18 @@ services:
tags:
- { name: wallabag_import.import, alias: chrome }
Wallabag\ImportBundle\Import\ShaarliImport:
tags:
- { name: wallabag_import.import, alias: shaarli }
Wallabag\ImportBundle\Import\PocketHtmlImport:
tags:
- { name: wallabag_import.import, alias: pocket_html }
Wallabag\ImportBundle\Import\PocketCsvImport:
tags:
- { name: wallabag_import.import, alias: pocket_csv }
# to factorize the proximity and bypass translation for prev & next
pagerfanta.view.default_wallabag:
class: Pagerfanta\View\OptionableView

View File

@ -19,6 +19,9 @@ services:
$deliciousConsumer: '@old_sound_rabbit_mq.import_delicious_consumer'
$elcuratorConsumer: '@old_sound_rabbit_mq.import_elcurator_consumer'
$omnivoreConsumer: '@old_sound_rabbit_mq.import_omnivore_consumer'
$shaarliConsumer: '@old_sound_rabbit_mq.import_shaarli_consumer'
$pocketHtmlConsumer: '@old_sound_rabbit_mq.import_pocket_html_consumer'
$pocketCsvConsumer: '@old_sound_rabbit_mq.import_pocket_csv_consumer'
wallabag_import.consumer.amqp.pocket:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
@ -74,3 +77,18 @@ services:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\ChromeImport'
wallabag_import.consumer.amqp.shaarli:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\ShaarliImport'
wallabag_import.consumer.amqp.pocket_html:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\PocketHtmlImport'
wallabag_import.consumer.amqp.pocket_csv:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\PocketCsvImport'

View File

@ -180,3 +180,51 @@ services:
class: Wallabag\ImportBundle\Consumer\RedisEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\ChromeImport'
# shaarli
wallabag_import.queue.redis.shaarli:
class: Simpleue\Queue\RedisQueue
arguments:
$queueName: "wallabag.import.shaarli"
wallabag_import.producer.redis.shaarli:
class: Wallabag\ImportBundle\Redis\Producer
arguments:
- "@wallabag_import.queue.redis.shaarli"
wallabag_import.consumer.redis.shaarli:
class: Wallabag\ImportBundle\Consumer\RedisEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\ShaarliImport'
# pocket html
wallabag_import.queue.redis.pocket_html:
class: Simpleue\Queue\RedisQueue
arguments:
$queueName: "wallabag.import.pocket_html"
wallabag_import.producer.redis.pocket_html:
class: Wallabag\ImportBundle\Redis\Producer
arguments:
- "@wallabag_import.queue.redis.pocket_html"
wallabag_import.consumer.redis.pocket_html:
class: Wallabag\ImportBundle\Consumer\RedisEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\PocketHtmlImport'
# pocket csv
wallabag_import.queue.redis.pocket_csv:
class: Simpleue\Queue\RedisQueue
arguments:
$queueName: "wallabag.import.pocket_csv"
wallabag_import.producer.redis.pocket_csv:
class: Wallabag\ImportBundle\Redis\Producer
arguments:
- "@wallabag_import.queue.redis.pocket_csv"
wallabag_import.consumer.redis.pocket_csv:
class: Wallabag\ImportBundle\Consumer\RedisEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\PocketCsvImport'

View File

@ -1,5 +1,5 @@
wallabag_core:
version: 2.6.12
version: 2.6.14
paypal_url: "https://liberapay.com/wallabag/donate"
languages:
en: 'English'
@ -167,5 +167,11 @@ wallabag_core:
rule: _all ~ "https?://www\.lemonde\.fr/tiny.*"
wallabag_import:
allow_mimetypes: ['application/octet-stream', 'application/json', 'text/plain', 'text/csv']
allow_mimetypes:
- 'application/octet-stream'
- 'application/json'
- 'text/plain'
- 'text/csv'
- 'text/html'
- 'application/vnd.ms-excel'
resource_dir: "%kernel.project_dir%/web/uploads/import"

View File

@ -80,6 +80,7 @@
"html2text/html2text": "^4.3.1",
"incenteev/composer-parameter-handler": "^2.1.5",
"j0k3r/graby": "^2.4.5",
"j0k3r/php-readability": "^1.2.13",
"javibravo/simpleue": "^2.1",
"jms/serializer": "^3.29.1",
"jms/serializer-bundle": "^5.3.1",

755
composer.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,4 +1,4 @@
FROM php:8.1-fpm AS rootless
FROM php:8.1-fpm-bookworm AS rootless
ARG DEBIAN_FRONTEND=noninteractive
ARG NODE_VERSION=20

View File

@ -57,5 +57,15 @@ parameters:
-
message: "#^Method FOS\\\\UserBundle\\\\Model\\\\UserManagerInterface\\:\\:updateUser()#"
count: 6
count: 7
path: src/Wallabag/CoreBundle/Controller/ConfigController.php
-
message: "#^Call to an undefined method Wallabag\\\\ImportBundle\\\\Import\\\\ImportInterface\\:\\:setUser\\(\\)\\.$#"
count: 1
path: src/Wallabag/ImportBundle/Controller/HtmlController.php
-
message: "#^Call to an undefined method Wallabag\\\\ImportBundle\\\\Import\\\\ImportInterface\\:\\:setFilepath\\(\\)\\.$#"
count: 1
path: src/Wallabag/ImportBundle/Controller/HtmlController.php

View File

@ -269,6 +269,16 @@ class EntryRestController extends WallabagRestController
* example="example.com",
* )
* ),
* @OA\Parameter(
* name="annotations",
* in="query",
* description="filter by entries with annotations. Use 1 for entries with annotations, 0 for entries without annotations. All entries by default",
* required=false,
* @OA\Schema(
* type="integer",
* enum={"1", "0"}
* )
* ),
* @OA\Response(
* response="200",
* description="Returned when successful"
@ -294,6 +304,7 @@ class EntryRestController extends WallabagRestController
$since = $request->query->get('since', 0);
$detail = strtolower($request->query->get('detail', 'full'));
$domainName = (null === $request->query->get('domain_name')) ? '' : (string) $request->query->get('domain_name');
$hasAnnotations = (null === $request->query->get('annotations')) ? null : (bool) $request->query->get('annotations');
try {
/** @var Pagerfanta $pager */
@ -307,7 +318,8 @@ class EntryRestController extends WallabagRestController
$since,
$tags,
$detail,
$domainName
$domainName,
$hasAnnotations
);
} catch (\Exception $e) {
throw new BadRequestHttpException($e->getMessage());
@ -332,6 +344,7 @@ class EntryRestController extends WallabagRestController
'tags' => $tags,
'since' => $since,
'detail' => $detail,
'annotations' => $hasAnnotations,
],
true
)
@ -489,7 +502,7 @@ class EntryRestController extends WallabagRestController
* @OA\Parameter(
* name="urls",
* in="query",
* description="Urls (as an array) to create. A JSON array of urls [{'url': 'http://...'}, {'url': 'http://...'}]",
* description="Urls (as an array) to create. A JSON array of urls ['http://...', 'http://...']",
* required=true,
* @OA\Schema(type="string")
* ),

View File

@ -402,12 +402,14 @@ class ConfigController extends AbstractController
throw new BadRequestHttpException('Bad CSRF token.');
}
$user = $this->getUser();
$isValid = $googleAuthenticator->checkCode(
$this->getUser(),
$user,
$request->get('_auth_code')
);
if (true === $isValid) {
if ($isValid) {
$this->addFlash(
'notice',
'flashes.config.notice.otp_enabled'
@ -417,14 +419,14 @@ class ConfigController extends AbstractController
}
$this->addFlash(
'two_factor',
'scheb_two_factor.code_invalid'
'notice',
'flashes.config.notice.otp_code_invalid'
);
$this->addFlash(
'notice',
'scheb_two_factor.code_invalid'
);
$user->setGoogleAuthenticatorSecret(null);
$user->setBackupCodes(null);
$this->userManager->updateUser($user, true);
return $this->redirect($this->generateUrl('config') . '#set3');
}

View File

@ -14,6 +14,7 @@ use Symfony\Component\Validator\Constraints as Assert;
use Wallabag\AnnotationBundle\Entity\Annotation;
use Wallabag\CoreBundle\Helper\EntityTimestampsTrait;
use Wallabag\CoreBundle\Helper\UrlHasher;
use Wallabag\CoreBundle\Tools\Utils;
use Wallabag\UserBundle\Entity\User;
/**
@ -658,6 +659,14 @@ class Entry
$this->readingTime = $readingTime;
}
/**
* @return float
*/
public function getUserReadingTime()
{
return round($this->readingTime / $this->getUser()->getConfig()->getReadingSpeed() * Utils::DEFAULT_WORDS_PER_MINUTE);
}
/**
* @return string
*/

View File

@ -16,6 +16,7 @@ use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\OptionsResolver\OptionsResolver;
use Symfony\Component\Security\Core\Authentication\Token\Storage\TokenStorageInterface;
use Wallabag\CoreBundle\Repository\EntryRepository;
use Wallabag\CoreBundle\Tools\Utils;
use Wallabag\UserBundle\Entity\User;
class EntryFilterType extends AbstractType
@ -57,8 +58,8 @@ class EntryFilterType extends AbstractType
return;
}
$min = (int) ($lower * $user->getConfig()->getReadingSpeed() / 200);
$max = (int) ($upper * $user->getConfig()->getReadingSpeed() / 200);
$min = (int) ($lower * $user->getConfig()->getReadingSpeed() / Utils::DEFAULT_WORDS_PER_MINUTE);
$max = (int) ($upper * $user->getConfig()->getReadingSpeed() / Utils::DEFAULT_WORDS_PER_MINUTE);
if (null === $lower && null !== $upper) {
// only lower value is defined: query all entries with reading LOWER THAN this value

View File

@ -210,7 +210,7 @@ class EntriesExport
$publishedDate = $entry->getPublishedAt()->format('Y-m-d');
}
$readingTime = round($entry->getReadingTime() / $user->getConfig()->getReadingSpeed() * 200);
$readingTime = $entry->getUserReadingTime();
$titlepage = $content_start .
'<h1>' . $entry->getTitle() . '</h1>' .
@ -331,7 +331,7 @@ class EntriesExport
$authors = implode(',', $publishedBy);
}
$readingTime = $entry->getReadingTime() / $user->getConfig()->getReadingSpeed() * 200;
$readingTime = $entry->getUserReadingTime();
$pdf->addPage();
$html = '<h1>' . $entry->getTitle() . '</h1>' .

View File

@ -133,7 +133,7 @@ class RuleBasedTagger
private function fixEntry(Entry $entry)
{
$clonedEntry = clone $entry;
$clonedEntry->setReadingTime($entry->getReadingTime() / $entry->getUser()->getConfig()->getReadingSpeed() * 200);
$clonedEntry->setReadingTime($entry->getUserReadingTime());
return $clonedEntry;
}

View File

@ -266,14 +266,15 @@ class EntryRepository extends ServiceEntityRepository
* @param string $order
* @param int $since
* @param string $tags
* @param string $detail 'metadata' or 'full'. Include content field if 'full'
* @param string $detail 'metadata' or 'full'. Include content field if 'full'
* @param string $domainName
* @param bool $hasAnnotations
*
* @todo Breaking change: replace default detail=full by detail=metadata in a future version
*
* @return Pagerfanta
*/
public function findEntries($userId, $isArchived = null, $isStarred = null, $isPublic = null, $sort = 'created', $order = 'asc', $since = 0, $tags = '', $detail = 'full', $domainName = '')
public function findEntries($userId, $isArchived = null, $isStarred = null, $isPublic = null, $sort = 'created', $order = 'asc', $since = 0, $tags = '', $detail = 'full', $domainName = '', $hasAnnotations = null)
{
if (!\in_array(strtolower($detail), ['full', 'metadata'], true)) {
throw new \Exception('Detail "' . $detail . '" parameter is wrong, allowed: full or metadata');
@ -332,6 +333,16 @@ class EntryRepository extends ServiceEntityRepository
$qb->andWhere('e.domainName = :domainName')->setParameter('domainName', $domainName);
}
if (null !== $hasAnnotations) {
if ($hasAnnotations) {
$qb->leftJoin('e.annotations', 'a')
->andWhere('a.id IS NOT NULL');
} else {
$qb->leftJoin('e.annotations', 'a')
->andWhere('a.id IS NULL');
}
}
if (!\in_array(strtolower($order), ['asc', 'desc'], true)) {
throw new \Exception('Order "' . $order . '" parameter is wrong, allowed: asc or desc');
}

View File

@ -45,7 +45,7 @@
<div class="row">
<div class="input-field col s12">
<label for="_auth_code">{{ "auth_code"|trans({}, 'SchebTwoFactorBundle') }}</label>
<input id="_auth_code" type="text" autocomplete="off" name="_auth_code" />
<input id="_auth_code" type="text" autocomplete="off" name="_auth_code" required="required" />
</div>
</div>
</div>

View File

@ -1,7 +1,7 @@
{% set reading_time = entry.readingTime / app.user.config.readingSpeed * 200 %}
{% set reading_time = entry.userReadingTime %}
<i class="material-icons grey-text">timer</i>
{% if reading_time > 0 %}
<span>{{ 'entry.list.reading_time_minutes_short'|trans({'%readingTime%': reading_time|round}) }}</span>
<span>{{ 'entry.list.reading_time_minutes_short'|trans({'%readingTime%': reading_time}) }}</span>
{% else %}
<span>{{ 'entry.list.reading_time_less_one_minute_short'|trans|raw }}</span>
{% endif %}

View File

@ -4,6 +4,8 @@ namespace Wallabag\CoreBundle\Tools;
class Utils
{
public const DEFAULT_WORDS_PER_MINUTE = 200;
/**
* Generate a token used for Feeds.
*
@ -28,6 +30,6 @@ class Utils
*/
public static function getReadingTime($text)
{
return floor(\count(preg_split('~([^\p{L}\p{N}\']+|(\p{Han}|\p{Hiragana}|\p{Katakana}|\p{Hangul}){1,2})~u', strip_tags($text))) / 200);
return floor(\count(preg_split('~([^\p{L}\p{N}\']+|(\p{Han}|\p{Hiragana}|\p{Katakana}|\p{Hangul}){1,2})~u', strip_tags($text))) / self::DEFAULT_WORDS_PER_MINUTE);
}
}

View File

@ -13,11 +13,15 @@ use Symfony\Component\Security\Core\Authentication\Token\Storage\TokenStorageInt
use Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken;
use Wallabag\ImportBundle\Import\ChromeImport;
use Wallabag\ImportBundle\Import\DeliciousImport;
use Wallabag\ImportBundle\Import\ElcuratorImport;
use Wallabag\ImportBundle\Import\FirefoxImport;
use Wallabag\ImportBundle\Import\InstapaperImport;
use Wallabag\ImportBundle\Import\OmnivoreImport;
use Wallabag\ImportBundle\Import\PinboardImport;
use Wallabag\ImportBundle\Import\PocketCsvImport;
use Wallabag\ImportBundle\Import\PocketHtmlImport;
use Wallabag\ImportBundle\Import\ReadabilityImport;
use Wallabag\ImportBundle\Import\ShaarliImport;
use Wallabag\ImportBundle\Import\WallabagV1Import;
use Wallabag\ImportBundle\Import\WallabagV2Import;
use Wallabag\UserBundle\Entity\User;
@ -37,9 +41,29 @@ class ImportCommand extends Command
private DeliciousImport $deliciousImport;
private OmnivoreImport $omnivoreImport;
private WallabagV1Import $wallabagV1Import;
private ElcuratorImport $elcuratorImport;
private ShaarliImport $shaarliImport;
private PocketHtmlImport $pocketHtmlImport;
private PocketCsvImport $pocketCsvImport;
public function __construct(EntityManagerInterface $entityManager, TokenStorageInterface $tokenStorage, UserRepository $userRepository, WallabagV2Import $wallabagV2Import, FirefoxImport $firefoxImport, ChromeImport $chromeImport, ReadabilityImport $readabilityImport, InstapaperImport $instapaperImport, PinboardImport $pinboardImport, DeliciousImport $deliciousImport, OmnivoreImport $omnivoreImport, WallabagV1Import $wallabagV1Import)
{
public function __construct(
EntityManagerInterface $entityManager,
TokenStorageInterface $tokenStorage,
UserRepository $userRepository,
WallabagV2Import $wallabagV2Import,
FirefoxImport $firefoxImport,
ChromeImport $chromeImport,
ReadabilityImport $readabilityImport,
InstapaperImport $instapaperImport,
PinboardImport $pinboardImport,
DeliciousImport $deliciousImport,
WallabagV1Import $wallabagV1Import,
ElcuratorImport $elcuratorImport,
OmnivoreImport $omnivoreImport,
ShaarliImport $shaarliImport,
PocketHtmlImport $pocketHtmlImport,
PocketCsvImport $pocketCsvImport
) {
$this->entityManager = $entityManager;
$this->tokenStorage = $tokenStorage;
$this->userRepository = $userRepository;
@ -52,6 +76,10 @@ class ImportCommand extends Command
$this->deliciousImport = $deliciousImport;
$this->omnivoreImport = $omnivoreImport;
$this->wallabagV1Import = $wallabagV1Import;
$this->elcuratorImport = $elcuratorImport;
$this->shaarliImport = $shaarliImport;
$this->pocketHtmlImport = $pocketHtmlImport;
$this->pocketCsvImport = $pocketCsvImport;
parent::__construct();
}
@ -63,7 +91,7 @@ class ImportCommand extends Command
->setDescription('Import entries from a JSON export')
->addArgument('username', InputArgument::REQUIRED, 'User to populate')
->addArgument('filepath', InputArgument::REQUIRED, 'Path to the JSON file')
->addOption('importer', null, InputOption::VALUE_OPTIONAL, 'The importer to use: v1, v2, instapaper, pinboard, delicious, readability, firefox or chrome', 'v1')
->addOption('importer', null, InputOption::VALUE_OPTIONAL, 'The importer to use: v1, v2, instapaper, pinboard, delicious, readability, firefox, chrome, elcurator, shaarli, pocket or pocket_csv', 'v1')
->addOption('markAsRead', null, InputOption::VALUE_OPTIONAL, 'Mark all entries as read', false)
->addOption('useUserId', null, InputOption::VALUE_NONE, 'Use user id instead of username to find account')
->addOption('disableContentUpdate', null, InputOption::VALUE_NONE, 'Disable fetching updated content from URL')
@ -125,6 +153,18 @@ class ImportCommand extends Command
break;
case 'omnivore':
$import = $this->omnivoreImport;
break;
case 'elcurator':
$import = $this->elcuratorImport;
break;
case 'shaarli':
$import = $this->shaarliImport;
break;
case 'pocket':
$import = $this->pocketHtmlImport;
break;
case 'pocket_csv':
$import = $this->pocketCsvImport;
break;
default:
$import = $this->wallabagV1Import;

View File

@ -21,9 +21,26 @@ class RabbitMQConsumerTotalProxy
private Consumer $deliciousConsumer;
private Consumer $elcuratorConsumer;
private Consumer $omnivoreConsumer;
private Consumer $shaarliConsumer;
private Consumer $pocketHtmlConsumer;
private Consumer $pocketCsvConsumer;
public function __construct(Consumer $pocketConsumer, Consumer $readabilityConsumer, Consumer $wallabagV1Consumer, Consumer $wallabagV2Consumer, Consumer $firefoxConsumer, Consumer $chromeConsumer, Consumer $instapaperConsumer, Consumer $pinboardConsumer, Consumer $deliciousConsumer, Consumer $elcuratorConsumer, Consumer $omnivoreConsumer)
{
public function __construct(
Consumer $pocketConsumer,
Consumer $readabilityConsumer,
Consumer $wallabagV1Consumer,
Consumer $wallabagV2Consumer,
Consumer $firefoxConsumer,
Consumer $chromeConsumer,
Consumer $instapaperConsumer,
Consumer $pinboardConsumer,
Consumer $deliciousConsumer,
Consumer $elcuratorConsumer,
Consumer $omnivoreConsumer,
Consumer $shaarliConsumer,
Consumer $pocketHtmlConsumer,
Consumer $pocketCsvConsumer
) {
$this->pocketConsumer = $pocketConsumer;
$this->readabilityConsumer = $readabilityConsumer;
$this->wallabagV1Consumer = $wallabagV1Consumer;
@ -35,6 +52,9 @@ class RabbitMQConsumerTotalProxy
$this->deliciousConsumer = $deliciousConsumer;
$this->elcuratorConsumer = $elcuratorConsumer;
$this->omnivoreConsumer = $omnivoreConsumer;
$this->shaarliConsumer = $shaarliConsumer;
$this->pocketHtmlConsumer = $pocketHtmlConsumer;
$this->pocketCsvConsumer = $pocketCsvConsumer;
}
/**
@ -82,6 +102,15 @@ class RabbitMQConsumerTotalProxy
case 'omnivore':
$consumer = $this->omnivoreConsumer;
break;
case 'shaarli':
$consumer = $this->shaarliConsumer;
break;
case 'pocket_html':
$consumer = $this->pocketHtmlConsumer;
break;
case 'pocket_csv':
$consumer = $this->pocketCsvConsumer;
break;
default:
return 0;
}

View File

@ -0,0 +1,83 @@
<?php
namespace Wallabag\ImportBundle\Controller;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wallabag\ImportBundle\Form\Type\UploadImportType;
use Wallabag\ImportBundle\Import\ImportInterface;
abstract class HtmlController extends AbstractController
{
/**
* @Route("/html", name="import_html")
*
* @return Response
*/
public function indexAction(Request $request, TranslatorInterface $translator)
{
$form = $this->createForm(UploadImportType::class);
$form->handleRequest($request);
$wallabag = $this->getImportService();
$wallabag->setUser($this->getUser());
if ($form->isSubmitted() && $form->isValid()) {
$file = $form->get('file')->getData();
$markAsRead = $form->get('mark_as_read')->getData();
$name = $this->getUser()->getId() . '.html';
if (null !== $file && \in_array($file->getClientMimeType(), $this->getParameter('wallabag_import.allow_mimetypes'), true) && $file->move($this->getParameter('wallabag_import.resource_dir'), $name)) {
$res = $wallabag
->setFilepath($this->getParameter('wallabag_import.resource_dir') . '/' . $name)
->setMarkAsRead($markAsRead)
->import();
$message = 'flashes.import.notice.failed';
if (true === $res) {
$summary = $wallabag->getSummary();
$message = $translator->trans('flashes.import.notice.summary', [
'%imported%' => $summary['imported'],
'%skipped%' => $summary['skipped'],
]);
if (0 < $summary['queued']) {
$message = $translator->trans('flashes.import.notice.summary_with_queue', [
'%queued%' => $summary['queued'],
]);
}
unlink($this->getParameter('wallabag_import.resource_dir') . '/' . $name);
}
$this->addFlash('notice', $message);
return $this->redirect($this->generateUrl('homepage'));
}
$this->addFlash('notice', 'flashes.import.notice.failed_on_file');
}
return $this->render($this->getImportTemplate(), [
'form' => $form->createView(),
'import' => $wallabag,
]);
}
/**
* Return the service to handle the import.
*
* @return ImportInterface
*/
abstract protected function getImportService();
/**
* Return the template used for the form.
*
* @return string
*/
abstract protected function getImportTemplate();
}

View File

@ -58,6 +58,9 @@ class ImportController extends AbstractController
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('delicious')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('elcurator')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('omnivore')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('shaarli')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('pocket_html')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('pocket_csv')
;
} catch (\Exception $e) {
$rabbitNotInstalled = true;
@ -77,6 +80,9 @@ class ImportController extends AbstractController
+ $redis->llen('wallabag.import.delicious')
+ $redis->llen('wallabag.import.elcurator')
+ $redis->llen('wallabag.import.omnivore')
+ $redis->llen('wallabag.import.shaarli')
+ $redis->llen('wallabag.import.pocket_html')
+ $redis->llen('wallabag.import.pocket_csv')
;
} catch (\Exception $e) {
$redisNotInstalled = true;

View File

@ -0,0 +1,57 @@
<?php
namespace Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use OldSound\RabbitMqBundle\RabbitMq\Producer as RabbitMqProducer;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wallabag\ImportBundle\Import\PocketCsvImport;
use Wallabag\ImportBundle\Redis\Producer as RedisProducer;
class PocketCsvController extends HtmlController
{
private PocketCsvImport $pocketCsvImport;
private Config $craueConfig;
private RabbitMqProducer $rabbitMqProducer;
private RedisProducer $redisProducer;
public function __construct(PocketCsvImport $pocketCsvImport, Config $craueConfig, RabbitMqProducer $rabbitMqProducer, RedisProducer $redisProducer)
{
$this->pocketCsvImport = $pocketCsvImport;
$this->craueConfig = $craueConfig;
$this->rabbitMqProducer = $rabbitMqProducer;
$this->redisProducer = $redisProducer;
}
/**
* @Route("/pocket_csv", name="import_pocket_csv")
*/
public function indexAction(Request $request, TranslatorInterface $translator)
{
return parent::indexAction($request, $translator);
}
/**
* {@inheritdoc}
*/
protected function getImportService()
{
if ($this->craueConfig->get('import_with_rabbitmq')) {
$this->pocketCsvImport->setProducer($this->rabbitMqProducer);
} elseif ($this->craueConfig->get('import_with_redis')) {
$this->pocketCsvImport->setProducer($this->redisProducer);
}
return $this->pocketCsvImport;
}
/**
* {@inheritdoc}
*/
protected function getImportTemplate()
{
return '@WallabagImport/PocketCsv/index.html.twig';
}
}

View File

@ -0,0 +1,57 @@
<?php
namespace Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use OldSound\RabbitMqBundle\RabbitMq\Producer as RabbitMqProducer;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wallabag\ImportBundle\Import\PocketHtmlImport;
use Wallabag\ImportBundle\Redis\Producer as RedisProducer;
class PocketHtmlController extends HtmlController
{
private PocketHtmlImport $pocketHtmlImport;
private Config $craueConfig;
private RabbitMqProducer $rabbitMqProducer;
private RedisProducer $redisProducer;
public function __construct(PocketHtmlImport $pocketHtmlImport, Config $craueConfig, RabbitMqProducer $rabbitMqProducer, RedisProducer $redisProducer)
{
$this->pocketHtmlImport = $pocketHtmlImport;
$this->craueConfig = $craueConfig;
$this->rabbitMqProducer = $rabbitMqProducer;
$this->redisProducer = $redisProducer;
}
/**
* @Route("/pocket_html", name="import_pocket_html")
*/
public function indexAction(Request $request, TranslatorInterface $translator)
{
return parent::indexAction($request, $translator);
}
/**
* {@inheritdoc}
*/
protected function getImportService()
{
if ($this->craueConfig->get('import_with_rabbitmq')) {
$this->pocketHtmlImport->setProducer($this->rabbitMqProducer);
} elseif ($this->craueConfig->get('import_with_redis')) {
$this->pocketHtmlImport->setProducer($this->redisProducer);
}
return $this->pocketHtmlImport;
}
/**
* {@inheritdoc}
*/
protected function getImportTemplate()
{
return '@WallabagImport/PocketHtml/index.html.twig';
}
}

View File

@ -0,0 +1,57 @@
<?php
namespace Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use OldSound\RabbitMqBundle\RabbitMq\Producer as RabbitMqProducer;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wallabag\ImportBundle\Import\ShaarliImport;
use Wallabag\ImportBundle\Redis\Producer as RedisProducer;
class ShaarliController extends HtmlController
{
private ShaarliImport $shaarliImport;
private Config $craueConfig;
private RabbitMqProducer $rabbitMqProducer;
private RedisProducer $redisProducer;
public function __construct(ShaarliImport $shaarliImport, Config $craueConfig, RabbitMqProducer $rabbitMqProducer, RedisProducer $redisProducer)
{
$this->shaarliImport = $shaarliImport;
$this->craueConfig = $craueConfig;
$this->rabbitMqProducer = $rabbitMqProducer;
$this->redisProducer = $redisProducer;
}
/**
* @Route("/shaarli", name="import_shaarli")
*/
public function indexAction(Request $request, TranslatorInterface $translator)
{
return parent::indexAction($request, $translator);
}
/**
* {@inheritdoc}
*/
protected function getImportService()
{
if ($this->craueConfig->get('import_with_rabbitmq')) {
$this->shaarliImport->setProducer($this->rabbitMqProducer);
} elseif ($this->craueConfig->get('import_with_redis')) {
$this->shaarliImport->setProducer($this->redisProducer);
}
return $this->shaarliImport;
}
/**
* {@inheritdoc}
*/
protected function getImportTemplate()
{
return '@WallabagImport/Shaarli/index.html.twig';
}
}

View File

@ -0,0 +1,210 @@
<?php
namespace Wallabag\ImportBundle\Import;
use Wallabag\CoreBundle\Entity\Entry;
use Wallabag\CoreBundle\Event\EntrySavedEvent;
abstract class HtmlImport extends AbstractImport
{
protected $filepath;
/**
* {@inheritdoc}
*/
abstract public function getName();
/**
* {@inheritdoc}
*/
abstract public function getUrl();
/**
* {@inheritdoc}
*/
abstract public function getDescription();
/**
* {@inheritdoc}
*/
public function import()
{
if (!$this->user) {
$this->logger->error('Wallabag HTML Import: user is not defined');
return false;
}
if (!file_exists($this->filepath) || !is_readable($this->filepath)) {
$this->logger->error('Wallabag HTML Import: unable to read file', ['filepath' => $this->filepath]);
return false;
}
$html = new \DOMDocument();
libxml_use_internal_errors(true);
$html->loadHTMLFile($this->filepath);
$hrefs = $html->getElementsByTagName('a');
libxml_use_internal_errors(false);
if (0 === $hrefs->length) {
$this->logger->error('Wallabag HTML: no entries in imported file');
return false;
}
$entries = [];
foreach ($hrefs as $href) {
$entry = [];
$entry['url'] = $href->getAttribute('href');
$entry['tags'] = $href->getAttribute('tags');
$entry['created_at'] = $href->getAttribute('add_date');
$entries[] = $entry;
}
if ($this->producer) {
$this->parseEntriesForProducer($entries);
return true;
}
$this->parseEntries($entries);
return true;
}
/**
* Set file path to the html file.
*
* @param string $filepath
*/
public function setFilepath($filepath)
{
$this->filepath = $filepath;
return $this;
}
/**
* {@inheritdoc}
*/
public function parseEntry(array $importedEntry)
{
$url = $importedEntry['url'];
$existingEntry = $this->em
->getRepository(Entry::class)
->findByUrlAndUserId($url, $this->user->getId());
if (false !== $existingEntry) {
++$this->skippedEntries;
return null;
}
$data = $this->prepareEntry($importedEntry);
$entry = new Entry($this->user);
$entry->setUrl($data['url']);
$entry->updateArchived($data['is_archived']);
$createdAt = new \DateTime();
$createdAt->setTimestamp($data['created_at']);
$entry->setCreatedAt($createdAt);
// update entry with content (in case fetching failed, the given entry will be return)
$this->fetchContent($entry, $data['url'], $data);
if (\array_key_exists('tags', $data)) {
$this->tagsAssigner->assignTagsToEntry(
$entry,
$data['tags']
);
}
$this->em->persist($entry);
++$this->importedEntries;
return $entry;
}
/**
* Parse and insert all given entries.
*/
protected function parseEntries(array $entries)
{
$i = 1;
$entryToBeFlushed = [];
foreach ($entries as $importedEntry) {
$entry = $this->parseEntry($importedEntry);
if (null === $entry) {
continue;
}
// @see AbstractImport
$entryToBeFlushed[] = $entry;
// flush every 20 entries
if (0 === ($i % 20)) {
$this->em->flush();
foreach ($entryToBeFlushed as $entry) {
$this->eventDispatcher->dispatch(new EntrySavedEvent($entry), EntrySavedEvent::NAME);
}
$entryToBeFlushed = [];
}
++$i;
}
$this->em->flush();
if (!empty($entryToBeFlushed)) {
foreach ($entryToBeFlushed as $entry) {
$this->eventDispatcher->dispatch(new EntrySavedEvent($entry), EntrySavedEvent::NAME);
}
}
}
/**
* Parse entries and send them to the queue.
* It should just be a simple loop on all item, no call to the database should be done
* to speedup queuing.
*
* Faster parse entries for Producer.
* We don't care to make check at this time. They'll be done by the consumer.
*/
protected function parseEntriesForProducer(array $entries)
{
foreach ($entries as $importedEntry) {
if ((array) $importedEntry !== $importedEntry) {
continue;
}
// set userId for the producer (it won't know which user is connected)
$importedEntry['userId'] = $this->user->getId();
if ($this->markAsRead) {
$importedEntry = $this->setEntryAsRead($importedEntry);
}
++$this->queuedEntries;
$this->producer->publish(json_encode($importedEntry));
}
}
/**
* {@inheritdoc}
*/
protected function setEntryAsRead(array $importedEntry)
{
$importedEntry['is_archived'] = 1;
return $importedEntry;
}
abstract protected function prepareEntry(array $entry = []);
}

View File

@ -0,0 +1,156 @@
<?php
namespace Wallabag\ImportBundle\Import;
use Wallabag\CoreBundle\Entity\Entry;
class PocketCsvImport extends AbstractImport
{
protected $filepath;
/**
* {@inheritdoc}
*/
public function getName()
{
return 'Pocket CSV';
}
/**
* {@inheritdoc}
*/
public function getUrl()
{
return 'import_pocket_csv';
}
/**
* {@inheritdoc}
*/
public function getDescription()
{
return 'import.pocket_csv.description';
}
/**
* Set file path to the csv file.
*
* @param string $filepath
*/
public function setFilepath($filepath)
{
$this->filepath = $filepath;
return $this;
}
/**
* {@inheritdoc}
*/
public function validateEntry(array $importedEntry)
{
if (empty($importedEntry['url'])) {
return false;
}
return true;
}
public function import()
{
if (!$this->user) {
$this->logger->error('Pocket CSV Import: user is not defined');
return false;
}
if (!file_exists($this->filepath) || !is_readable($this->filepath)) {
$this->logger->error('Pocket CSV Import: unable to read file', ['filepath' => $this->filepath]);
return false;
}
$entries = [];
$handle = fopen($this->filepath, 'r');
while (false !== ($data = fgetcsv($handle, 10240))) {
if ('title' === $data[0]) {
continue;
}
$entries[] = [
'url' => $data[1],
'title' => $data[0],
'is_archived' => 'archive' === $data[4],
'created_at' => $data[2],
'tags' => $data[3],
];
}
fclose($handle);
if (empty($entries)) {
$this->logger->error('PocketCsvImport: no entries in imported file');
return false;
}
if ($this->producer) {
$this->parseEntriesForProducer($entries);
return true;
}
$this->parseEntries($entries);
return true;
}
/**
* {@inheritdoc}
*/
public function parseEntry(array $importedEntry)
{
$existingEntry = $this->em
->getRepository(Entry::class)
->findByUrlAndUserId($importedEntry['url'], $this->user->getId());
if (false !== $existingEntry) {
++$this->skippedEntries;
return;
}
$entry = new Entry($this->user);
$entry->setUrl($importedEntry['url']);
$entry->setTitle($importedEntry['title']);
// update entry with content (in case fetching failed, the given entry will be return)
$this->fetchContent($entry, $importedEntry['url'], $importedEntry);
if (!empty($importedEntry['tags'])) {
$tags = str_replace('|', ',', $importedEntry['tags']);
$this->tagsAssigner->assignTagsToEntry(
$entry,
$tags,
$this->em->getUnitOfWork()->getScheduledEntityInsertions()
);
}
$entry->updateArchived($importedEntry['is_archived']);
$entry->setCreatedAt(\DateTime::createFromFormat('U', $importedEntry['created_at']));
$this->em->persist($entry);
++$this->importedEntries;
return $entry;
}
/**
* {@inheritdoc}
*/
protected function setEntryAsRead(array $importedEntry)
{
$importedEntry['is_archived'] = 'archive';
return $importedEntry;
}
}

View File

@ -0,0 +1,113 @@
<?php
namespace Wallabag\ImportBundle\Import;
class PocketHtmlImport extends HtmlImport
{
protected $filepath;
/**
* {@inheritdoc}
*/
public function getName()
{
return 'Pocket HTML';
}
/**
* {@inheritdoc}
*/
public function getUrl()
{
return 'import_pocket_html';
}
/**
* {@inheritdoc}
*/
public function getDescription()
{
return 'import.pocket_html.description';
}
/**
* {@inheritdoc}
*/
public function validateEntry(array $importedEntry)
{
if (empty($importedEntry['url'])) {
return false;
}
return true;
}
public function import()
{
if (!$this->user) {
$this->logger->error('Pocket HTML Import: user is not defined');
return false;
}
if (!file_exists($this->filepath) || !is_readable($this->filepath)) {
$this->logger->error('Pocket HTML Import: unable to read file', ['filepath' => $this->filepath]);
return false;
}
$html = new \DOMDocument();
libxml_use_internal_errors(true);
$html->loadHTMLFile($this->filepath);
$hrefs = $html->getElementsByTagName('a');
libxml_use_internal_errors(false);
if (0 === $hrefs->length) {
$this->logger->error('Pocket HTML: no entries in imported file');
return false;
}
$entries = [];
foreach ($hrefs as $href) {
$entry = [];
$entry['url'] = $href->getAttribute('href');
$entry['tags'] = $href->getAttribute('tags');
$entry['created_at'] = $href->getAttribute('time_added');
$entries[] = $entry;
}
if ($this->producer) {
$this->parseEntriesForProducer($entries);
return true;
}
$this->parseEntries($entries);
return true;
}
/**
* {@inheritdoc}
*/
protected function prepareEntry(array $entry = [])
{
$data = [
'title' => '',
'html' => false,
'url' => $entry['url'],
'is_archived' => (int) $this->markAsRead,
'is_starred' => false,
'tags' => '',
'created_at' => $entry['created_at'],
];
if (\array_key_exists('tags', $entry) && '' !== $entry['tags']) {
$data['tags'] = $entry['tags'];
}
return $data;
}
}

View File

@ -0,0 +1,66 @@
<?php
namespace Wallabag\ImportBundle\Import;
class ShaarliImport extends HtmlImport
{
protected $filepath;
/**
* {@inheritdoc}
*/
public function getName()
{
return 'Shaarli';
}
/**
* {@inheritdoc}
*/
public function getUrl()
{
return 'import_shaarli';
}
/**
* {@inheritdoc}
*/
public function getDescription()
{
return 'import.shaarli.description';
}
/**
* {@inheritdoc}
*/
public function validateEntry(array $importedEntry)
{
if (empty($importedEntry['url'])) {
return false;
}
return true;
}
/**
* {@inheritdoc}
*/
protected function prepareEntry(array $entry = [])
{
$data = [
'title' => '',
'html' => false,
'url' => $entry['url'],
'is_archived' => (int) $this->markAsRead,
'is_starred' => false,
'tags' => '',
'created_at' => $entry['created_at'],
];
if (\array_key_exists('tags', $entry) && '' !== $entry['tags']) {
$data['tags'] = $entry['tags'];
}
return $data;
}
}

View File

@ -0,0 +1,45 @@
{% extends "@WallabagCore/layout.html.twig" %}
{% block title %}{{ 'import.pocket_csv.page_title'|trans }}{% endblock %}
{% block content %}
<div class="row">
<div class="col s12">
<div class="card-panel settings">
{% include '@WallabagImport/Import/_information.html.twig' %}
<div class="row">
<blockquote>{{ import.description|trans|raw }}</blockquote>
<p>{{ 'import.pocket_csv.how_to'|trans }}</p>
<div class="col s12">
{{ form_start(form, {'method': 'POST'}) }}
{{ form_errors(form) }}
<div class="row">
<div class="file-field input-field col s12">
{{ form_errors(form.file) }}
<div class="btn">
<span>{{ form.file.vars.label|trans }}</span>
{{ form_widget(form.file) }}
</div>
<div class="file-path-wrapper">
<input class="file-path validate" type="text">
</div>
</div>
<div class="input-field col s6 with-checkbox">
<h6>{{ 'import.form.mark_as_read_title'|trans }}</h6>
{{ form_widget(form.mark_as_read) }}
{{ form_label(form.mark_as_read) }}
</div>
</div>
{{ form_widget(form.save, {'attr': {'class': 'btn waves-effect waves-light'}}) }}
{{ form_rest(form) }}
</form>
</div>
</div>
</div>
</div>
</div>
{% endblock %}

View File

@ -0,0 +1,45 @@
{% extends "@WallabagCore/layout.html.twig" %}
{% block title %}{{ 'import.pocket_html.page_title'|trans }}{% endblock %}
{% block content %}
<div class="row">
<div class="col s12">
<div class="card-panel settings">
{% include '@WallabagImport/Import/_information.html.twig' %}
<div class="row">
<blockquote>{{ import.description|trans|raw }}</blockquote>
<p>{{ 'import.pocket_html.how_to'|trans }}</p>
<div class="col s12">
{{ form_start(form, {'method': 'POST'}) }}
{{ form_errors(form) }}
<div class="row">
<div class="file-field input-field col s12">
{{ form_errors(form.file) }}
<div class="btn">
<span>{{ form.file.vars.label|trans }}</span>
{{ form_widget(form.file) }}
</div>
<div class="file-path-wrapper">
<input class="file-path validate" type="text">
</div>
</div>
<div class="input-field col s6 with-checkbox">
<h6>{{ 'import.form.mark_as_read_title'|trans }}</h6>
{{ form_widget(form.mark_as_read) }}
{{ form_label(form.mark_as_read) }}
</div>
</div>
{{ form_widget(form.save, {'attr': {'class': 'btn waves-effect waves-light'}}) }}
{{ form_rest(form) }}
</form>
</div>
</div>
</div>
</div>
</div>
{% endblock %}

View File

@ -0,0 +1,45 @@
{% extends "@WallabagCore/layout.html.twig" %}
{% block title %}{{ 'import.shaarli.page_title'|trans }}{% endblock %}
{% block content %}
<div class="row">
<div class="col s12">
<div class="card-panel settings">
{% include '@WallabagImport/Import/_information.html.twig' %}
<div class="row">
<blockquote>{{ import.description|trans|raw }}</blockquote>
<p>{{ 'import.shaarli.how_to'|trans }}</p>
<div class="col s12">
{{ form_start(form, {'method': 'POST'}) }}
{{ form_errors(form) }}
<div class="row">
<div class="file-field input-field col s12">
{{ form_errors(form.file) }}
<div class="btn">
<span>{{ form.file.vars.label|trans }}</span>
{{ form_widget(form.file) }}
</div>
<div class="file-path-wrapper">
<input class="file-path validate" type="text">
</div>
</div>
<div class="input-field col s6 with-checkbox">
<h6>{{ 'import.form.mark_as_read_title'|trans }}</h6>
{{ form_widget(form.mark_as_read) }}
{{ form_label(form.mark_as_read) }}
</div>
</div>
{{ form_widget(form.save, {'attr': {'class': 'btn waves-effect waves-light'}}) }}
{{ form_rest(form) }}
</form>
</div>
</div>
</div>
</div>
</div>
{% endblock %}

View File

@ -190,6 +190,7 @@ class EntryRestControllerTest extends WallabagApiTestCase
'tags' => 'foo',
'since' => 1443274283,
'public' => 0,
'annotations' => 1,
]);
$this->assertSame(200, $this->client->getResponse()->getStatusCode());
@ -217,6 +218,7 @@ class EntryRestControllerTest extends WallabagApiTestCase
$this->assertStringContainsString('tags=foo', $content['_links'][$link]['href']);
$this->assertStringContainsString('since=1443274283', $content['_links'][$link]['href']);
$this->assertStringContainsString('public=0', $content['_links'][$link]['href']);
$this->assertStringContainsString('annotations=1', $content['_links'][$link]['href']);
}
$this->assertSame('application/json', $this->client->getResponse()->headers->get('Content-Type'));
@ -1383,4 +1385,85 @@ class EntryRestControllerTest extends WallabagApiTestCase
$this->assertGreaterThan(0, $content['id']);
$this->assertSame('https://www.lemonde.fr/m-perso/article/2017/06/25/antoine-de-caunes-je-veux-avoir-le-droit-de-tatonner_5150728_4497916.html', $content['url']);
}
public function testGetEntriesWithAnnotationsFilter()
{
// Test filter for entries WITH annotations
// From fixtures: entry1, entry2 have annotations (for admin-user), entry3 has annotations (for bob-user)
// entry4, entry5, entry6 don't have annotations
$this->client->request('GET', '/api/entries', [
'annotations' => 1,
]);
$this->assertSame(200, $this->client->getResponse()->getStatusCode());
$content = json_decode($this->client->getResponse()->getContent(), true);
$this->assertArrayHasKey('items', $content['_embedded']);
// Check that only entries with annotations are returned
$entriesWithAnnotations = ['http://0.0.0.0/entry1', 'http://0.0.0.0/entry2'];
$entriesWithoutAnnotations = ['http://0.0.0.0/entry4', 'http://0.0.0.0/entry5', 'http://0.0.0.0/entry6'];
foreach ($content['_embedded']['items'] as $item) {
if (\in_array($item['url'], $entriesWithAnnotations, true)) {
$this->assertNotEmpty($item['annotations'], 'Entry with URL ' . $item['url'] . ' should have annotations');
}
$this->assertNotContains($item['url'], $entriesWithoutAnnotations, 'Entry without annotations should NOT be in the results');
}
// Ensure we have at least the entries with annotations for admin-user
$foundUrls = array_column($content['_embedded']['items'], 'url');
$this->assertContains('http://0.0.0.0/entry1', $foundUrls, 'entry1 with annotations should be in the results');
$this->assertContains('http://0.0.0.0/entry2', $foundUrls, 'entry2 with annotations should be in the results');
// Check pagination links contain the filter
$this->assertArrayHasKey('_links', $content);
foreach (['self', 'first', 'last'] as $link) {
$this->assertArrayHasKey('href', $content['_links'][$link]);
$this->assertStringContainsString('annotations=1', $content['_links'][$link]['href']);
}
$this->assertSame('application/json', $this->client->getResponse()->headers->get('Content-Type'));
}
public function testGetEntriesWithoutAnnotationsFilter()
{
// Test filter for entries WITHOUT annotations
$this->client->request('GET', '/api/entries', [
'annotations' => 0,
]);
$this->assertSame(200, $this->client->getResponse()->getStatusCode());
$content = json_decode($this->client->getResponse()->getContent(), true);
$this->assertArrayHasKey('items', $content['_embedded']);
// Check that only entries without annotations are returned
$entriesWithoutAnnotations = ['http://0.0.0.0/entry4', 'http://0.0.0.0/entry5', 'http://0.0.0.0/entry6'];
$entriesWithAnnotations = ['http://0.0.0.0/entry1', 'http://0.0.0.0/entry2'];
foreach ($content['_embedded']['items'] as $item) {
if (\in_array($item['url'], $entriesWithoutAnnotations, true)) {
$this->assertEmpty($item['annotations'], 'Entry with URL ' . $item['url'] . ' should NOT have annotations');
}
$this->assertNotContains($item['url'], $entriesWithAnnotations, 'Entry with annotations should NOT be in the results');
}
// Ensure we have the entries without annotations for admin-user
$foundUrls = array_column($content['_embedded']['items'], 'url');
$this->assertContains('http://0.0.0.0/entry4', $foundUrls, 'entry4 without annotations should be in the results');
$this->assertContains('http://0.0.0.0/entry5', $foundUrls, 'entry5 without annotations should be in the results');
$this->assertContains('http://0.0.0.0/entry6', $foundUrls, 'entry6 without annotations should be in the results');
// Check pagination links contain the filter
$this->assertArrayHasKey('_links', $content);
foreach (['self', 'first', 'last'] as $link) {
$this->assertArrayHasKey('href', $content['_links'][$link]);
$this->assertStringContainsString('annotations=0', $content['_links'][$link]['href']);
}
$this->assertSame('application/json', $this->client->getResponse()->headers->get('Content-Type'));
}
}

View File

@ -123,9 +123,9 @@ class FirefoxControllerTest extends WallabagCoreTestCase
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for http://lexpansion.lexpress.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for http://lexpansion.lexpress.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for http://lexpansion.lexpress.fr is ok');
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
$this->assertCount(3, $content->getTags());
}

View File

@ -24,6 +24,6 @@ class ImportControllerTest extends WallabagCoreTestCase
$crawler = $client->request('GET', '/import/');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(11, $crawler->filter('blockquote')->count());
$this->assertSame(14, $crawler->filter('blockquote')->count());
}
}

View File

@ -0,0 +1,168 @@
<?php
namespace Tests\Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use Doctrine\ORM\EntityManagerInterface;
use Predis\Client;
use Symfony\Component\HttpFoundation\File\UploadedFile;
use Tests\Wallabag\CoreBundle\WallabagCoreTestCase;
use Wallabag\CoreBundle\Entity\Entry;
class PocketCsvControllerTest extends WallabagCoreTestCase
{
public function testImportPocketCsv()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_csv');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
}
public function testImportPocketCsvWithRabbitEnabled()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 1);
$crawler = $client->request('GET', '/import/pocket_csv');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 0);
}
public function testImportPocketCsvBadFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_csv');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$data = [
'upload_import_file[file]' => '',
];
$client->submit($form, $data);
$this->assertSame(200, $client->getResponse()->getStatusCode());
}
public function testImportPocketCsvWithRedisEnabled()
{
$this->checkRedis();
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_redis', 1);
$crawler = $client->request('GET', '/import/pocket_csv');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/pocket.csv', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$this->assertNotEmpty($client->getContainer()->get(Client::class)->lpop('wallabag.import.pocket_csv'));
$client->getContainer()->get(Config::class)->set('import_with_redis', 0);
}
public function testImportWallabagWithPocketCsvFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_csv');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/pocket.csv', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$entries = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findBy(['user' => $this->getLoggedInUserId()]);
$expectedEntries = [
'http://youmightnotneedjquery.com/,1600322788',
'https://jp-lambert.me/est-ce-que-jai-besoin-d-un-scrum-master-604f5a471c73',
'https://www.monde-diplomatique.fr/2020/09/HALIMI/62165',
'https://www.reddit.com/r/DataHoarder/comments/ioupbk/archivebox_question_how_do_i_import_links_from_a/',
'https://www.numerama.com/politique/646826-tu-vas-pleurer-les-premieres-fois-que-se-passe-t-il-au-sein-du-studio-dubisoft-derriere-trackmania.html#utm_medium=distibuted&utm_source=rss&utm_campaign=646826',
'https://www.nouvelobs.com/rue89/20200911.OBS33165/comment-konbini-s-est-fait-pieger-par-un-pere-masculiniste.html',
'https://reporterre.net/Des-abeilles-pour-resoudre-les-conflits-entre-les-humains-et-les-elephants',
];
$matchedEntries = array_map(function ($expectedUrl) use ($entries) {
foreach ($entries as $entry) {
if ($entry->getUrl() === $expectedUrl) {
return $entry;
}
}
return null;
}, $expectedEntries);
$this->assertCount(\count($expectedEntries), $matchedEntries, 'Should have 7 entries imported from pocket.csv');
}
public function testImportWallabagWithEmptyFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_csv');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/test.csv', 'test.csv');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.failed', $body[0]);
}
}

View File

@ -0,0 +1,168 @@
<?php
namespace Tests\Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use Doctrine\ORM\EntityManagerInterface;
use Predis\Client;
use Symfony\Component\HttpFoundation\File\UploadedFile;
use Tests\Wallabag\CoreBundle\WallabagCoreTestCase;
use Wallabag\CoreBundle\Entity\Entry;
class PocketHtmlControllerTest extends WallabagCoreTestCase
{
public function testImportPocketHtml()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_html');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
}
public function testImportPocketHtmlWithRabbitEnabled()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 1);
$crawler = $client->request('GET', '/import/pocket_html');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 0);
}
public function testImportPocketHtmlBadFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_html');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$data = [
'upload_import_file[file]' => '',
];
$client->submit($form, $data);
$this->assertSame(200, $client->getResponse()->getStatusCode());
}
public function testImportPocketHtmlWithRedisEnabled()
{
$this->checkRedis();
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_redis', 1);
$crawler = $client->request('GET', '/import/pocket_html');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/ril_export.html', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$this->assertNotEmpty($client->getContainer()->get(Client::class)->lpop('wallabag.import.pocket_html'));
$client->getContainer()->get(Config::class)->set('import_with_redis', 0);
}
public function testImportWallabagWithPocketHtmlFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_html');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/ril_export.html', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$content = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findByUrlAndUserId(
'https://www.20minutes.fr/sport/4002755-20220928-tarn-lapins-ravagent-terrain-match-rugby-doit-etre-annule',
$this->getLoggedInUserId()
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
$this->assertCount(3, $content->getTags());
$content = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findByUrlAndUserId(
'https://www.20minutes.fr/paris/4100740-20240715-jo-paris-2024-courir-capitale-maintenant-quais-fermes',
$this->getLoggedInUserId()
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
}
public function testImportWallabagWithEmptyFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_html');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/test.html', 'test.html');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.failed', $body[0]);
}
}

View File

@ -0,0 +1,168 @@
<?php
namespace Tests\Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use Doctrine\ORM\EntityManagerInterface;
use Predis\Client;
use Symfony\Component\HttpFoundation\File\UploadedFile;
use Tests\Wallabag\CoreBundle\WallabagCoreTestCase;
use Wallabag\CoreBundle\Entity\Entry;
class ShaarliControllerTest extends WallabagCoreTestCase
{
public function testImportShaarli()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/shaarli');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
}
public function testImportShaarliWithRabbitEnabled()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 1);
$crawler = $client->request('GET', '/import/shaarli');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 0);
}
public function testImportShaarliBadFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/shaarli');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$data = [
'upload_import_file[file]' => '',
];
$client->submit($form, $data);
$this->assertSame(200, $client->getResponse()->getStatusCode());
}
public function testImportShaarliWithRedisEnabled()
{
$this->checkRedis();
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_redis', 1);
$crawler = $client->request('GET', '/import/shaarli');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/shaarli-bookmarks.html', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$this->assertNotEmpty($client->getContainer()->get(Client::class)->lpop('wallabag.import.shaarli'));
$client->getContainer()->get(Config::class)->set('import_with_redis', 0);
}
public function testImportWallabagWithShaarliFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/shaarli');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/shaarli-bookmarks.html', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$content = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findByUrlAndUserId(
'https://www.20minutes.fr/sport/4002755-20220928-tarn-lapins-ravagent-terrain-match-rugby-doit-etre-annule',
$this->getLoggedInUserId()
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
$this->assertCount(2, $content->getTags());
$content = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findByUrlAndUserId(
'https://www.20minutes.fr/paris/4100740-20240715-jo-paris-2024-courir-capitale-maintenant-quais-fermes',
$this->getLoggedInUserId()
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
}
public function testImportWallabagWithEmptyFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/shaarli');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/test.html', 'test.html');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.failed', $body[0]);
}
}

View File

@ -0,0 +1,252 @@
<?php
namespace Tests\Wallabag\ImportBundle\Import;
use Doctrine\ORM\EntityManager;
use M6Web\Component\RedisMock\RedisMockFactory;
use Monolog\Handler\TestHandler;
use Monolog\Logger;
use PHPUnit\Framework\TestCase;
use Predis\Client;
use Simpleue\Queue\RedisQueue;
use Symfony\Component\EventDispatcher\EventDispatcher;
use Wallabag\CoreBundle\Entity\Entry;
use Wallabag\CoreBundle\Helper\ContentProxy;
use Wallabag\CoreBundle\Helper\TagsAssigner;
use Wallabag\CoreBundle\Repository\EntryRepository;
use Wallabag\ImportBundle\Import\PocketCsvImport;
use Wallabag\ImportBundle\Redis\Producer;
use Wallabag\UserBundle\Entity\User;
class PocketCsvImportTest extends TestCase
{
protected $user;
protected $em;
protected $logHandler;
protected $contentProxy;
protected $tagsAssigner;
public function testInit()
{
$pocketCsvImport = $this->getPocketCsvImport();
$this->assertSame('Pocket CSV', $pocketCsvImport->getName());
$this->assertNotEmpty($pocketCsvImport->getUrl());
$this->assertSame('import.pocket_csv.description', $pocketCsvImport->getDescription());
}
public function testImport()
{
$pocketCsvImport = $this->getPocketCsvImport(false, 7);
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(7))
->method('findByUrlAndUserId')
->willReturn(false);
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->exactly(7))
->method('updateEntry')
->willReturn($entry);
$res = $pocketCsvImport->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 7, 'queued' => 0], $pocketCsvImport->getSummary());
}
public function testImportAndMarkAllAsRead()
{
$pocketCsvImport = $this->getPocketCsvImport(false, 1);
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(7))
->method('findByUrlAndUserId')
->will($this->onConsecutiveCalls(false, true));
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$this->contentProxy
->expects($this->exactly(1))
->method('updateEntry')
->willReturn(new Entry($this->user));
// check that every entry persisted are archived
$this->em
->expects($this->any())
->method('persist')
->with($this->callback(fn ($persistedEntry) => (bool) $persistedEntry->isArchived()));
$res = $pocketCsvImport
->setMarkAsRead(true)
->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 6, 'imported' => 1, 'queued' => 0], $pocketCsvImport->getSummary());
}
public function testImportWithRabbit()
{
$pocketCsvImport = $this->getPocketCsvImport();
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$producer = $this->getMockBuilder(\OldSound\RabbitMqBundle\RabbitMq\Producer::class)
->disableOriginalConstructor()
->getMock();
$producer
->expects($this->exactly(7))
->method('publish');
$pocketCsvImport->setProducer($producer);
$res = $pocketCsvImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 7], $pocketCsvImport->getSummary());
}
public function testImportWithRedis()
{
$pocketCsvImport = $this->getPocketCsvImport();
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$factory = new RedisMockFactory();
$redisMock = $factory->getAdapter(Client::class, true);
$queue = new RedisQueue($redisMock, 'pocket_csv');
$producer = new Producer($queue);
$pocketCsvImport->setProducer($producer);
$res = $pocketCsvImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 7], $pocketCsvImport->getSummary());
$this->assertNotEmpty($redisMock->lpop('pocket_csv'));
}
public function testImportBadFile()
{
$pocketCsvImport = $this->getPocketCsvImport();
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/Import/wallabag-v1.jsonx');
$res = $pocketCsvImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Pocket CSV Import: unable to read file', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
public function testImportUserNotDefined()
{
$pocketCsvImport = $this->getPocketCsvImport(true);
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$res = $pocketCsvImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Pocket CSV Import: user is not defined', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
private function getPocketCsvImport($unsetUser = false, $dispatched = 0)
{
$this->user = new User();
$this->em = $this->getMockBuilder(EntityManager::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy = $this->getMockBuilder(ContentProxy::class)
->disableOriginalConstructor()
->getMock();
$this->tagsAssigner = $this->getMockBuilder(TagsAssigner::class)
->disableOriginalConstructor()
->getMock();
$dispatcher = $this->getMockBuilder(EventDispatcher::class)
->disableOriginalConstructor()
->getMock();
$dispatcher
->expects($this->exactly($dispatched))
->method('dispatch');
$this->logHandler = new TestHandler();
$logger = new Logger('test', [$this->logHandler]);
$wallabag = new PocketCsvImport($this->em, $this->contentProxy, $this->tagsAssigner, $dispatcher, $logger);
if (false === $unsetUser) {
$wallabag->setUser($this->user);
}
return $wallabag;
}
}

View File

@ -0,0 +1,254 @@
<?php
namespace Tests\Wallabag\ImportBundle\Import;
use Doctrine\ORM\EntityManager;
use M6Web\Component\RedisMock\RedisMockFactory;
use Monolog\Handler\TestHandler;
use Monolog\Logger;
use PHPUnit\Framework\TestCase;
use Predis\Client;
use Simpleue\Queue\RedisQueue;
use Symfony\Component\EventDispatcher\EventDispatcher;
use Wallabag\CoreBundle\Entity\Entry;
use Wallabag\CoreBundle\Helper\ContentProxy;
use Wallabag\CoreBundle\Helper\TagsAssigner;
use Wallabag\CoreBundle\Repository\EntryRepository;
use Wallabag\ImportBundle\Import\PocketHtmlImport;
use Wallabag\ImportBundle\Redis\Producer;
use Wallabag\UserBundle\Entity\User;
class PocketHtmlImportTest extends TestCase
{
protected $user;
protected $em;
protected $logHandler;
protected $contentProxy;
protected $tagsAssigner;
public function testInit()
{
$pocketHtmlImport = $this->getPocketHtmlImport();
$this->assertSame('Pocket HTML', $pocketHtmlImport->getName());
$this->assertNotEmpty($pocketHtmlImport->getUrl());
$this->assertSame('import.pocket_html.description', $pocketHtmlImport->getDescription());
}
public function testImport()
{
$pocketHtmlImport = $this->getPocketHtmlImport(false, 2);
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(2))
->method('findByUrlAndUserId')
->willReturn(false);
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->exactly(2))
->method('updateEntry')
->willReturn($entry);
$res = $pocketHtmlImport->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 2, 'queued' => 0], $pocketHtmlImport->getSummary());
}
public function testImportAndMarkAllAsRead()
{
$pocketHtmlImport = $this->getPocketHtmlImport(false, 1);
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(2))
->method('findByUrlAndUserId')
->will($this->onConsecutiveCalls(false, true));
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$this->contentProxy
->expects($this->exactly(1))
->method('updateEntry')
->willReturn(new Entry($this->user));
// check that every entry persisted are archived
$this->em
->expects($this->any())
->method('persist')
->with($this->callback(function ($persistedEntry) {
return (bool) $persistedEntry->isArchived();
}));
$res = $pocketHtmlImport
->setMarkAsRead(true)
->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 1, 'imported' => 1, 'queued' => 0], $pocketHtmlImport->getSummary());
}
public function testImportWithRabbit()
{
$pocketHtmlImport = $this->getPocketHtmlImport();
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$producer = $this->getMockBuilder(\OldSound\RabbitMqBundle\RabbitMq\Producer::class)
->disableOriginalConstructor()
->getMock();
$producer
->expects($this->exactly(2))
->method('publish');
$pocketHtmlImport->setProducer($producer);
$res = $pocketHtmlImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 2], $pocketHtmlImport->getSummary());
}
public function testImportWithRedis()
{
$pocketHtmlImport = $this->getPocketHtmlImport();
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$factory = new RedisMockFactory();
$redisMock = $factory->getAdapter(Client::class, true);
$queue = new RedisQueue($redisMock, 'pocket_html');
$producer = new Producer($queue);
$pocketHtmlImport->setProducer($producer);
$res = $pocketHtmlImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 2], $pocketHtmlImport->getSummary());
$this->assertNotEmpty($redisMock->lpop('pocket_html'));
}
public function testImportBadFile()
{
$pocketHtmlImport = $this->getPocketHtmlImport();
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/wallabag-v1.jsonx');
$res = $pocketHtmlImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Pocket HTML Import: unable to read file', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
public function testImportUserNotDefined()
{
$pocketHtmlImport = $this->getPocketHtmlImport(true);
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$res = $pocketHtmlImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Pocket HTML Import: user is not defined', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
private function getPocketHtmlImport($unsetUser = false, $dispatched = 0)
{
$this->user = new User();
$this->em = $this->getMockBuilder(EntityManager::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy = $this->getMockBuilder(ContentProxy::class)
->disableOriginalConstructor()
->getMock();
$this->tagsAssigner = $this->getMockBuilder(TagsAssigner::class)
->disableOriginalConstructor()
->getMock();
$dispatcher = $this->getMockBuilder(EventDispatcher::class)
->disableOriginalConstructor()
->getMock();
$dispatcher
->expects($this->exactly($dispatched))
->method('dispatch');
$this->logHandler = new TestHandler();
$logger = new Logger('test', [$this->logHandler]);
$wallabag = new PocketHtmlImport($this->em, $this->contentProxy, $this->tagsAssigner, $dispatcher, $logger);
if (false === $unsetUser) {
$wallabag->setUser($this->user);
}
return $wallabag;
}
}

View File

@ -0,0 +1,254 @@
<?php
namespace Tests\Wallabag\ImportBundle\Import;
use Doctrine\ORM\EntityManager;
use M6Web\Component\RedisMock\RedisMockFactory;
use Monolog\Handler\TestHandler;
use Monolog\Logger;
use PHPUnit\Framework\TestCase;
use Predis\Client;
use Simpleue\Queue\RedisQueue;
use Symfony\Component\EventDispatcher\EventDispatcher;
use Wallabag\CoreBundle\Entity\Entry;
use Wallabag\CoreBundle\Helper\ContentProxy;
use Wallabag\CoreBundle\Helper\TagsAssigner;
use Wallabag\CoreBundle\Repository\EntryRepository;
use Wallabag\ImportBundle\Import\ShaarliImport;
use Wallabag\ImportBundle\Redis\Producer;
use Wallabag\UserBundle\Entity\User;
class ShaarliImportTest extends TestCase
{
protected $user;
protected $em;
protected $logHandler;
protected $contentProxy;
protected $tagsAssigner;
public function testInit()
{
$shaarliImport = $this->getShaarliImport();
$this->assertSame('Shaarli', $shaarliImport->getName());
$this->assertNotEmpty($shaarliImport->getUrl());
$this->assertSame('import.shaarli.description', $shaarliImport->getDescription());
}
public function testImport()
{
$shaarliImport = $this->getShaarliImport(false, 2);
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(2))
->method('findByUrlAndUserId')
->willReturn(false);
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->exactly(2))
->method('updateEntry')
->willReturn($entry);
$res = $shaarliImport->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 2, 'queued' => 0], $shaarliImport->getSummary());
}
public function testImportAndMarkAllAsRead()
{
$shaarliImport = $this->getShaarliImport(false, 1);
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(2))
->method('findByUrlAndUserId')
->will($this->onConsecutiveCalls(false, true));
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$this->contentProxy
->expects($this->exactly(1))
->method('updateEntry')
->willReturn(new Entry($this->user));
// check that every entry persisted are archived
$this->em
->expects($this->any())
->method('persist')
->with($this->callback(function ($persistedEntry) {
return (bool) $persistedEntry->isArchived();
}));
$res = $shaarliImport
->setMarkAsRead(true)
->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 1, 'imported' => 1, 'queued' => 0], $shaarliImport->getSummary());
}
public function testImportWithRabbit()
{
$shaarliImport = $this->getShaarliImport();
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$producer = $this->getMockBuilder(\OldSound\RabbitMqBundle\RabbitMq\Producer::class)
->disableOriginalConstructor()
->getMock();
$producer
->expects($this->exactly(2))
->method('publish');
$shaarliImport->setProducer($producer);
$res = $shaarliImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 2], $shaarliImport->getSummary());
}
public function testImportWithRedis()
{
$shaarliImport = $this->getShaarliImport();
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$factory = new RedisMockFactory();
$redisMock = $factory->getAdapter(Client::class, true);
$queue = new RedisQueue($redisMock, 'shaarli');
$producer = new Producer($queue);
$shaarliImport->setProducer($producer);
$res = $shaarliImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 2], $shaarliImport->getSummary());
$this->assertNotEmpty($redisMock->lpop('shaarli'));
}
public function testImportBadFile()
{
$shaarliImport = $this->getShaarliImport();
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/wallabag-v1.jsonx');
$res = $shaarliImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Wallabag HTML Import: unable to read file', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
public function testImportUserNotDefined()
{
$shaarliImport = $this->getShaarliImport(true);
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$res = $shaarliImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Wallabag HTML Import: user is not defined', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
private function getShaarliImport($unsetUser = false, $dispatched = 0)
{
$this->user = new User();
$this->em = $this->getMockBuilder(EntityManager::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy = $this->getMockBuilder(ContentProxy::class)
->disableOriginalConstructor()
->getMock();
$this->tagsAssigner = $this->getMockBuilder(TagsAssigner::class)
->disableOriginalConstructor()
->getMock();
$dispatcher = $this->getMockBuilder(EventDispatcher::class)
->disableOriginalConstructor()
->getMock();
$dispatcher
->expects($this->exactly($dispatched))
->method('dispatch');
$this->logHandler = new TestHandler();
$logger = new Logger('test', [$this->logHandler]);
$wallabag = new ShaarliImport($this->em, $this->contentProxy, $this->tagsAssigner, $dispatcher, $logger);
if (false === $unsetUser) {
$wallabag->setUser($this->user);
}
return $wallabag;
}
}

View File

@ -0,0 +1,10 @@
title,url,time_added,tags,status
You Might Not Need jQuery,http://youmightnotneedjquery.com/,1600322788,,unread
Est-ce que jai besoin dun Scrum Master ? | by Jean-Pierre Lambert | Jean-,https://jp-lambert.me/est-ce-que-jai-besoin-d-un-scrum-master-604f5a471c73,1600172739,,unread
"Avec les accusés dEl Halia, par Gisèle Halimi (Le Monde diplomatique, sept",https://www.monde-diplomatique.fr/2020/09/HALIMI/62165,1599806347,,unread
ArchiveBox question: How do I import links from a RSS feed?,https://www.reddit.com/r/DataHoarder/comments/ioupbk/archivebox_question_how_do_i_import_links_from_a/,1600961496,,archive
« Tu vas pleurer les premières fois » : que se passe-t-il au sein du studio,https://www.numerama.com/politique/646826-tu-vas-pleurer-les-premieres-fois-que-se-passe-t-il-au-sein-du-studio-dubisoft-derriere-trackmania.html#utm_medium=distibuted&utm_source=rss&utm_campaign=646826,1599809025,,unread
Comment Konbini sest fait piéger par un « père masculiniste »,https://www.nouvelobs.com/rue89/20200911.OBS33165/comment-konbini-s-est-fait-pieger-par-un-pere-masculiniste.html,1599819251,,archive
"Des abeilles pour résoudre les « conflits » entre les humains
et les élépha",https://reporterre.net/Des-abeilles-pour-resoudre-les-conflits-entre-les-humains-et-les-elephants,1599890673,,unread
1 title url time_added tags status
2 You Might Not Need jQuery http://youmightnotneedjquery.com/ 1600322788 unread
3 Est-ce que j’ai besoin d’un Scrum Master ? | by Jean-Pierre Lambert | Jean- https://jp-lambert.me/est-ce-que-jai-besoin-d-un-scrum-master-604f5a471c73 1600172739 unread
4 Avec les accusés d’El Halia, par Gisèle Halimi (Le Monde diplomatique, sept https://www.monde-diplomatique.fr/2020/09/HALIMI/62165 1599806347 unread
5 ArchiveBox question: How do I import links from a RSS feed? https://www.reddit.com/r/DataHoarder/comments/ioupbk/archivebox_question_how_do_i_import_links_from_a/ 1600961496 archive
6 « Tu vas pleurer les premières fois » : que se passe-t-il au sein du studio https://www.numerama.com/politique/646826-tu-vas-pleurer-les-premieres-fois-que-se-passe-t-il-au-sein-du-studio-dubisoft-derriere-trackmania.html#utm_medium=distibuted&utm_source=rss&utm_campaign=646826 1599809025 unread
7 Comment Konbini s’est fait piéger par un « père masculiniste » https://www.nouvelobs.com/rue89/20200911.OBS33165/comment-konbini-s-est-fait-pieger-par-un-pere-masculiniste.html 1599819251 archive
8 Des abeilles pour résoudre les « conflits » entre les humains et les élépha https://reporterre.net/Des-abeilles-pour-resoudre-les-conflits-entre-les-humains-et-les-elephants 1599890673 unread

View File

@ -0,0 +1,21 @@
<!DOCTYPE html>
<html>
<!--So long and thanks for all the fish-->
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Pocket Export</title>
</head>
<body>
<h1>Unread</h1>
<ul>
<li><a href="https://www.20minutes.fr/sport/4002755-20220928-tarn-lapins-ravagent-terrain-match-rugby-doit-etre-annule" time_added="1688628695" tags="ifttt,new_entry_simple">Tarn : Des lapins ravagent le terrain, le match de rugby doit être annulé</a></li>
<li><a href="https://www.20minutes.fr/paris/4100740-20240715-jo-paris-2024-courir-capitale-maintenant-quais-fermes" time_added="1688627412" tags="ifttt,new_entry_simple">JO Paris 2024 : Où courir dans la capitale maintenant que les quais sont fermés ?</a></li>
</ul>
<h1>Read Archive</h1>
<ul>
</ul>
</body>
</html>

View File

@ -0,0 +1,13 @@
<!DOCTYPE NETSCAPE-Bookmark-file-1>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
<!-- This is an automatically generated file.
It will be read and overwritten.
Do Not Edit! -->
<TITLE>Bookmarks</TITLE>
<H1>Shaarli export of all bookmarks on Mon, 17 Jul 23 14:31:25 +0200</H1>
<DL><p>
<DT><A HREF="https://www.20minutes.fr/sport/4002755-20220928-tarn-lapins-ravagent-terrain-match-rugby-doit-etre-annule" ADD_DATE="1686813518" LAST_MODIFIED="1686813519" PRIVATE="0" TAGS="firefoxos">The Legacy of Firefox OS. In the two years or so since Mozilla… | by Ben Francis | Medium</A>
<DD>In the two years or so since Mozilla announced the end of Firefox OS as a Mozilla-run project, the B2G source code has found its way into a surprising number of commercial products.
<DT><A HREF="https://www.20minutes.fr/paris/4100740-20240715-jo-paris-2024-courir-capitale-maintenant-quais-fermes" ADD_DATE="1683376565" LAST_MODIFIED="1686813519" PRIVATE="0" TAGS="firefoxos">JO Paris 2024 : Où courir dans la capitale maintenant que les quais sont fermés ?</A>
</DL><p>
<script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script>

View File

@ -534,6 +534,18 @@ import:
page_title: Import > del.icio.us
description: This importer will import all your Delicious bookmarks. Since 2021, you can export again your data from it using the export page (https://del.icio.us/export). Choose the "JSON" format and download it (like "delicious_export.2021.02.06_21.10.json").
how_to: Please select your Delicious export and click on the button below to upload and import it.
shaarli:
page_title: Import > Shaarli
description: This importer will import all your Shaarli bookmarks. Just go to the Tools section, then into "Export database", choose your bookmarks and export them. You will obtain a HTML file.
how_to: Please choose the bookmark backup file and click on the button below to import it. Note that the process may take a long time since all articles have to be fetched.
pocket_html:
page_title: Import > Pocket HTML
description: This importer will import all your Pocket bookmarks (via HTML export). Just go to https://getpocket.com/export, then export the HTML file. An HTML file will be downloaded (like "ril_export.html").
how_to: Please choose the bookmark backup file and click on the button below to import it. Note that the process may take a long time since all articles have to be fetched.
pocket_csv:
page_title: Import > Pocket CSV
description: This importer will import all your Pocket bookmarks (via CSV export). Just go to https://getpocket.com/export, then export the file. A ZIP file will be downloaded (like "pocket.zip"). Extract it, you will obtain a CSV file, called "part_000000.csv".
how_to: Please choose the bookmark backup file and click on the button below to import it. Note that the process may take a long time since all articles have to be fetched.
developer:
page_title: API clients management
welcome_message: Welcome to the wallabag API
@ -668,6 +680,7 @@ flashes:
archived_reset: Archived entries deleted
otp_enabled: Two-factor authentication enabled
otp_disabled: Two-factor authentication disabled
otp_code_invalid: Invalid two-factor authentication code
tagging_rules_imported: Tagging rules imported
tagging_rules_not_imported: Error while importing tagging rules
ignore_origin_rules_deleted: 'Ignore origin rule deleted'

View File

@ -530,6 +530,19 @@ import:
page_title: Importer > del.icio.us
how_to: Choisissez le fichier de votre export Delicious et cliquez sur le bouton ci-dessous pour l'importer.
description: Depuis 2021, vous pouvez à nouveau exporter vos données depuis Delicious (https://del.icio.us/export). Choisissez le format "JSON" et téléchargez le (un fichier du genre "delicious_export.2021.02.06_21.10.json").
shaarli:
page_title: Importer > Shaarli
description: Cet importateur importera toutes vos signets Shaarli. Il suffit d'aller à la section Outils, puis dans « Base de données d'exportation », choisissez vos signets et exportez-les. Vous obtiendrez un fichier HTML.
how_to: Veuillez sélectionner le fichier de sauvegarde de signet et cliquez sur le bouton ci-dessous pour l'importer. Notez que le processus peut prendre beaucoup de temps puisque tous les articles doivent être récupérés.
pocket_html:
page_title: Importer > Pocket HTML
description: Cet importateur importera toutes vos signets Pocket (via exportation HTML). Il suffit d'aller à https://getpocket.com/export, puis d'exporter le fichier HTML. Un fichier HTML sera téléchargé (comme « ril_export.html »).
how_to: Veuillez choisir le fichier de sauvegarde de signets et cliquez sur le bouton ci-dessous pour l'importer. Pensez au fait que le processus peut prendre longtemps puisque tous les articles doivent être récupérés.
pocket_csv:
page_title: Importer > Pocket CSV
description: Cet importateur importera toutes vos signets Pocket (via exportation CSV). Il suffit d'aller à https://getpocket.com/export, puis d'exporter le fichier. Un fichier ZIP sera téléchargé (comme « pocket.zip »). Décompressez le et vous obtiendrez un fichier CSV appelé "part_000000.csv".
how_to: Veuillez choisir le fichier de sauvegarde de signets et cliquez sur le bouton ci-dessous pour l'importer. Pensez au fait que le processus peut prendre longtemps puisque tous les articles doivent être récupérés.
developer:
page_title: Gestion des clients API
welcome_message: Bienvenue sur lAPI de wallabag