Compare commits

...

17 Commits

Author SHA1 Message Date
6ffd7382c7 Merge pull request #8229 from wallabag/prepare-2613
Prepare 2.6.13 release
2025-06-04 17:29:19 +02:00
46d6e4d923 Update CHANGELOG 2025-06-04 13:59:00 +02:00
09af127446 Unlock php-readability 2025-06-04 13:59:00 +02:00
4c23196304 Prepare 2.6.13 release 2025-06-04 13:59:00 +02:00
42746f418e Merge pull request #8250 from wallabag/update-site-config
Update joker/graby-site-config
2025-06-04 13:54:23 +02:00
35c4feedd8 Update joker/graby-site-config 2025-06-04 13:42:08 +02:00
c451cc96e5 Merge pull request #8240 from wallabag/import/pocket-csv
Add support of Pocket CSV import
2025-06-03 13:32:51 +02:00
c4240c866b PocketCsvImport: rework test to ensure we import all entries
This also adds a test for #8217

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2025-06-03 13:24:27 +02:00
27d66d9e1d Add 'application/vnd.ms-excel' to allowed MIME types config
Backport #8036
2025-06-03 13:24:27 +02:00
c1397f43ac Add Pocket CSV import 2025-06-03 13:24:27 +02:00
52a16bb75f Merge pull request #8193 from wallabag/add-pocket-shaarli-import
Backport Pocket and Shaarli imports from master
2025-06-02 14:23:33 +02:00
f82c87b520 import: fix some tests
Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2025-06-02 13:42:44 +02:00
772a802596 Merge pull request #8194 from wallabag/fix-8158
Add j0k3r/php-readability:1.2.10 as a dependency
2025-06-01 16:51:35 +02:00
29162bde9d Pin j0k3r/php-readability to 1.2.10
We pin php-readability to 1.2.10 because of a regression in 1.2.12

Workaround for #8151

Signed-off-by: Kevin Decherf <kevin@kdecherf.com>
2025-06-01 16:37:22 +02:00
b1614e9267 Add Pocket and Shaarli imports 2025-05-24 16:25:48 +02:00
70999075a6 Merge pull request #8139 from wallabag/fix/otp-wrongly-enabled
Avoid non-validated OTP to be enabled
2025-04-14 09:46:17 +02:00
262f674245 Avoid non-validated OTP to be enabled
The OTP code must be required when enabling OTP. If the provided code is wrong, disable OTP, redirect and notice the user.
2025-04-14 09:18:37 +02:00
39 changed files with 2547 additions and 34 deletions

View File

@ -1,5 +1,21 @@
# Changelog
## [2.6.13](https://github.com/wallabag/wallabag/tree/2.6.13)
[Full Changelog](https://github.com/wallabag/wallabag/compare/2.6.12...2.6.13)
### Improvements
* Add support of Pocket CSV import by @kdecherf and @nicosomb in [https://github.com/wallabag/wallabag/pull/8240](https://github.com/wallabag/wallabag/pull/8240)
* Backport Pocket and Shaarli HTML imports from master by @nicosomb in [https://github.com/wallabag/wallabag/pull/8193](https://github.com/wallabag/wallabag/pull/8193)
### Fixes
* Avoid non-validated OTP to be enabled #8139 by @j0k3r in [https://github.com/wallabag/wallabag/pull/8139](https://github.com/wallabag/wallabag/pull/8139)
### Technical stuff
* Update j0k3r/php-readability:1.2.13 to fix regression (about latin1 instead of UTF-8 used for entries) by @nicosomb [https://github.com/wallabag/wallabag/pull/8194](https://github.com/wallabag/wallabag/pull/8194)
## [2.6.12](https://github.com/wallabag/wallabag/tree/2.6.12)
[Full Changelog](https://github.com/wallabag/wallabag/compare/2.6.11...2.6.12)

View File

@ -283,6 +283,21 @@ old_sound_rabbit_mq:
exchange_options:
name: 'wallabag.import.chrome'
type: topic
import_shaarli:
connection: default
exchange_options:
name: 'wallabag.import.shaarli'
type: topic
import_pocket_html:
connection: default
exchange_options:
name: 'wallabag.import.pocket_html'
type: topic
import_pocket_csv:
connection: default
exchange_options:
name: 'wallabag.import.pocket_csv'
type: topic
consumers:
import_pocket:
connection: default
@ -383,6 +398,33 @@ old_sound_rabbit_mq:
name: 'wallabag.import.chrome'
callback: wallabag_import.consumer.amqp.chrome
qos_options: {prefetch_count: "%rabbitmq_prefetch_count%"}
import_shaarli:
connection: default
exchange_options:
name: 'wallabag.import.shaarli'
type: topic
queue_options:
name: 'wallabag.import.shaarli'
callback: wallabag_import.consumer.amqp.shaarli
qos_options: {prefetch_count: "%rabbitmq_prefetch_count%"}
import_pocket_html:
connection: default
exchange_options:
name: 'wallabag.import.pocket_html'
type: topic
queue_options:
name: 'wallabag.import.pocket_html'
callback: wallabag_import.consumer.amqp.pocket_html
qos_options: {prefetch_count: "%rabbitmq_prefetch_count%"}
import_pocket_csv:
connection: default
exchange_options:
name: 'wallabag.import.pocket_csv'
type: topic
queue_options:
name: 'wallabag.import.pocket_csv'
callback: wallabag_import.consumer.amqp.pocket_csv
qos_options: {prefetch_count: "%rabbitmq_prefetch_count%"}
fos_js_routing:
routes_to_expose:

View File

@ -121,6 +121,21 @@ services:
$rabbitMqProducer: '@old_sound_rabbit_mq.import_wallabag_v2_producer'
$redisProducer: '@wallabag_import.producer.redis.wallabag_v2'
Wallabag\ImportBundle\Controller\ShaarliController:
arguments:
$rabbitMqProducer: '@old_sound_rabbit_mq.import_shaarli_producer'
$redisProducer: '@wallabag_import.producer.redis.shaarli'
Wallabag\ImportBundle\Controller\PocketHtmlController:
arguments:
$rabbitMqProducer: '@old_sound_rabbit_mq.import_pocket_html_producer'
$redisProducer: '@wallabag_import.producer.redis.pocket_html'
Wallabag\ImportBundle\Controller\PocketCsvController:
arguments:
$rabbitMqProducer: '@old_sound_rabbit_mq.import_pocket_csv_producer'
$redisProducer: '@wallabag_import.producer.redis.pocket_csv'
Wallabag\ImportBundle\:
resource: '../../src/Wallabag/ImportBundle/*'
exclude: '../../src/Wallabag/ImportBundle/{Consumer,Controller,Redis}'
@ -394,6 +409,18 @@ services:
tags:
- { name: wallabag_import.import, alias: chrome }
Wallabag\ImportBundle\Import\ShaarliImport:
tags:
- { name: wallabag_import.import, alias: shaarli }
Wallabag\ImportBundle\Import\PocketHtmlImport:
tags:
- { name: wallabag_import.import, alias: pocket_html }
Wallabag\ImportBundle\Import\PocketCsvImport:
tags:
- { name: wallabag_import.import, alias: pocket_csv }
# to factorize the proximity and bypass translation for prev & next
pagerfanta.view.default_wallabag:
class: Pagerfanta\View\OptionableView

View File

@ -19,6 +19,9 @@ services:
$deliciousConsumer: '@old_sound_rabbit_mq.import_delicious_consumer'
$elcuratorConsumer: '@old_sound_rabbit_mq.import_elcurator_consumer'
$omnivoreConsumer: '@old_sound_rabbit_mq.import_omnivore_consumer'
$shaarliConsumer: '@old_sound_rabbit_mq.import_shaarli_consumer'
$pocketHtmlConsumer: '@old_sound_rabbit_mq.import_pocket_html_consumer'
$pocketCsvConsumer: '@old_sound_rabbit_mq.import_pocket_csv_consumer'
wallabag_import.consumer.amqp.pocket:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
@ -74,3 +77,18 @@ services:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\ChromeImport'
wallabag_import.consumer.amqp.shaarli:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\ShaarliImport'
wallabag_import.consumer.amqp.pocket_html:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\PocketHtmlImport'
wallabag_import.consumer.amqp.pocket_csv:
class: Wallabag\ImportBundle\Consumer\AMQPEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\PocketCsvImport'

View File

@ -180,3 +180,51 @@ services:
class: Wallabag\ImportBundle\Consumer\RedisEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\ChromeImport'
# shaarli
wallabag_import.queue.redis.shaarli:
class: Simpleue\Queue\RedisQueue
arguments:
$queueName: "wallabag.import.shaarli"
wallabag_import.producer.redis.shaarli:
class: Wallabag\ImportBundle\Redis\Producer
arguments:
- "@wallabag_import.queue.redis.shaarli"
wallabag_import.consumer.redis.shaarli:
class: Wallabag\ImportBundle\Consumer\RedisEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\ShaarliImport'
# pocket html
wallabag_import.queue.redis.pocket_html:
class: Simpleue\Queue\RedisQueue
arguments:
$queueName: "wallabag.import.pocket_html"
wallabag_import.producer.redis.pocket_html:
class: Wallabag\ImportBundle\Redis\Producer
arguments:
- "@wallabag_import.queue.redis.pocket_html"
wallabag_import.consumer.redis.pocket_html:
class: Wallabag\ImportBundle\Consumer\RedisEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\PocketHtmlImport'
# pocket csv
wallabag_import.queue.redis.pocket_csv:
class: Simpleue\Queue\RedisQueue
arguments:
$queueName: "wallabag.import.pocket_csv"
wallabag_import.producer.redis.pocket_csv:
class: Wallabag\ImportBundle\Redis\Producer
arguments:
- "@wallabag_import.queue.redis.pocket_csv"
wallabag_import.consumer.redis.pocket_csv:
class: Wallabag\ImportBundle\Consumer\RedisEntryConsumer
arguments:
$import: '@Wallabag\ImportBundle\Import\PocketCsvImport'

View File

@ -1,5 +1,5 @@
wallabag_core:
version: 2.6.12
version: 2.6.13
paypal_url: "https://liberapay.com/wallabag/donate"
languages:
en: 'English'
@ -167,5 +167,11 @@ wallabag_core:
rule: _all ~ "https?://www\.lemonde\.fr/tiny.*"
wallabag_import:
allow_mimetypes: ['application/octet-stream', 'application/json', 'text/plain', 'text/csv']
allow_mimetypes:
- 'application/octet-stream'
- 'application/json'
- 'text/plain'
- 'text/csv'
- 'text/html'
- 'application/vnd.ms-excel'
resource_dir: "%kernel.project_dir%/web/uploads/import"

View File

@ -80,6 +80,7 @@
"html2text/html2text": "^4.3.1",
"incenteev/composer-parameter-handler": "^2.1.5",
"j0k3r/graby": "^2.4.5",
"j0k3r/php-readability": "^1.2.13",
"javibravo/simpleue": "^2.1",
"jms/serializer": "^3.29.1",
"jms/serializer-bundle": "^5.3.1",

26
composer.lock generated
View File

@ -4,7 +4,7 @@
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies",
"This file is @generated automatically"
],
"content-hash": "313f6b39034c27c58a102bfadb5ae30f",
"content-hash": "f55cde30f67f014d2bf0b589a2fbe5c4",
"packages": [
{
"name": "babdev/pagerfanta-bundle",
@ -4604,16 +4604,16 @@
},
{
"name": "j0k3r/graby-site-config",
"version": "1.0.198",
"version": "1.0.200",
"source": {
"type": "git",
"url": "https://github.com/j0k3r/graby-site-config.git",
"reference": "4ff9bf9ed32186a74d27782a196659d2b319b577"
"reference": "e9cbbc776a7efcc9c06c07ff360744f266bbfc09"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/j0k3r/graby-site-config/zipball/4ff9bf9ed32186a74d27782a196659d2b319b577",
"reference": "4ff9bf9ed32186a74d27782a196659d2b319b577",
"url": "https://api.github.com/repos/j0k3r/graby-site-config/zipball/e9cbbc776a7efcc9c06c07ff360744f266bbfc09",
"reference": "e9cbbc776a7efcc9c06c07ff360744f266bbfc09",
"shasum": ""
},
"require": {
@ -4642,9 +4642,9 @@
"description": "Graby site config files",
"support": {
"issues": "https://github.com/j0k3r/graby-site-config/issues",
"source": "https://github.com/j0k3r/graby-site-config/tree/1.0.198"
"source": "https://github.com/j0k3r/graby-site-config/tree/1.0.200"
},
"time": "2025-04-01T02:31:18+00:00"
"time": "2025-06-01T02:42:59+00:00"
},
{
"name": "j0k3r/httplug-ssrf-plugin",
@ -4719,16 +4719,16 @@
},
{
"name": "j0k3r/php-readability",
"version": "1.2.12",
"version": "1.2.13",
"source": {
"type": "git",
"url": "https://github.com/j0k3r/php-readability.git",
"reference": "109a22662de0d703f01387e5714ad4f9a03b95c0"
"reference": "b9dde0f4cd46e9fc082bb37f75dc94ecd2f8faad"
},
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/j0k3r/php-readability/zipball/109a22662de0d703f01387e5714ad4f9a03b95c0",
"reference": "109a22662de0d703f01387e5714ad4f9a03b95c0",
"url": "https://api.github.com/repos/j0k3r/php-readability/zipball/b9dde0f4cd46e9fc082bb37f75dc94ecd2f8faad",
"reference": "b9dde0f4cd46e9fc082bb37f75dc94ecd2f8faad",
"shasum": ""
},
"require": {
@ -4790,7 +4790,7 @@
],
"support": {
"issues": "https://github.com/j0k3r/php-readability/issues",
"source": "https://github.com/j0k3r/php-readability/tree/1.2.12"
"source": "https://github.com/j0k3r/php-readability/tree/1.2.13"
},
"funding": [
{
@ -4798,7 +4798,7 @@
"type": "github"
}
],
"time": "2025-03-04T09:20:40+00:00"
"time": "2025-06-03T08:02:58+00:00"
},
{
"name": "javibravo/simpleue",

View File

@ -57,5 +57,15 @@ parameters:
-
message: "#^Method FOS\\\\UserBundle\\\\Model\\\\UserManagerInterface\\:\\:updateUser()#"
count: 6
count: 7
path: src/Wallabag/CoreBundle/Controller/ConfigController.php
-
message: "#^Call to an undefined method Wallabag\\\\ImportBundle\\\\Import\\\\ImportInterface\\:\\:setUser\\(\\)\\.$#"
count: 1
path: src/Wallabag/ImportBundle/Controller/HtmlController.php
-
message: "#^Call to an undefined method Wallabag\\\\ImportBundle\\\\Import\\\\ImportInterface\\:\\:setFilepath\\(\\)\\.$#"
count: 1
path: src/Wallabag/ImportBundle/Controller/HtmlController.php

View File

@ -402,12 +402,14 @@ class ConfigController extends AbstractController
throw new BadRequestHttpException('Bad CSRF token.');
}
$user = $this->getUser();
$isValid = $googleAuthenticator->checkCode(
$this->getUser(),
$user,
$request->get('_auth_code')
);
if (true === $isValid) {
if ($isValid) {
$this->addFlash(
'notice',
'flashes.config.notice.otp_enabled'
@ -417,14 +419,14 @@ class ConfigController extends AbstractController
}
$this->addFlash(
'two_factor',
'scheb_two_factor.code_invalid'
'notice',
'flashes.config.notice.otp_code_invalid'
);
$this->addFlash(
'notice',
'scheb_two_factor.code_invalid'
);
$user->setGoogleAuthenticatorSecret(null);
$user->setBackupCodes(null);
$this->userManager->updateUser($user, true);
return $this->redirect($this->generateUrl('config') . '#set3');
}

View File

@ -45,7 +45,7 @@
<div class="row">
<div class="input-field col s12">
<label for="_auth_code">{{ "auth_code"|trans({}, 'SchebTwoFactorBundle') }}</label>
<input id="_auth_code" type="text" autocomplete="off" name="_auth_code" />
<input id="_auth_code" type="text" autocomplete="off" name="_auth_code" required="required" />
</div>
</div>
</div>

View File

@ -13,11 +13,15 @@ use Symfony\Component\Security\Core\Authentication\Token\Storage\TokenStorageInt
use Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken;
use Wallabag\ImportBundle\Import\ChromeImport;
use Wallabag\ImportBundle\Import\DeliciousImport;
use Wallabag\ImportBundle\Import\ElcuratorImport;
use Wallabag\ImportBundle\Import\FirefoxImport;
use Wallabag\ImportBundle\Import\InstapaperImport;
use Wallabag\ImportBundle\Import\OmnivoreImport;
use Wallabag\ImportBundle\Import\PinboardImport;
use Wallabag\ImportBundle\Import\PocketCsvImport;
use Wallabag\ImportBundle\Import\PocketHtmlImport;
use Wallabag\ImportBundle\Import\ReadabilityImport;
use Wallabag\ImportBundle\Import\ShaarliImport;
use Wallabag\ImportBundle\Import\WallabagV1Import;
use Wallabag\ImportBundle\Import\WallabagV2Import;
use Wallabag\UserBundle\Entity\User;
@ -37,9 +41,29 @@ class ImportCommand extends Command
private DeliciousImport $deliciousImport;
private OmnivoreImport $omnivoreImport;
private WallabagV1Import $wallabagV1Import;
private ElcuratorImport $elcuratorImport;
private ShaarliImport $shaarliImport;
private PocketHtmlImport $pocketHtmlImport;
private PocketCsvImport $pocketCsvImport;
public function __construct(EntityManagerInterface $entityManager, TokenStorageInterface $tokenStorage, UserRepository $userRepository, WallabagV2Import $wallabagV2Import, FirefoxImport $firefoxImport, ChromeImport $chromeImport, ReadabilityImport $readabilityImport, InstapaperImport $instapaperImport, PinboardImport $pinboardImport, DeliciousImport $deliciousImport, OmnivoreImport $omnivoreImport, WallabagV1Import $wallabagV1Import)
{
public function __construct(
EntityManagerInterface $entityManager,
TokenStorageInterface $tokenStorage,
UserRepository $userRepository,
WallabagV2Import $wallabagV2Import,
FirefoxImport $firefoxImport,
ChromeImport $chromeImport,
ReadabilityImport $readabilityImport,
InstapaperImport $instapaperImport,
PinboardImport $pinboardImport,
DeliciousImport $deliciousImport,
WallabagV1Import $wallabagV1Import,
ElcuratorImport $elcuratorImport,
OmnivoreImport $omnivoreImport,
ShaarliImport $shaarliImport,
PocketHtmlImport $pocketHtmlImport,
PocketCsvImport $pocketCsvImport
) {
$this->entityManager = $entityManager;
$this->tokenStorage = $tokenStorage;
$this->userRepository = $userRepository;
@ -52,6 +76,10 @@ class ImportCommand extends Command
$this->deliciousImport = $deliciousImport;
$this->omnivoreImport = $omnivoreImport;
$this->wallabagV1Import = $wallabagV1Import;
$this->elcuratorImport = $elcuratorImport;
$this->shaarliImport = $shaarliImport;
$this->pocketHtmlImport = $pocketHtmlImport;
$this->pocketCsvImport = $pocketCsvImport;
parent::__construct();
}
@ -63,7 +91,7 @@ class ImportCommand extends Command
->setDescription('Import entries from a JSON export')
->addArgument('username', InputArgument::REQUIRED, 'User to populate')
->addArgument('filepath', InputArgument::REQUIRED, 'Path to the JSON file')
->addOption('importer', null, InputOption::VALUE_OPTIONAL, 'The importer to use: v1, v2, instapaper, pinboard, delicious, readability, firefox or chrome', 'v1')
->addOption('importer', null, InputOption::VALUE_OPTIONAL, 'The importer to use: v1, v2, instapaper, pinboard, delicious, readability, firefox, chrome, elcurator, shaarli, pocket or pocket_csv', 'v1')
->addOption('markAsRead', null, InputOption::VALUE_OPTIONAL, 'Mark all entries as read', false)
->addOption('useUserId', null, InputOption::VALUE_NONE, 'Use user id instead of username to find account')
->addOption('disableContentUpdate', null, InputOption::VALUE_NONE, 'Disable fetching updated content from URL')
@ -125,6 +153,18 @@ class ImportCommand extends Command
break;
case 'omnivore':
$import = $this->omnivoreImport;
break;
case 'elcurator':
$import = $this->elcuratorImport;
break;
case 'shaarli':
$import = $this->shaarliImport;
break;
case 'pocket':
$import = $this->pocketHtmlImport;
break;
case 'pocket_csv':
$import = $this->pocketCsvImport;
break;
default:
$import = $this->wallabagV1Import;

View File

@ -21,9 +21,26 @@ class RabbitMQConsumerTotalProxy
private Consumer $deliciousConsumer;
private Consumer $elcuratorConsumer;
private Consumer $omnivoreConsumer;
private Consumer $shaarliConsumer;
private Consumer $pocketHtmlConsumer;
private Consumer $pocketCsvConsumer;
public function __construct(Consumer $pocketConsumer, Consumer $readabilityConsumer, Consumer $wallabagV1Consumer, Consumer $wallabagV2Consumer, Consumer $firefoxConsumer, Consumer $chromeConsumer, Consumer $instapaperConsumer, Consumer $pinboardConsumer, Consumer $deliciousConsumer, Consumer $elcuratorConsumer, Consumer $omnivoreConsumer)
{
public function __construct(
Consumer $pocketConsumer,
Consumer $readabilityConsumer,
Consumer $wallabagV1Consumer,
Consumer $wallabagV2Consumer,
Consumer $firefoxConsumer,
Consumer $chromeConsumer,
Consumer $instapaperConsumer,
Consumer $pinboardConsumer,
Consumer $deliciousConsumer,
Consumer $elcuratorConsumer,
Consumer $omnivoreConsumer,
Consumer $shaarliConsumer,
Consumer $pocketHtmlConsumer,
Consumer $pocketCsvConsumer
) {
$this->pocketConsumer = $pocketConsumer;
$this->readabilityConsumer = $readabilityConsumer;
$this->wallabagV1Consumer = $wallabagV1Consumer;
@ -35,6 +52,9 @@ class RabbitMQConsumerTotalProxy
$this->deliciousConsumer = $deliciousConsumer;
$this->elcuratorConsumer = $elcuratorConsumer;
$this->omnivoreConsumer = $omnivoreConsumer;
$this->shaarliConsumer = $shaarliConsumer;
$this->pocketHtmlConsumer = $pocketHtmlConsumer;
$this->pocketCsvConsumer = $pocketCsvConsumer;
}
/**
@ -82,6 +102,15 @@ class RabbitMQConsumerTotalProxy
case 'omnivore':
$consumer = $this->omnivoreConsumer;
break;
case 'shaarli':
$consumer = $this->shaarliConsumer;
break;
case 'pocket_html':
$consumer = $this->pocketHtmlConsumer;
break;
case 'pocket_csv':
$consumer = $this->pocketCsvConsumer;
break;
default:
return 0;
}

View File

@ -0,0 +1,83 @@
<?php
namespace Wallabag\ImportBundle\Controller;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wallabag\ImportBundle\Form\Type\UploadImportType;
use Wallabag\ImportBundle\Import\ImportInterface;
abstract class HtmlController extends AbstractController
{
/**
* @Route("/html", name="import_html")
*
* @return Response
*/
public function indexAction(Request $request, TranslatorInterface $translator)
{
$form = $this->createForm(UploadImportType::class);
$form->handleRequest($request);
$wallabag = $this->getImportService();
$wallabag->setUser($this->getUser());
if ($form->isSubmitted() && $form->isValid()) {
$file = $form->get('file')->getData();
$markAsRead = $form->get('mark_as_read')->getData();
$name = $this->getUser()->getId() . '.html';
if (null !== $file && \in_array($file->getClientMimeType(), $this->getParameter('wallabag_import.allow_mimetypes'), true) && $file->move($this->getParameter('wallabag_import.resource_dir'), $name)) {
$res = $wallabag
->setFilepath($this->getParameter('wallabag_import.resource_dir') . '/' . $name)
->setMarkAsRead($markAsRead)
->import();
$message = 'flashes.import.notice.failed';
if (true === $res) {
$summary = $wallabag->getSummary();
$message = $translator->trans('flashes.import.notice.summary', [
'%imported%' => $summary['imported'],
'%skipped%' => $summary['skipped'],
]);
if (0 < $summary['queued']) {
$message = $translator->trans('flashes.import.notice.summary_with_queue', [
'%queued%' => $summary['queued'],
]);
}
unlink($this->getParameter('wallabag_import.resource_dir') . '/' . $name);
}
$this->addFlash('notice', $message);
return $this->redirect($this->generateUrl('homepage'));
}
$this->addFlash('notice', 'flashes.import.notice.failed_on_file');
}
return $this->render($this->getImportTemplate(), [
'form' => $form->createView(),
'import' => $wallabag,
]);
}
/**
* Return the service to handle the import.
*
* @return ImportInterface
*/
abstract protected function getImportService();
/**
* Return the template used for the form.
*
* @return string
*/
abstract protected function getImportTemplate();
}

View File

@ -58,6 +58,9 @@ class ImportController extends AbstractController
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('delicious')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('elcurator')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('omnivore')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('shaarli')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('pocket_html')
+ $this->rabbitMQConsumerTotalProxy->getTotalMessage('pocket_csv')
;
} catch (\Exception $e) {
$rabbitNotInstalled = true;
@ -77,6 +80,9 @@ class ImportController extends AbstractController
+ $redis->llen('wallabag.import.delicious')
+ $redis->llen('wallabag.import.elcurator')
+ $redis->llen('wallabag.import.omnivore')
+ $redis->llen('wallabag.import.shaarli')
+ $redis->llen('wallabag.import.pocket_html')
+ $redis->llen('wallabag.import.pocket_csv')
;
} catch (\Exception $e) {
$redisNotInstalled = true;

View File

@ -0,0 +1,57 @@
<?php
namespace Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use OldSound\RabbitMqBundle\RabbitMq\Producer as RabbitMqProducer;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wallabag\ImportBundle\Import\PocketCsvImport;
use Wallabag\ImportBundle\Redis\Producer as RedisProducer;
class PocketCsvController extends HtmlController
{
private PocketCsvImport $pocketCsvImport;
private Config $craueConfig;
private RabbitMqProducer $rabbitMqProducer;
private RedisProducer $redisProducer;
public function __construct(PocketCsvImport $pocketCsvImport, Config $craueConfig, RabbitMqProducer $rabbitMqProducer, RedisProducer $redisProducer)
{
$this->pocketCsvImport = $pocketCsvImport;
$this->craueConfig = $craueConfig;
$this->rabbitMqProducer = $rabbitMqProducer;
$this->redisProducer = $redisProducer;
}
/**
* @Route("/pocket_csv", name="import_pocket_csv")
*/
public function indexAction(Request $request, TranslatorInterface $translator)
{
return parent::indexAction($request, $translator);
}
/**
* {@inheritdoc}
*/
protected function getImportService()
{
if ($this->craueConfig->get('import_with_rabbitmq')) {
$this->pocketCsvImport->setProducer($this->rabbitMqProducer);
} elseif ($this->craueConfig->get('import_with_redis')) {
$this->pocketCsvImport->setProducer($this->redisProducer);
}
return $this->pocketCsvImport;
}
/**
* {@inheritdoc}
*/
protected function getImportTemplate()
{
return '@WallabagImport/PocketCsv/index.html.twig';
}
}

View File

@ -0,0 +1,57 @@
<?php
namespace Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use OldSound\RabbitMqBundle\RabbitMq\Producer as RabbitMqProducer;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wallabag\ImportBundle\Import\PocketHtmlImport;
use Wallabag\ImportBundle\Redis\Producer as RedisProducer;
class PocketHtmlController extends HtmlController
{
private PocketHtmlImport $pocketHtmlImport;
private Config $craueConfig;
private RabbitMqProducer $rabbitMqProducer;
private RedisProducer $redisProducer;
public function __construct(PocketHtmlImport $pocketHtmlImport, Config $craueConfig, RabbitMqProducer $rabbitMqProducer, RedisProducer $redisProducer)
{
$this->pocketHtmlImport = $pocketHtmlImport;
$this->craueConfig = $craueConfig;
$this->rabbitMqProducer = $rabbitMqProducer;
$this->redisProducer = $redisProducer;
}
/**
* @Route("/pocket_html", name="import_pocket_html")
*/
public function indexAction(Request $request, TranslatorInterface $translator)
{
return parent::indexAction($request, $translator);
}
/**
* {@inheritdoc}
*/
protected function getImportService()
{
if ($this->craueConfig->get('import_with_rabbitmq')) {
$this->pocketHtmlImport->setProducer($this->rabbitMqProducer);
} elseif ($this->craueConfig->get('import_with_redis')) {
$this->pocketHtmlImport->setProducer($this->redisProducer);
}
return $this->pocketHtmlImport;
}
/**
* {@inheritdoc}
*/
protected function getImportTemplate()
{
return '@WallabagImport/PocketHtml/index.html.twig';
}
}

View File

@ -0,0 +1,57 @@
<?php
namespace Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use OldSound\RabbitMqBundle\RabbitMq\Producer as RabbitMqProducer;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Contracts\Translation\TranslatorInterface;
use Wallabag\ImportBundle\Import\ShaarliImport;
use Wallabag\ImportBundle\Redis\Producer as RedisProducer;
class ShaarliController extends HtmlController
{
private ShaarliImport $shaarliImport;
private Config $craueConfig;
private RabbitMqProducer $rabbitMqProducer;
private RedisProducer $redisProducer;
public function __construct(ShaarliImport $shaarliImport, Config $craueConfig, RabbitMqProducer $rabbitMqProducer, RedisProducer $redisProducer)
{
$this->shaarliImport = $shaarliImport;
$this->craueConfig = $craueConfig;
$this->rabbitMqProducer = $rabbitMqProducer;
$this->redisProducer = $redisProducer;
}
/**
* @Route("/shaarli", name="import_shaarli")
*/
public function indexAction(Request $request, TranslatorInterface $translator)
{
return parent::indexAction($request, $translator);
}
/**
* {@inheritdoc}
*/
protected function getImportService()
{
if ($this->craueConfig->get('import_with_rabbitmq')) {
$this->shaarliImport->setProducer($this->rabbitMqProducer);
} elseif ($this->craueConfig->get('import_with_redis')) {
$this->shaarliImport->setProducer($this->redisProducer);
}
return $this->shaarliImport;
}
/**
* {@inheritdoc}
*/
protected function getImportTemplate()
{
return '@WallabagImport/Shaarli/index.html.twig';
}
}

View File

@ -0,0 +1,210 @@
<?php
namespace Wallabag\ImportBundle\Import;
use Wallabag\CoreBundle\Entity\Entry;
use Wallabag\CoreBundle\Event\EntrySavedEvent;
abstract class HtmlImport extends AbstractImport
{
protected $filepath;
/**
* {@inheritdoc}
*/
abstract public function getName();
/**
* {@inheritdoc}
*/
abstract public function getUrl();
/**
* {@inheritdoc}
*/
abstract public function getDescription();
/**
* {@inheritdoc}
*/
public function import()
{
if (!$this->user) {
$this->logger->error('Wallabag HTML Import: user is not defined');
return false;
}
if (!file_exists($this->filepath) || !is_readable($this->filepath)) {
$this->logger->error('Wallabag HTML Import: unable to read file', ['filepath' => $this->filepath]);
return false;
}
$html = new \DOMDocument();
libxml_use_internal_errors(true);
$html->loadHTMLFile($this->filepath);
$hrefs = $html->getElementsByTagName('a');
libxml_use_internal_errors(false);
if (0 === $hrefs->length) {
$this->logger->error('Wallabag HTML: no entries in imported file');
return false;
}
$entries = [];
foreach ($hrefs as $href) {
$entry = [];
$entry['url'] = $href->getAttribute('href');
$entry['tags'] = $href->getAttribute('tags');
$entry['created_at'] = $href->getAttribute('add_date');
$entries[] = $entry;
}
if ($this->producer) {
$this->parseEntriesForProducer($entries);
return true;
}
$this->parseEntries($entries);
return true;
}
/**
* Set file path to the html file.
*
* @param string $filepath
*/
public function setFilepath($filepath)
{
$this->filepath = $filepath;
return $this;
}
/**
* {@inheritdoc}
*/
public function parseEntry(array $importedEntry)
{
$url = $importedEntry['url'];
$existingEntry = $this->em
->getRepository(Entry::class)
->findByUrlAndUserId($url, $this->user->getId());
if (false !== $existingEntry) {
++$this->skippedEntries;
return null;
}
$data = $this->prepareEntry($importedEntry);
$entry = new Entry($this->user);
$entry->setUrl($data['url']);
$entry->updateArchived($data['is_archived']);
$createdAt = new \DateTime();
$createdAt->setTimestamp($data['created_at']);
$entry->setCreatedAt($createdAt);
// update entry with content (in case fetching failed, the given entry will be return)
$this->fetchContent($entry, $data['url'], $data);
if (\array_key_exists('tags', $data)) {
$this->tagsAssigner->assignTagsToEntry(
$entry,
$data['tags']
);
}
$this->em->persist($entry);
++$this->importedEntries;
return $entry;
}
/**
* Parse and insert all given entries.
*/
protected function parseEntries(array $entries)
{
$i = 1;
$entryToBeFlushed = [];
foreach ($entries as $importedEntry) {
$entry = $this->parseEntry($importedEntry);
if (null === $entry) {
continue;
}
// @see AbstractImport
$entryToBeFlushed[] = $entry;
// flush every 20 entries
if (0 === ($i % 20)) {
$this->em->flush();
foreach ($entryToBeFlushed as $entry) {
$this->eventDispatcher->dispatch(new EntrySavedEvent($entry), EntrySavedEvent::NAME);
}
$entryToBeFlushed = [];
}
++$i;
}
$this->em->flush();
if (!empty($entryToBeFlushed)) {
foreach ($entryToBeFlushed as $entry) {
$this->eventDispatcher->dispatch(new EntrySavedEvent($entry), EntrySavedEvent::NAME);
}
}
}
/**
* Parse entries and send them to the queue.
* It should just be a simple loop on all item, no call to the database should be done
* to speedup queuing.
*
* Faster parse entries for Producer.
* We don't care to make check at this time. They'll be done by the consumer.
*/
protected function parseEntriesForProducer(array $entries)
{
foreach ($entries as $importedEntry) {
if ((array) $importedEntry !== $importedEntry) {
continue;
}
// set userId for the producer (it won't know which user is connected)
$importedEntry['userId'] = $this->user->getId();
if ($this->markAsRead) {
$importedEntry = $this->setEntryAsRead($importedEntry);
}
++$this->queuedEntries;
$this->producer->publish(json_encode($importedEntry));
}
}
/**
* {@inheritdoc}
*/
protected function setEntryAsRead(array $importedEntry)
{
$importedEntry['is_archived'] = 1;
return $importedEntry;
}
abstract protected function prepareEntry(array $entry = []);
}

View File

@ -0,0 +1,156 @@
<?php
namespace Wallabag\ImportBundle\Import;
use Wallabag\CoreBundle\Entity\Entry;
class PocketCsvImport extends AbstractImport
{
protected $filepath;
/**
* {@inheritdoc}
*/
public function getName()
{
return 'Pocket CSV';
}
/**
* {@inheritdoc}
*/
public function getUrl()
{
return 'import_pocket_csv';
}
/**
* {@inheritdoc}
*/
public function getDescription()
{
return 'import.pocket_csv.description';
}
/**
* Set file path to the csv file.
*
* @param string $filepath
*/
public function setFilepath($filepath)
{
$this->filepath = $filepath;
return $this;
}
/**
* {@inheritdoc}
*/
public function validateEntry(array $importedEntry)
{
if (empty($importedEntry['url'])) {
return false;
}
return true;
}
public function import()
{
if (!$this->user) {
$this->logger->error('Pocket CSV Import: user is not defined');
return false;
}
if (!file_exists($this->filepath) || !is_readable($this->filepath)) {
$this->logger->error('Pocket CSV Import: unable to read file', ['filepath' => $this->filepath]);
return false;
}
$entries = [];
$handle = fopen($this->filepath, 'r');
while (false !== ($data = fgetcsv($handle, 10240))) {
if ('title' === $data[0]) {
continue;
}
$entries[] = [
'url' => $data[1],
'title' => $data[0],
'is_archived' => 'archive' === $data[4],
'created_at' => $data[2],
'tags' => $data[3],
];
}
fclose($handle);
if (empty($entries)) {
$this->logger->error('PocketCsvImport: no entries in imported file');
return false;
}
if ($this->producer) {
$this->parseEntriesForProducer($entries);
return true;
}
$this->parseEntries($entries);
return true;
}
/**
* {@inheritdoc}
*/
public function parseEntry(array $importedEntry)
{
$existingEntry = $this->em
->getRepository(Entry::class)
->findByUrlAndUserId($importedEntry['url'], $this->user->getId());
if (false !== $existingEntry) {
++$this->skippedEntries;
return;
}
$entry = new Entry($this->user);
$entry->setUrl($importedEntry['url']);
$entry->setTitle($importedEntry['title']);
// update entry with content (in case fetching failed, the given entry will be return)
$this->fetchContent($entry, $importedEntry['url'], $importedEntry);
if (!empty($importedEntry['tags'])) {
$tags = str_replace('|', ',', $importedEntry['tags']);
$this->tagsAssigner->assignTagsToEntry(
$entry,
$tags,
$this->em->getUnitOfWork()->getScheduledEntityInsertions()
);
}
$entry->updateArchived($importedEntry['is_archived']);
$entry->setCreatedAt(\DateTime::createFromFormat('U', $importedEntry['created_at']));
$this->em->persist($entry);
++$this->importedEntries;
return $entry;
}
/**
* {@inheritdoc}
*/
protected function setEntryAsRead(array $importedEntry)
{
$importedEntry['is_archived'] = 'archive';
return $importedEntry;
}
}

View File

@ -0,0 +1,113 @@
<?php
namespace Wallabag\ImportBundle\Import;
class PocketHtmlImport extends HtmlImport
{
protected $filepath;
/**
* {@inheritdoc}
*/
public function getName()
{
return 'Pocket HTML';
}
/**
* {@inheritdoc}
*/
public function getUrl()
{
return 'import_pocket_html';
}
/**
* {@inheritdoc}
*/
public function getDescription()
{
return 'import.pocket_html.description';
}
/**
* {@inheritdoc}
*/
public function validateEntry(array $importedEntry)
{
if (empty($importedEntry['url'])) {
return false;
}
return true;
}
public function import()
{
if (!$this->user) {
$this->logger->error('Pocket HTML Import: user is not defined');
return false;
}
if (!file_exists($this->filepath) || !is_readable($this->filepath)) {
$this->logger->error('Pocket HTML Import: unable to read file', ['filepath' => $this->filepath]);
return false;
}
$html = new \DOMDocument();
libxml_use_internal_errors(true);
$html->loadHTMLFile($this->filepath);
$hrefs = $html->getElementsByTagName('a');
libxml_use_internal_errors(false);
if (0 === $hrefs->length) {
$this->logger->error('Pocket HTML: no entries in imported file');
return false;
}
$entries = [];
foreach ($hrefs as $href) {
$entry = [];
$entry['url'] = $href->getAttribute('href');
$entry['tags'] = $href->getAttribute('tags');
$entry['created_at'] = $href->getAttribute('time_added');
$entries[] = $entry;
}
if ($this->producer) {
$this->parseEntriesForProducer($entries);
return true;
}
$this->parseEntries($entries);
return true;
}
/**
* {@inheritdoc}
*/
protected function prepareEntry(array $entry = [])
{
$data = [
'title' => '',
'html' => false,
'url' => $entry['url'],
'is_archived' => (int) $this->markAsRead,
'is_starred' => false,
'tags' => '',
'created_at' => $entry['created_at'],
];
if (\array_key_exists('tags', $entry) && '' !== $entry['tags']) {
$data['tags'] = $entry['tags'];
}
return $data;
}
}

View File

@ -0,0 +1,66 @@
<?php
namespace Wallabag\ImportBundle\Import;
class ShaarliImport extends HtmlImport
{
protected $filepath;
/**
* {@inheritdoc}
*/
public function getName()
{
return 'Shaarli';
}
/**
* {@inheritdoc}
*/
public function getUrl()
{
return 'import_shaarli';
}
/**
* {@inheritdoc}
*/
public function getDescription()
{
return 'import.shaarli.description';
}
/**
* {@inheritdoc}
*/
public function validateEntry(array $importedEntry)
{
if (empty($importedEntry['url'])) {
return false;
}
return true;
}
/**
* {@inheritdoc}
*/
protected function prepareEntry(array $entry = [])
{
$data = [
'title' => '',
'html' => false,
'url' => $entry['url'],
'is_archived' => (int) $this->markAsRead,
'is_starred' => false,
'tags' => '',
'created_at' => $entry['created_at'],
];
if (\array_key_exists('tags', $entry) && '' !== $entry['tags']) {
$data['tags'] = $entry['tags'];
}
return $data;
}
}

View File

@ -0,0 +1,45 @@
{% extends "@WallabagCore/layout.html.twig" %}
{% block title %}{{ 'import.pocket_csv.page_title'|trans }}{% endblock %}
{% block content %}
<div class="row">
<div class="col s12">
<div class="card-panel settings">
{% include '@WallabagImport/Import/_information.html.twig' %}
<div class="row">
<blockquote>{{ import.description|trans|raw }}</blockquote>
<p>{{ 'import.pocket_csv.how_to'|trans }}</p>
<div class="col s12">
{{ form_start(form, {'method': 'POST'}) }}
{{ form_errors(form) }}
<div class="row">
<div class="file-field input-field col s12">
{{ form_errors(form.file) }}
<div class="btn">
<span>{{ form.file.vars.label|trans }}</span>
{{ form_widget(form.file) }}
</div>
<div class="file-path-wrapper">
<input class="file-path validate" type="text">
</div>
</div>
<div class="input-field col s6 with-checkbox">
<h6>{{ 'import.form.mark_as_read_title'|trans }}</h6>
{{ form_widget(form.mark_as_read) }}
{{ form_label(form.mark_as_read) }}
</div>
</div>
{{ form_widget(form.save, {'attr': {'class': 'btn waves-effect waves-light'}}) }}
{{ form_rest(form) }}
</form>
</div>
</div>
</div>
</div>
</div>
{% endblock %}

View File

@ -0,0 +1,45 @@
{% extends "@WallabagCore/layout.html.twig" %}
{% block title %}{{ 'import.pocket_html.page_title'|trans }}{% endblock %}
{% block content %}
<div class="row">
<div class="col s12">
<div class="card-panel settings">
{% include '@WallabagImport/Import/_information.html.twig' %}
<div class="row">
<blockquote>{{ import.description|trans|raw }}</blockquote>
<p>{{ 'import.pocket_html.how_to'|trans }}</p>
<div class="col s12">
{{ form_start(form, {'method': 'POST'}) }}
{{ form_errors(form) }}
<div class="row">
<div class="file-field input-field col s12">
{{ form_errors(form.file) }}
<div class="btn">
<span>{{ form.file.vars.label|trans }}</span>
{{ form_widget(form.file) }}
</div>
<div class="file-path-wrapper">
<input class="file-path validate" type="text">
</div>
</div>
<div class="input-field col s6 with-checkbox">
<h6>{{ 'import.form.mark_as_read_title'|trans }}</h6>
{{ form_widget(form.mark_as_read) }}
{{ form_label(form.mark_as_read) }}
</div>
</div>
{{ form_widget(form.save, {'attr': {'class': 'btn waves-effect waves-light'}}) }}
{{ form_rest(form) }}
</form>
</div>
</div>
</div>
</div>
</div>
{% endblock %}

View File

@ -0,0 +1,45 @@
{% extends "@WallabagCore/layout.html.twig" %}
{% block title %}{{ 'import.shaarli.page_title'|trans }}{% endblock %}
{% block content %}
<div class="row">
<div class="col s12">
<div class="card-panel settings">
{% include '@WallabagImport/Import/_information.html.twig' %}
<div class="row">
<blockquote>{{ import.description|trans|raw }}</blockquote>
<p>{{ 'import.shaarli.how_to'|trans }}</p>
<div class="col s12">
{{ form_start(form, {'method': 'POST'}) }}
{{ form_errors(form) }}
<div class="row">
<div class="file-field input-field col s12">
{{ form_errors(form.file) }}
<div class="btn">
<span>{{ form.file.vars.label|trans }}</span>
{{ form_widget(form.file) }}
</div>
<div class="file-path-wrapper">
<input class="file-path validate" type="text">
</div>
</div>
<div class="input-field col s6 with-checkbox">
<h6>{{ 'import.form.mark_as_read_title'|trans }}</h6>
{{ form_widget(form.mark_as_read) }}
{{ form_label(form.mark_as_read) }}
</div>
</div>
{{ form_widget(form.save, {'attr': {'class': 'btn waves-effect waves-light'}}) }}
{{ form_rest(form) }}
</form>
</div>
</div>
</div>
</div>
</div>
{% endblock %}

View File

@ -123,9 +123,9 @@ class FirefoxControllerTest extends WallabagCoreTestCase
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for http://lexpansion.lexpress.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for http://lexpansion.lexpress.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for http://lexpansion.lexpress.fr is ok');
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
$this->assertCount(3, $content->getTags());
}

View File

@ -24,6 +24,6 @@ class ImportControllerTest extends WallabagCoreTestCase
$crawler = $client->request('GET', '/import/');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(11, $crawler->filter('blockquote')->count());
$this->assertSame(14, $crawler->filter('blockquote')->count());
}
}

View File

@ -0,0 +1,168 @@
<?php
namespace Tests\Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use Doctrine\ORM\EntityManagerInterface;
use Predis\Client;
use Symfony\Component\HttpFoundation\File\UploadedFile;
use Tests\Wallabag\CoreBundle\WallabagCoreTestCase;
use Wallabag\CoreBundle\Entity\Entry;
class PocketCsvControllerTest extends WallabagCoreTestCase
{
public function testImportPocketCsv()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_csv');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
}
public function testImportPocketCsvWithRabbitEnabled()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 1);
$crawler = $client->request('GET', '/import/pocket_csv');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 0);
}
public function testImportPocketCsvBadFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_csv');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$data = [
'upload_import_file[file]' => '',
];
$client->submit($form, $data);
$this->assertSame(200, $client->getResponse()->getStatusCode());
}
public function testImportPocketCsvWithRedisEnabled()
{
$this->checkRedis();
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_redis', 1);
$crawler = $client->request('GET', '/import/pocket_csv');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/pocket.csv', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$this->assertNotEmpty($client->getContainer()->get(Client::class)->lpop('wallabag.import.pocket_csv'));
$client->getContainer()->get(Config::class)->set('import_with_redis', 0);
}
public function testImportWallabagWithPocketCsvFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_csv');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/pocket.csv', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$entries = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findBy(['user' => $this->getLoggedInUserId()]);
$expectedEntries = [
'http://youmightnotneedjquery.com/,1600322788',
'https://jp-lambert.me/est-ce-que-jai-besoin-d-un-scrum-master-604f5a471c73',
'https://www.monde-diplomatique.fr/2020/09/HALIMI/62165',
'https://www.reddit.com/r/DataHoarder/comments/ioupbk/archivebox_question_how_do_i_import_links_from_a/',
'https://www.numerama.com/politique/646826-tu-vas-pleurer-les-premieres-fois-que-se-passe-t-il-au-sein-du-studio-dubisoft-derriere-trackmania.html#utm_medium=distibuted&utm_source=rss&utm_campaign=646826',
'https://www.nouvelobs.com/rue89/20200911.OBS33165/comment-konbini-s-est-fait-pieger-par-un-pere-masculiniste.html',
'https://reporterre.net/Des-abeilles-pour-resoudre-les-conflits-entre-les-humains-et-les-elephants',
];
$matchedEntries = array_map(function ($expectedUrl) use ($entries) {
foreach ($entries as $entry) {
if ($entry->getUrl() === $expectedUrl) {
return $entry;
}
}
return null;
}, $expectedEntries);
$this->assertCount(\count($expectedEntries), $matchedEntries, 'Should have 7 entries imported from pocket.csv');
}
public function testImportWallabagWithEmptyFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_csv');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/test.csv', 'test.csv');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.failed', $body[0]);
}
}

View File

@ -0,0 +1,168 @@
<?php
namespace Tests\Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use Doctrine\ORM\EntityManagerInterface;
use Predis\Client;
use Symfony\Component\HttpFoundation\File\UploadedFile;
use Tests\Wallabag\CoreBundle\WallabagCoreTestCase;
use Wallabag\CoreBundle\Entity\Entry;
class PocketHtmlControllerTest extends WallabagCoreTestCase
{
public function testImportPocketHtml()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_html');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
}
public function testImportPocketHtmlWithRabbitEnabled()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 1);
$crawler = $client->request('GET', '/import/pocket_html');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 0);
}
public function testImportPocketHtmlBadFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_html');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$data = [
'upload_import_file[file]' => '',
];
$client->submit($form, $data);
$this->assertSame(200, $client->getResponse()->getStatusCode());
}
public function testImportPocketHtmlWithRedisEnabled()
{
$this->checkRedis();
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_redis', 1);
$crawler = $client->request('GET', '/import/pocket_html');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/ril_export.html', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$this->assertNotEmpty($client->getContainer()->get(Client::class)->lpop('wallabag.import.pocket_html'));
$client->getContainer()->get(Config::class)->set('import_with_redis', 0);
}
public function testImportWallabagWithPocketHtmlFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_html');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/ril_export.html', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$content = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findByUrlAndUserId(
'https://www.20minutes.fr/sport/4002755-20220928-tarn-lapins-ravagent-terrain-match-rugby-doit-etre-annule',
$this->getLoggedInUserId()
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
$this->assertCount(3, $content->getTags());
$content = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findByUrlAndUserId(
'https://www.20minutes.fr/paris/4100740-20240715-jo-paris-2024-courir-capitale-maintenant-quais-fermes',
$this->getLoggedInUserId()
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
}
public function testImportWallabagWithEmptyFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/pocket_html');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/test.html', 'test.html');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.failed', $body[0]);
}
}

View File

@ -0,0 +1,168 @@
<?php
namespace Tests\Wallabag\ImportBundle\Controller;
use Craue\ConfigBundle\Util\Config;
use Doctrine\ORM\EntityManagerInterface;
use Predis\Client;
use Symfony\Component\HttpFoundation\File\UploadedFile;
use Tests\Wallabag\CoreBundle\WallabagCoreTestCase;
use Wallabag\CoreBundle\Entity\Entry;
class ShaarliControllerTest extends WallabagCoreTestCase
{
public function testImportShaarli()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/shaarli');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
}
public function testImportShaarliWithRabbitEnabled()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 1);
$crawler = $client->request('GET', '/import/shaarli');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$client->getContainer()->get(Config::class)->set('import_with_rabbitmq', 0);
}
public function testImportShaarliBadFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/shaarli');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$data = [
'upload_import_file[file]' => '',
];
$client->submit($form, $data);
$this->assertSame(200, $client->getResponse()->getStatusCode());
}
public function testImportShaarliWithRedisEnabled()
{
$this->checkRedis();
$this->logInAs('admin');
$client = $this->getTestClient();
$client->getContainer()->get(Config::class)->set('import_with_redis', 1);
$crawler = $client->request('GET', '/import/shaarli');
$this->assertSame(200, $client->getResponse()->getStatusCode());
$this->assertSame(1, $crawler->filter('form[name=upload_import_file] > button[type=submit]')->count());
$this->assertSame(1, $crawler->filter('input[type=file]')->count());
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/shaarli-bookmarks.html', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$this->assertNotEmpty($client->getContainer()->get(Client::class)->lpop('wallabag.import.shaarli'));
$client->getContainer()->get(Config::class)->set('import_with_redis', 0);
}
public function testImportWallabagWithShaarliFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/shaarli');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/shaarli-bookmarks.html', 'Bookmarks');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.summary', $body[0]);
$content = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findByUrlAndUserId(
'https://www.20minutes.fr/sport/4002755-20220928-tarn-lapins-ravagent-terrain-match-rugby-doit-etre-annule',
$this->getLoggedInUserId()
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
$this->assertCount(2, $content->getTags());
$content = $client->getContainer()
->get(EntityManagerInterface::class)
->getRepository(Entry::class)
->findByUrlAndUserId(
'https://www.20minutes.fr/paris/4100740-20240715-jo-paris-2024-courir-capitale-maintenant-quais-fermes',
$this->getLoggedInUserId()
);
$this->assertInstanceOf(Entry::class, $content);
$this->assertNotEmpty($content->getMimetype(), 'Mimetype for 20minutes.fr is ok');
$this->assertNotEmpty($content->getPreviewPicture(), 'Preview picture for 20minutes.fr is ok');
$this->assertNotEmpty($content->getLanguage(), 'Language for 20minutes.fr is ok');
}
public function testImportWallabagWithEmptyFile()
{
$this->logInAs('admin');
$client = $this->getTestClient();
$crawler = $client->request('GET', '/import/shaarli');
$form = $crawler->filter('form[name=upload_import_file] > button[type=submit]')->form();
$file = new UploadedFile(__DIR__ . '/../fixtures/test.html', 'test.html');
$data = [
'upload_import_file[file]' => $file,
];
$client->submit($form, $data);
$this->assertSame(302, $client->getResponse()->getStatusCode());
$crawler = $client->followRedirect();
$this->assertGreaterThan(1, $body = $crawler->filter('body')->extract(['_text']));
$this->assertStringContainsString('flashes.import.notice.failed', $body[0]);
}
}

View File

@ -0,0 +1,252 @@
<?php
namespace Tests\Wallabag\ImportBundle\Import;
use Doctrine\ORM\EntityManager;
use M6Web\Component\RedisMock\RedisMockFactory;
use Monolog\Handler\TestHandler;
use Monolog\Logger;
use PHPUnit\Framework\TestCase;
use Predis\Client;
use Simpleue\Queue\RedisQueue;
use Symfony\Component\EventDispatcher\EventDispatcher;
use Wallabag\CoreBundle\Entity\Entry;
use Wallabag\CoreBundle\Helper\ContentProxy;
use Wallabag\CoreBundle\Helper\TagsAssigner;
use Wallabag\CoreBundle\Repository\EntryRepository;
use Wallabag\ImportBundle\Import\PocketCsvImport;
use Wallabag\ImportBundle\Redis\Producer;
use Wallabag\UserBundle\Entity\User;
class PocketCsvImportTest extends TestCase
{
protected $user;
protected $em;
protected $logHandler;
protected $contentProxy;
protected $tagsAssigner;
public function testInit()
{
$pocketCsvImport = $this->getPocketCsvImport();
$this->assertSame('Pocket CSV', $pocketCsvImport->getName());
$this->assertNotEmpty($pocketCsvImport->getUrl());
$this->assertSame('import.pocket_csv.description', $pocketCsvImport->getDescription());
}
public function testImport()
{
$pocketCsvImport = $this->getPocketCsvImport(false, 7);
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(7))
->method('findByUrlAndUserId')
->willReturn(false);
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->exactly(7))
->method('updateEntry')
->willReturn($entry);
$res = $pocketCsvImport->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 7, 'queued' => 0], $pocketCsvImport->getSummary());
}
public function testImportAndMarkAllAsRead()
{
$pocketCsvImport = $this->getPocketCsvImport(false, 1);
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(7))
->method('findByUrlAndUserId')
->will($this->onConsecutiveCalls(false, true));
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$this->contentProxy
->expects($this->exactly(1))
->method('updateEntry')
->willReturn(new Entry($this->user));
// check that every entry persisted are archived
$this->em
->expects($this->any())
->method('persist')
->with($this->callback(fn ($persistedEntry) => (bool) $persistedEntry->isArchived()));
$res = $pocketCsvImport
->setMarkAsRead(true)
->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 6, 'imported' => 1, 'queued' => 0], $pocketCsvImport->getSummary());
}
public function testImportWithRabbit()
{
$pocketCsvImport = $this->getPocketCsvImport();
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$producer = $this->getMockBuilder(\OldSound\RabbitMqBundle\RabbitMq\Producer::class)
->disableOriginalConstructor()
->getMock();
$producer
->expects($this->exactly(7))
->method('publish');
$pocketCsvImport->setProducer($producer);
$res = $pocketCsvImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 7], $pocketCsvImport->getSummary());
}
public function testImportWithRedis()
{
$pocketCsvImport = $this->getPocketCsvImport();
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$factory = new RedisMockFactory();
$redisMock = $factory->getAdapter(Client::class, true);
$queue = new RedisQueue($redisMock, 'pocket_csv');
$producer = new Producer($queue);
$pocketCsvImport->setProducer($producer);
$res = $pocketCsvImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 7], $pocketCsvImport->getSummary());
$this->assertNotEmpty($redisMock->lpop('pocket_csv'));
}
public function testImportBadFile()
{
$pocketCsvImport = $this->getPocketCsvImport();
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/Import/wallabag-v1.jsonx');
$res = $pocketCsvImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Pocket CSV Import: unable to read file', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
public function testImportUserNotDefined()
{
$pocketCsvImport = $this->getPocketCsvImport(true);
$pocketCsvImport->setFilepath(__DIR__ . '/../fixtures/pocket.csv');
$res = $pocketCsvImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Pocket CSV Import: user is not defined', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
private function getPocketCsvImport($unsetUser = false, $dispatched = 0)
{
$this->user = new User();
$this->em = $this->getMockBuilder(EntityManager::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy = $this->getMockBuilder(ContentProxy::class)
->disableOriginalConstructor()
->getMock();
$this->tagsAssigner = $this->getMockBuilder(TagsAssigner::class)
->disableOriginalConstructor()
->getMock();
$dispatcher = $this->getMockBuilder(EventDispatcher::class)
->disableOriginalConstructor()
->getMock();
$dispatcher
->expects($this->exactly($dispatched))
->method('dispatch');
$this->logHandler = new TestHandler();
$logger = new Logger('test', [$this->logHandler]);
$wallabag = new PocketCsvImport($this->em, $this->contentProxy, $this->tagsAssigner, $dispatcher, $logger);
if (false === $unsetUser) {
$wallabag->setUser($this->user);
}
return $wallabag;
}
}

View File

@ -0,0 +1,254 @@
<?php
namespace Tests\Wallabag\ImportBundle\Import;
use Doctrine\ORM\EntityManager;
use M6Web\Component\RedisMock\RedisMockFactory;
use Monolog\Handler\TestHandler;
use Monolog\Logger;
use PHPUnit\Framework\TestCase;
use Predis\Client;
use Simpleue\Queue\RedisQueue;
use Symfony\Component\EventDispatcher\EventDispatcher;
use Wallabag\CoreBundle\Entity\Entry;
use Wallabag\CoreBundle\Helper\ContentProxy;
use Wallabag\CoreBundle\Helper\TagsAssigner;
use Wallabag\CoreBundle\Repository\EntryRepository;
use Wallabag\ImportBundle\Import\PocketHtmlImport;
use Wallabag\ImportBundle\Redis\Producer;
use Wallabag\UserBundle\Entity\User;
class PocketHtmlImportTest extends TestCase
{
protected $user;
protected $em;
protected $logHandler;
protected $contentProxy;
protected $tagsAssigner;
public function testInit()
{
$pocketHtmlImport = $this->getPocketHtmlImport();
$this->assertSame('Pocket HTML', $pocketHtmlImport->getName());
$this->assertNotEmpty($pocketHtmlImport->getUrl());
$this->assertSame('import.pocket_html.description', $pocketHtmlImport->getDescription());
}
public function testImport()
{
$pocketHtmlImport = $this->getPocketHtmlImport(false, 2);
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(2))
->method('findByUrlAndUserId')
->willReturn(false);
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->exactly(2))
->method('updateEntry')
->willReturn($entry);
$res = $pocketHtmlImport->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 2, 'queued' => 0], $pocketHtmlImport->getSummary());
}
public function testImportAndMarkAllAsRead()
{
$pocketHtmlImport = $this->getPocketHtmlImport(false, 1);
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(2))
->method('findByUrlAndUserId')
->will($this->onConsecutiveCalls(false, true));
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$this->contentProxy
->expects($this->exactly(1))
->method('updateEntry')
->willReturn(new Entry($this->user));
// check that every entry persisted are archived
$this->em
->expects($this->any())
->method('persist')
->with($this->callback(function ($persistedEntry) {
return (bool) $persistedEntry->isArchived();
}));
$res = $pocketHtmlImport
->setMarkAsRead(true)
->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 1, 'imported' => 1, 'queued' => 0], $pocketHtmlImport->getSummary());
}
public function testImportWithRabbit()
{
$pocketHtmlImport = $this->getPocketHtmlImport();
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$producer = $this->getMockBuilder(\OldSound\RabbitMqBundle\RabbitMq\Producer::class)
->disableOriginalConstructor()
->getMock();
$producer
->expects($this->exactly(2))
->method('publish');
$pocketHtmlImport->setProducer($producer);
$res = $pocketHtmlImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 2], $pocketHtmlImport->getSummary());
}
public function testImportWithRedis()
{
$pocketHtmlImport = $this->getPocketHtmlImport();
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$factory = new RedisMockFactory();
$redisMock = $factory->getAdapter(Client::class, true);
$queue = new RedisQueue($redisMock, 'pocket_html');
$producer = new Producer($queue);
$pocketHtmlImport->setProducer($producer);
$res = $pocketHtmlImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 2], $pocketHtmlImport->getSummary());
$this->assertNotEmpty($redisMock->lpop('pocket_html'));
}
public function testImportBadFile()
{
$pocketHtmlImport = $this->getPocketHtmlImport();
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/wallabag-v1.jsonx');
$res = $pocketHtmlImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Pocket HTML Import: unable to read file', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
public function testImportUserNotDefined()
{
$pocketHtmlImport = $this->getPocketHtmlImport(true);
$pocketHtmlImport->setFilepath(__DIR__ . '/../fixtures/ril_export.html');
$res = $pocketHtmlImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Pocket HTML Import: user is not defined', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
private function getPocketHtmlImport($unsetUser = false, $dispatched = 0)
{
$this->user = new User();
$this->em = $this->getMockBuilder(EntityManager::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy = $this->getMockBuilder(ContentProxy::class)
->disableOriginalConstructor()
->getMock();
$this->tagsAssigner = $this->getMockBuilder(TagsAssigner::class)
->disableOriginalConstructor()
->getMock();
$dispatcher = $this->getMockBuilder(EventDispatcher::class)
->disableOriginalConstructor()
->getMock();
$dispatcher
->expects($this->exactly($dispatched))
->method('dispatch');
$this->logHandler = new TestHandler();
$logger = new Logger('test', [$this->logHandler]);
$wallabag = new PocketHtmlImport($this->em, $this->contentProxy, $this->tagsAssigner, $dispatcher, $logger);
if (false === $unsetUser) {
$wallabag->setUser($this->user);
}
return $wallabag;
}
}

View File

@ -0,0 +1,254 @@
<?php
namespace Tests\Wallabag\ImportBundle\Import;
use Doctrine\ORM\EntityManager;
use M6Web\Component\RedisMock\RedisMockFactory;
use Monolog\Handler\TestHandler;
use Monolog\Logger;
use PHPUnit\Framework\TestCase;
use Predis\Client;
use Simpleue\Queue\RedisQueue;
use Symfony\Component\EventDispatcher\EventDispatcher;
use Wallabag\CoreBundle\Entity\Entry;
use Wallabag\CoreBundle\Helper\ContentProxy;
use Wallabag\CoreBundle\Helper\TagsAssigner;
use Wallabag\CoreBundle\Repository\EntryRepository;
use Wallabag\ImportBundle\Import\ShaarliImport;
use Wallabag\ImportBundle\Redis\Producer;
use Wallabag\UserBundle\Entity\User;
class ShaarliImportTest extends TestCase
{
protected $user;
protected $em;
protected $logHandler;
protected $contentProxy;
protected $tagsAssigner;
public function testInit()
{
$shaarliImport = $this->getShaarliImport();
$this->assertSame('Shaarli', $shaarliImport->getName());
$this->assertNotEmpty($shaarliImport->getUrl());
$this->assertSame('import.shaarli.description', $shaarliImport->getDescription());
}
public function testImport()
{
$shaarliImport = $this->getShaarliImport(false, 2);
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(2))
->method('findByUrlAndUserId')
->willReturn(false);
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->exactly(2))
->method('updateEntry')
->willReturn($entry);
$res = $shaarliImport->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 2, 'queued' => 0], $shaarliImport->getSummary());
}
public function testImportAndMarkAllAsRead()
{
$shaarliImport = $this->getShaarliImport(false, 1);
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->exactly(2))
->method('findByUrlAndUserId')
->will($this->onConsecutiveCalls(false, true));
$this->em
->expects($this->any())
->method('getRepository')
->willReturn($entryRepo);
$this->contentProxy
->expects($this->exactly(1))
->method('updateEntry')
->willReturn(new Entry($this->user));
// check that every entry persisted are archived
$this->em
->expects($this->any())
->method('persist')
->with($this->callback(function ($persistedEntry) {
return (bool) $persistedEntry->isArchived();
}));
$res = $shaarliImport
->setMarkAsRead(true)
->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 1, 'imported' => 1, 'queued' => 0], $shaarliImport->getSummary());
}
public function testImportWithRabbit()
{
$shaarliImport = $this->getShaarliImport();
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$producer = $this->getMockBuilder(\OldSound\RabbitMqBundle\RabbitMq\Producer::class)
->disableOriginalConstructor()
->getMock();
$producer
->expects($this->exactly(2))
->method('publish');
$shaarliImport->setProducer($producer);
$res = $shaarliImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 2], $shaarliImport->getSummary());
}
public function testImportWithRedis()
{
$shaarliImport = $this->getShaarliImport();
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$entryRepo = $this->getMockBuilder(EntryRepository::class)
->disableOriginalConstructor()
->getMock();
$entryRepo->expects($this->never())
->method('findByUrlAndUserId');
$this->em
->expects($this->never())
->method('getRepository');
$entry = $this->getMockBuilder(Entry::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy
->expects($this->never())
->method('updateEntry');
$factory = new RedisMockFactory();
$redisMock = $factory->getAdapter(Client::class, true);
$queue = new RedisQueue($redisMock, 'shaarli');
$producer = new Producer($queue);
$shaarliImport->setProducer($producer);
$res = $shaarliImport->setMarkAsRead(true)->import();
$this->assertTrue($res);
$this->assertSame(['skipped' => 0, 'imported' => 0, 'queued' => 2], $shaarliImport->getSummary());
$this->assertNotEmpty($redisMock->lpop('shaarli'));
}
public function testImportBadFile()
{
$shaarliImport = $this->getShaarliImport();
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/wallabag-v1.jsonx');
$res = $shaarliImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Wallabag HTML Import: unable to read file', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
public function testImportUserNotDefined()
{
$shaarliImport = $this->getShaarliImport(true);
$shaarliImport->setFilepath(__DIR__ . '/../fixtures/shaarli-bookmarks.html');
$res = $shaarliImport->import();
$this->assertFalse($res);
$records = $this->logHandler->getRecords();
$this->assertStringContainsString('Wallabag HTML Import: user is not defined', $records[0]['message']);
$this->assertSame('ERROR', $records[0]['level_name']);
}
private function getShaarliImport($unsetUser = false, $dispatched = 0)
{
$this->user = new User();
$this->em = $this->getMockBuilder(EntityManager::class)
->disableOriginalConstructor()
->getMock();
$this->contentProxy = $this->getMockBuilder(ContentProxy::class)
->disableOriginalConstructor()
->getMock();
$this->tagsAssigner = $this->getMockBuilder(TagsAssigner::class)
->disableOriginalConstructor()
->getMock();
$dispatcher = $this->getMockBuilder(EventDispatcher::class)
->disableOriginalConstructor()
->getMock();
$dispatcher
->expects($this->exactly($dispatched))
->method('dispatch');
$this->logHandler = new TestHandler();
$logger = new Logger('test', [$this->logHandler]);
$wallabag = new ShaarliImport($this->em, $this->contentProxy, $this->tagsAssigner, $dispatcher, $logger);
if (false === $unsetUser) {
$wallabag->setUser($this->user);
}
return $wallabag;
}
}

View File

@ -0,0 +1,10 @@
title,url,time_added,tags,status
You Might Not Need jQuery,http://youmightnotneedjquery.com/,1600322788,,unread
Est-ce que jai besoin dun Scrum Master ? | by Jean-Pierre Lambert | Jean-,https://jp-lambert.me/est-ce-que-jai-besoin-d-un-scrum-master-604f5a471c73,1600172739,,unread
"Avec les accusés dEl Halia, par Gisèle Halimi (Le Monde diplomatique, sept",https://www.monde-diplomatique.fr/2020/09/HALIMI/62165,1599806347,,unread
ArchiveBox question: How do I import links from a RSS feed?,https://www.reddit.com/r/DataHoarder/comments/ioupbk/archivebox_question_how_do_i_import_links_from_a/,1600961496,,archive
« Tu vas pleurer les premières fois » : que se passe-t-il au sein du studio,https://www.numerama.com/politique/646826-tu-vas-pleurer-les-premieres-fois-que-se-passe-t-il-au-sein-du-studio-dubisoft-derriere-trackmania.html#utm_medium=distibuted&utm_source=rss&utm_campaign=646826,1599809025,,unread
Comment Konbini sest fait piéger par un « père masculiniste »,https://www.nouvelobs.com/rue89/20200911.OBS33165/comment-konbini-s-est-fait-pieger-par-un-pere-masculiniste.html,1599819251,,archive
"Des abeilles pour résoudre les « conflits » entre les humains
et les élépha",https://reporterre.net/Des-abeilles-pour-resoudre-les-conflits-entre-les-humains-et-les-elephants,1599890673,,unread
1 title url time_added tags status
2 You Might Not Need jQuery http://youmightnotneedjquery.com/ 1600322788 unread
3 Est-ce que j’ai besoin d’un Scrum Master ? | by Jean-Pierre Lambert | Jean- https://jp-lambert.me/est-ce-que-jai-besoin-d-un-scrum-master-604f5a471c73 1600172739 unread
4 Avec les accusés d’El Halia, par Gisèle Halimi (Le Monde diplomatique, sept https://www.monde-diplomatique.fr/2020/09/HALIMI/62165 1599806347 unread
5 ArchiveBox question: How do I import links from a RSS feed? https://www.reddit.com/r/DataHoarder/comments/ioupbk/archivebox_question_how_do_i_import_links_from_a/ 1600961496 archive
6 « Tu vas pleurer les premières fois » : que se passe-t-il au sein du studio https://www.numerama.com/politique/646826-tu-vas-pleurer-les-premieres-fois-que-se-passe-t-il-au-sein-du-studio-dubisoft-derriere-trackmania.html#utm_medium=distibuted&utm_source=rss&utm_campaign=646826 1599809025 unread
7 Comment Konbini s’est fait piéger par un « père masculiniste » https://www.nouvelobs.com/rue89/20200911.OBS33165/comment-konbini-s-est-fait-pieger-par-un-pere-masculiniste.html 1599819251 archive
8 Des abeilles pour résoudre les « conflits » entre les humains et les élépha https://reporterre.net/Des-abeilles-pour-resoudre-les-conflits-entre-les-humains-et-les-elephants 1599890673 unread

View File

@ -0,0 +1,21 @@
<!DOCTYPE html>
<html>
<!--So long and thanks for all the fish-->
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Pocket Export</title>
</head>
<body>
<h1>Unread</h1>
<ul>
<li><a href="https://www.20minutes.fr/sport/4002755-20220928-tarn-lapins-ravagent-terrain-match-rugby-doit-etre-annule" time_added="1688628695" tags="ifttt,new_entry_simple">Tarn : Des lapins ravagent le terrain, le match de rugby doit être annulé</a></li>
<li><a href="https://www.20minutes.fr/paris/4100740-20240715-jo-paris-2024-courir-capitale-maintenant-quais-fermes" time_added="1688627412" tags="ifttt,new_entry_simple">JO Paris 2024 : Où courir dans la capitale maintenant que les quais sont fermés ?</a></li>
</ul>
<h1>Read Archive</h1>
<ul>
</ul>
</body>
</html>

View File

@ -0,0 +1,13 @@
<!DOCTYPE NETSCAPE-Bookmark-file-1>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
<!-- This is an automatically generated file.
It will be read and overwritten.
Do Not Edit! -->
<TITLE>Bookmarks</TITLE>
<H1>Shaarli export of all bookmarks on Mon, 17 Jul 23 14:31:25 +0200</H1>
<DL><p>
<DT><A HREF="https://www.20minutes.fr/sport/4002755-20220928-tarn-lapins-ravagent-terrain-match-rugby-doit-etre-annule" ADD_DATE="1686813518" LAST_MODIFIED="1686813519" PRIVATE="0" TAGS="firefoxos">The Legacy of Firefox OS. In the two years or so since Mozilla… | by Ben Francis | Medium</A>
<DD>In the two years or so since Mozilla announced the end of Firefox OS as a Mozilla-run project, the B2G source code has found its way into a surprising number of commercial products.
<DT><A HREF="https://www.20minutes.fr/paris/4100740-20240715-jo-paris-2024-courir-capitale-maintenant-quais-fermes" ADD_DATE="1683376565" LAST_MODIFIED="1686813519" PRIVATE="0" TAGS="firefoxos">JO Paris 2024 : Où courir dans la capitale maintenant que les quais sont fermés ?</A>
</DL><p>
<script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script>

View File

@ -534,6 +534,18 @@ import:
page_title: Import > del.icio.us
description: This importer will import all your Delicious bookmarks. Since 2021, you can export again your data from it using the export page (https://del.icio.us/export). Choose the "JSON" format and download it (like "delicious_export.2021.02.06_21.10.json").
how_to: Please select your Delicious export and click on the button below to upload and import it.
shaarli:
page_title: Import > Shaarli
description: This importer will import all your Shaarli bookmarks. Just go to the Tools section, then into "Export database", choose your bookmarks and export them. You will obtain a HTML file.
how_to: Please choose the bookmark backup file and click on the button below to import it. Note that the process may take a long time since all articles have to be fetched.
pocket_html:
page_title: Import > Pocket HTML
description: This importer will import all your Pocket bookmarks (via HTML export). Just go to https://getpocket.com/export, then export the HTML file. An HTML file will be downloaded (like "ril_export.html").
how_to: Please choose the bookmark backup file and click on the button below to import it. Note that the process may take a long time since all articles have to be fetched.
pocket_csv:
page_title: Import > Pocket CSV
description: This importer will import all your Pocket bookmarks (via CSV export). Just go to https://getpocket.com/export, then export the file. A ZIP file will be downloaded (like "pocket.zip"). Extract it, you will obtain a CSV file, called "part_000000.csv".
how_to: Please choose the bookmark backup file and click on the button below to import it. Note that the process may take a long time since all articles have to be fetched.
developer:
page_title: API clients management
welcome_message: Welcome to the wallabag API
@ -668,6 +680,7 @@ flashes:
archived_reset: Archived entries deleted
otp_enabled: Two-factor authentication enabled
otp_disabled: Two-factor authentication disabled
otp_code_invalid: Invalid two-factor authentication code
tagging_rules_imported: Tagging rules imported
tagging_rules_not_imported: Error while importing tagging rules
ignore_origin_rules_deleted: 'Ignore origin rule deleted'

View File

@ -530,6 +530,19 @@ import:
page_title: Importer > del.icio.us
how_to: Choisissez le fichier de votre export Delicious et cliquez sur le bouton ci-dessous pour l'importer.
description: Depuis 2021, vous pouvez à nouveau exporter vos données depuis Delicious (https://del.icio.us/export). Choisissez le format "JSON" et téléchargez le (un fichier du genre "delicious_export.2021.02.06_21.10.json").
shaarli:
page_title: Importer > Shaarli
description: Cet importateur importera toutes vos signets Shaarli. Il suffit d'aller à la section Outils, puis dans « Base de données d'exportation », choisissez vos signets et exportez-les. Vous obtiendrez un fichier HTML.
how_to: Veuillez sélectionner le fichier de sauvegarde de signet et cliquez sur le bouton ci-dessous pour l'importer. Notez que le processus peut prendre beaucoup de temps puisque tous les articles doivent être récupérés.
pocket_html:
page_title: Importer > Pocket HTML
description: Cet importateur importera toutes vos signets Pocket (via exportation HTML). Il suffit d'aller à https://getpocket.com/export, puis d'exporter le fichier HTML. Un fichier HTML sera téléchargé (comme « ril_export.html »).
how_to: Veuillez choisir le fichier de sauvegarde de signets et cliquez sur le bouton ci-dessous pour l'importer. Pensez au fait que le processus peut prendre longtemps puisque tous les articles doivent être récupérés.
pocket_csv:
page_title: Importer > Pocket CSV
description: Cet importateur importera toutes vos signets Pocket (via exportation CSV). Il suffit d'aller à https://getpocket.com/export, puis d'exporter le fichier. Un fichier ZIP sera téléchargé (comme « pocket.zip »). Décompressez le et vous obtiendrez un fichier CSV appelé "part_000000.csv".
how_to: Veuillez choisir le fichier de sauvegarde de signets et cliquez sur le bouton ci-dessous pour l'importer. Pensez au fait que le processus peut prendre longtemps puisque tous les articles doivent être récupérés.
developer:
page_title: Gestion des clients API
welcome_message: Bienvenue sur lAPI de wallabag